Entity portrait processing method and device and related product

Abstract

The invention discloses an entity portrait processing method and device and a related product. For each configuration dimension, performing calculation processing based on the dimension weight corresponding to the configuration dimension, the dimension weights corresponding to the plurality of configuration dimensions and the upper limit value of the fragmented memory to obtain a dimension fragmentation number corresponding to the configuration dimension; for each to-be-processed entity and each configuration dimension, performing calculation processing based on an entity identifier corresponding to the to-be-processed entity, a dimension identifier corresponding to the configuration dimension and the total number of dimension fragments corresponding to the plurality of configuration dimensions to obtain a dimension fragment position of the to-be-processed entity under the configuration dimension; and for each to-be-processed entity and each configuration dimension, based on the to-be-processed entity, determining an entity portrait label of the to-be-processed entity under the configuration dimension, and storing the entity portrait label to a dimension fragment position of the to-be-processed entity under the configuration dimension. Therefore, the entity portrait processing performance is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and related products for processing physical images. Background Technology

[0002] As enterprises' digital transformation enters a more complex phase, entity profiling has become a core infrastructure for precision marketing and personalized services. In ultra-large-scale scenarios involving hundreds of millions of entities and trillions of data points, relevant technical solutions typically employ entity identifier-based storage or a hybrid single-mode storage approach to store entity profiles. However, these methods struggle to handle diverse and heterogeneous query demands, resulting in poor storage performance for entity profiles.

[0003] Therefore, how to improve the processing performance of entity images is a key issue of concern to those skilled in the art. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a method, apparatus, and related products for processing entity images, thereby improving the processing performance of entity images.

[0005] The embodiments of this application disclose the following technical solutions:

[0006] The first aspect of this application provides a method for entity image processing. This entity image processing method includes:

[0007] Obtain multiple preset configuration dimensions, as well as multiple entities to be processed and shard memory upper limits, wherein the shard memory upper limits include the upper limit of memory for a single dimension shard, and the dimension shard is used to store entity profile tags;

[0008] For each configuration dimension, the dimension weight is calculated based on the tag information in that configuration dimension. The tag information is related to determining that the entity to be processed belongs to the configuration profile tag under that configuration dimension.

[0009] For each configuration dimension, the number of dimension shards corresponding to that configuration dimension is calculated based on the dimension weight corresponding to that configuration dimension, the dimension weights corresponding to the multiple configuration dimensions respectively, and the upper limit of the shard memory.

[0010] For each entity to be processed and each configuration dimension, the dimension fragment position of the entity to be processed under the configuration dimension is obtained by calculation based on the entity identifier corresponding to the entity to be processed, the dimension identifier corresponding to the configuration dimension, and the total number of dimension fragments corresponding to the multiple configuration dimensions.

[0011] For each entity to be processed and each configuration dimension, based on the entity to be processed, determine the entity profile label of the entity under the configuration dimension, and store the entity profile label to the dimension fragment position of the entity to be processed under the configuration dimension.

[0012] A second aspect of this application provides a solid image processing apparatus. The solid image processing apparatus includes:

[0013] The dimension entity acquisition unit is used to acquire multiple preset configuration dimensions, as well as multiple entities to be processed and the upper limit value of the shard memory, wherein the upper limit value of the shard memory includes the upper limit value of the memory of a single dimension shard, and the dimension shard is used to store entity profile tags.

[0014] The dimension weight acquisition unit is used to calculate and process the label information in each configuration dimension to obtain the dimension weight corresponding to the configuration dimension, wherein the label information is related to determining that the entity to be processed belongs to the configuration profile label under the configuration dimension.

[0015] The sharding quantity acquisition unit is used to calculate the sharding quantity of each configuration dimension based on the dimension weight corresponding to the configuration dimension, the dimension weights corresponding to the multiple configuration dimensions respectively, and the sharding memory limit value.

[0016] The sharding position acquisition unit is used to calculate and process, for each entity to be processed and each configuration dimension, based on the entity identifier corresponding to the entity to be processed, the dimension identifier corresponding to the configuration dimension, and the total number of dimension shards corresponding to the multiple configuration dimensions, to obtain the dimension sharding position of the entity to be processed under the configuration dimension.

[0017] The image tag storage unit is used to determine the entity image tag of each entity to be processed under the configuration dimension based on the entity to be processed, and to store the entity image tag at the dimension fragment position of the entity to be processed under the configuration dimension.

[0018] A third aspect of this application provides a computer device. The computer device includes:

[0019] A memory on which computer programs are stored;

[0020] A processor for executing the computer program in the memory to implement the steps of the entity portrait processing method provided in the first aspect.

[0021] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the program implements the steps of the entity image processing method provided in the first aspect.

[0022] Compared with the prior art, this application has the following beneficial effects:

[0023] In this application, firstly, multiple preset configuration dimensions, multiple entities to be processed, and shard memory limits are obtained. Next, for each configuration dimension, the dimension weight corresponding to that configuration dimension is calculated based on the tag information within that configuration dimension. Then, for each configuration dimension, the number of dimension shards corresponding to that configuration dimension is calculated based on the dimension weight, the dimension weights corresponding to the multiple configuration dimensions, and the shard memory limit. Next, for each entity to be processed and each configuration dimension, the dimension shard position of the entity to be processed under that configuration dimension is calculated based on the entity identifier, the dimension identifier corresponding to the configuration dimension, and the total number of dimension shards corresponding to the multiple configuration dimensions. Finally, for each entity to be processed and each configuration dimension, the entity profile tag of the entity to be processed under that configuration dimension is determined based on the entity to be processed, and the entity profile tag is stored at the dimension shard position of the entity to be processed under that configuration dimension. The maximum memory limit for shards includes the maximum memory limit for a single dimension shard. Dimension shards are used to store entity profile tags, and the tag information is related to determining whether the entity to be processed belongs to the configuration profile tag under that configuration dimension.

[0024] As can be seen, this application uses dimensions as the basic unit, determines the number of dimension shards corresponding to each dimension, and then, for each entity to be processed and each dimension, determines the dimension shard position of the entity under that dimension based on the entity identifier of the entity to be processed, the dimension identifier of that dimension, and the total number of dimension shards across all dimensions. The entity profile tag of the entity under that dimension is then stored at that dimension shard position. Thus, this application not only stores entity profile tags based on dimensions but also clearly defines the specific shard position for each entity profile. This allows for centralized storage of profile tags of the same dimension for the same entity and even distribution of profile tags of the same dimension for different entities. This reduces the number of cross-table or cross-region queries required to query a complete entity profile and also reduces the frequency of frequent queries impacting a few storage nodes, thereby improving the processing performance of entity profiles. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of an entity image processing method provided in this application embodiment;

[0027] Figure 2 A logical diagram illustrating the storage of image tags in an entity image processing method provided in this application embodiment;

[0028] Figure 3 A logical diagram illustrating the storage of image tags in another entity image processing method provided in this application embodiment;

[0029] Figure 4 This is a schematic diagram illustrating the processing of entity images in an entity image processing method provided in an embodiment of this application;

[0030] Figure 5 A flowchart illustrating the updating of fragment positions in an entity image processing method provided in this application embodiment;

[0031] Figure 6 A flowchart illustrating the execution of adjustment instructions in an entity portrait processing method provided in this application embodiment;

[0032] Figure 7 A flowchart illustrating the execution of adjustment instructions in another entity portrait processing method provided in this application embodiment;

[0033] Figure 8 This is a schematic diagram illustrating the use of a state machine in an entity image processing method provided in an embodiment of this application.

[0034] Figure 9 A flowchart illustrating the execution instructions in an entity image processing method provided in this application embodiment;

[0035] Figure 10 A flowchart illustrating the execution instructions in another entity image processing method provided in this application embodiment;

[0036] Figure 11 This is a flowchart illustrating the entire process of entity image processing in an embodiment of this application.

[0037] Figure 12 A complete flowchart of entity image processing in another entity image processing method provided in this application embodiment;

[0038] Figure 13 This is a schematic diagram of the structure of a physical image processing device provided in an embodiment of this application. Detailed Implementation

[0039] As described earlier, with enterprise digital transformation entering a more complex phase, entity profiling has become a core infrastructure for precision marketing and personalized services. In ultra-large-scale scenarios involving hundreds of millions of entities and trillions of data points, relevant technical solutions typically employ entity identifier-based storage or a hybrid single-mode storage approach to store entity profiles. Understandably, these solutions use a unified data storage strategy across the overall storage architecture to store entity profiles for all entities. However, this approach struggles to handle diverse and heterogeneous query demands, resulting in poor storage performance for entity profiles. Therefore, improving the processing performance of entity profiles is a key concern for those skilled in the art.

[0040] To address the above issues, this application provides a solution, proposing an entity profiling processing method, apparatus, and related products, aiming to improve the processing performance of entity profiling. The technical solution of this application first obtains multiple preset configuration dimensions, multiple entities to be processed, and the upper limit of shard memory. Next, for each configuration dimension, calculations are performed based on the tag information in that configuration dimension to obtain the dimension weight corresponding to that configuration dimension. Then, for each configuration dimension, calculations are performed based on the dimension weight corresponding to that configuration dimension, the dimension weights corresponding to the multiple configuration dimensions, and the upper limit of shard memory to obtain the number of dimension shards corresponding to that configuration dimension. Next, for each entity to be processed and each configuration dimension, calculations are performed based on the entity identifier corresponding to that entity, the dimension identifier corresponding to that configuration dimension, and the total number of dimension shards corresponding to the multiple configuration dimensions to obtain the dimension shard position of the entity to be processed under that configuration dimension. Finally, for each entity to be processed and each configuration dimension, based on the entity to be processed, the entity profiling tag under that configuration dimension is determined, and the entity profiling tag is stored at the dimension shard position of the entity to be processed under that configuration dimension. The maximum memory limit for shards includes the maximum memory limit for a single dimension shard. Dimension shards are used to store entity profile tags, and the tag information is related to determining whether the entity to be processed belongs to the configuration profile tag under that configuration dimension.

[0041] As can be seen, this application uses dimensions as the basic unit, determines the number of dimension shards corresponding to each dimension, and then, for each entity to be processed and each dimension, determines the dimension shard position of the entity to be processed under that dimension based on the entity identifier of the entity to be processed, the dimension identifier of that dimension, and the total number of dimension shards across all dimensions. The entity profile tag of the entity to be processed under that dimension is then stored at that dimension shard position. Thus, compared to related technical solutions, this application not only stores entity profile tags based on dimensions but also clearly defines the specific shard position for each entity profile. This allows for centralized storage of profile tags of the same dimension for the same entity and even distribution of profile tags of the same dimension for different entities. This reduces the number of cross-table or cross-region queries required to retrieve a complete entity profile and also reduces the frequency of frequent queries impacting a few storage nodes, thereby improving the processing performance of entity profiles.

[0042] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0043] See Figure 1 This figure is a flowchart of an entity image processing method provided in an embodiment of this application. Figure 1 The entity image processing methods shown include:

[0044] S101: Obtain multiple preset configuration dimensions, as well as multiple entities to be processed and the maximum memory limit of the shard.

[0045] In this step, the preset configuration dimensions include those pre-configured by data engineers. These dimensions can be categorized by domain, such as basic information, health information, and employment information. These dimensions are used to distinguish different dimensions of an entity's information. An entity can be understood as a natural person or an organization. Entities can also be determined based on actual needs in practical applications. In this application, the entity to be processed can be understood as an entity whose entity profile has not yet been stored, or an entity whose entity profile has been updated.

[0046] The maximum memory limit for a shard includes the maximum memory limit for a single dimension shard. Understandably, the maximum memory limit for a single dimension shard can be nMB (n is greater than 1). The maximum amount of data that can be stored within a dimension shard is nMB. Dimension shards are used to store entity profile tags.

[0047] In one feasible implementation, multiple configuration dimensions can be represented as follows: ,in Representation configuration dimension 1, Representation configuration dimension 2, The configuration dimension n represents multiple entities to be processed. ,in Characterize entity 1 to be processed. Characterize entity 2 to be processed. Characterize the entity N to be processed.

[0048] S102: For each configuration dimension, calculate and process based on the label information in that configuration dimension to obtain the dimension weight corresponding to that configuration dimension.

[0049] In this step, the dimension weight calculator is used to calculate the dimension weight corresponding to each configuration dimension. For each configuration dimension, the label information within that dimension is related to determining whether the entity to be processed belongs to the configuration profile label under that configuration dimension. It can be understood that each configuration dimension includes at least one configuration profile label, which is initially empty. This facilitates subsequent determination of whether the entity to be processed matches the configuration profile label within the configuration dimension, thus enabling the storage of profile labels. For example, a configuration dimension might include a basic information dimension, which could include a gender profile label.

[0050] For each configuration dimension, the tag information includes the number of tag dependency fields and the average tag computational complexity. The number of tag dependency fields can be understood as the number of fields that need to be extracted from the entity data of the entity to be processed when calculating whether the entity belongs to the configuration profile tag under this configuration dimension. For example, the gender profile tag requires the entity's gender field. The average tag computational complexity can be understood as the average computational complexity of all configuration profile tags in this configuration dimension. The computational complexity of a configuration profile tag can be determined based on the number of calculation steps or judgments required for the tag. For example, calculating whether entity A belongs to the gender profile tag under this configuration dimension requires only one step, which is the computational complexity.

[0051] In one feasible implementation, the dimension weights can be obtained using formula (1), which is as follows:

[0052] Formula (1)

[0053] in, The dimension weights corresponding to dimension k represent the configuration dimensions. Characterizes the memory resources used in the server that support this application. The characterization server does not utilize the memory resources specified in this application. This characterizes the percentage of memory resources used in the server that correspond to this application. The number of label dependency fields representing configuration dimension k. The average computational complexity of the labels representing the configuration dimension k. The percentage of search popularity for configuration dimension k.

[0054] It should be noted that since steps S101-S105 in this application are the initialization phase, there is no historical data for the configuration dimension k at this time. You can set a preset search popularity percentage, which can be preset according to the actual situation.

[0055] S103: For each configuration dimension, calculate the number of dimension shards corresponding to that configuration dimension based on the dimension weight corresponding to that configuration dimension, the dimension weights corresponding to the multiple configuration dimensions respectively, and the upper limit of the shard memory.

[0056] In this step, the dimension weights corresponding to multiple configuration dimensions can be summed to obtain the sum of dimension weights and values ​​for multiple configuration dimensions. Then, for each configuration dimension, a sharding calculator can be used to calculate the number of shards corresponding to that configuration dimension based on its dimension weight, shard memory limit, and the sum of dimension weights and values ​​for multiple configuration dimensions.

[0057] The number of dimension shards can be understood as the number of dimension shards required in the configuration dimension. Thus, this application ensures that configuration dimensions with high dimension weights receive more shards, while guaranteeing that the shard memory of a single dimension shard does not exceed the upper limit.

[0058] In one feasible implementation, the number of dimension partitions can be obtained by formula (2), which is as follows:

[0059] Formula (2)

[0060] in, The number of dimension partitions corresponding to dimension k represents the configuration dimension. Characterizes the upper limit of memory slices. Represents the dimension weights and values ​​corresponding to multiple configuration dimensions.

[0061] S104: For each entity to be processed and each configuration dimension, calculate and process based on the entity identifier corresponding to the entity to be processed, the dimension identifier corresponding to the configuration dimension, and the total number of dimension fragments corresponding to the multiple configuration dimensions to obtain the dimension fragment position of the entity to be processed under the configuration dimension.

[0062] It should be noted that before executing step S104, this application can sort the various configuration dimensions (e.g., by name A, Z) to obtain the sorted configuration dimensions. Then, the dimension fragments corresponding to each configuration dimension can be sorted according to the number of dimension fragments for each configuration dimension, where all dimension fragments within each configuration dimension are contiguous. For example, in configuration dimension A and configuration dimension B, configuration dimension A has dimension fragments a1 and a2, and configuration dimension B has dimension fragments b1 and b2. In this case, the sorting of the dimension fragments in configuration dimension A and configuration dimension B can be: dimension fragment a1 - dimension fragment a2 - dimension fragment b1 - dimension fragment b2. This facilitates the rapid location of subsequent dimension fragment positions.

[0063] In this step, for each entity to be processed and each configuration dimension, the entity identifier corresponding to the entity to be processed can be hashed to obtain a hashed entity identifier. The entity identifier can be understood as a unique identifier for the entity to be processed, and in this application, it can be represented in an encoded form. Entities with the same entity identifier will have the same hash value. Similarly, the dimension identifier corresponding to the configuration dimension can be hashed to obtain a hashed dimension identifier. The dimension identifier can be understood as a unique identifier for the configuration dimension, and in this application, it can be represented in an encoded form. Different configuration dimensions have different codes, resulting in different hash values, thus separating different configuration dimensions.

[0064] Specifically, in this application, for each entity to be processed and each configuration dimension, a sharding calculator can be used to calculate the hashed entity identifier of the entity to be processed, the hashed dimension identifier of the configuration dimension, and the total number of dimension shards corresponding to multiple configuration dimensions, thereby obtaining the dimension shard position of the entity to be processed under that configuration dimension. Thus, in this application, when the configuration dimensions are the same, the profile tags of the same entity to be processed can fall into the same dimension shard, ensuring that profile tags of the same dimension for the same entity are stored in the same dimension shard, and that profile tags of the same dimension for different entities are evenly distributed across the dimension shards.

[0065] In one feasible implementation, the dimensional fragment position of the entity to be processed under the configuration dimension can be obtained by formula (3), which is as follows:

[0066] Formula (3)

[0067] in, This represents the dimension partitioning position of the entity i to be processed under the configured dimension k. This represents the total number of dimension partitions corresponding to multiple configuration dimensions. The dimension identifier representing the hashed configuration dimension k. The hashed entity identifier representing entity i to be processed. Characterizes the degree of balance in configuration dimension k. The maximum value representing the degree of balance across all configuration dimensions.

[0068] It should be noted that the balance of configuration dimensions is related to the search popularity of profile tags within those dimensions. Since we are currently in the initialization phase, then... =0.

[0069] S105: For each entity to be processed and each configuration dimension, based on the entity to be processed, determine the entity profile label of the entity to be processed under the configuration dimension, and store the entity profile label to the dimension fragment position of the entity to be processed under the configuration dimension.

[0070] In this step, for each entity to be processed and each configuration dimension, the entity dataset of the entity to be processed can be calculated based on the label calculation rules in the configuration profile labels under that configuration dimension. This yields the profile label identifier and the corresponding entity profile value for that entity under that configuration dimension. Since each configuration profile label is different, the label calculation rules within each configuration profile label are also different. For example, the profile label identifier can be a gender profile label, and the corresponding entity profile value can be female.

[0071] Next, the image label identifier and the corresponding entity image value can be determined as the entity image label of the entity to be processed under this configuration dimension. Thus, in this application, when the entity dataset of the entity to be processed is input, the corresponding entity image label can be automatically calculated based on the label calculation rules in the configured image label, thereby improving label calculation efficiency.

[0072] It should be noted that, in this application, while calculating and obtaining the entity profile label, the data coordinator can write the entity profile label of the entity to be processed into the dimension shard position of the entity to be processed under the configured dimension.

[0073] As shown in Table 1, Table 1 is a schematic table of image tags in an entity image processing method provided in an embodiment of this application. In Table 1, Entity_id represents the entity identifier corresponding to the entity to be processed, which can be a certificate number; Dimension_code represents the encoding dictionary and compressed encoding set of the configuration dimension, for example: {"demension_code1":1", "demension_code2":2", "demension_code3":3, ...} and [1, 2, 3, ...], which is the same as the compressed encoding of the configuration dimension, which can be understood as the dimension identifier; Tag_code represents the encoding dictionary and compressed encoding set of the configuration profile tag, for example: {"label_code1":1", "label_code2":2", "label_code3":3, ...} and [1, 2, 3, ...], which can be understood as the tag identifier; Tag_value represents the set of data values ​​(i.e., entity profile values) corresponding to Tag_code, for example: [label_value1, label_value2, label_value3, ...]; Creation time represents the creation time of the configuration profile tag; Update time represents the time when the tag calculation rule in the configuration profile tag was last used. Thus, the encoding compression in this application further saves storage space, mapping complex codes to numbers, which facilitates retrieval for the profile service. Furthermore, compared to row-based storage in related technical solutions, this application uses column-based storage to persist entity profile tags, compressing the profile tags and entity profile values ​​into separate columns. This eliminates the need to add fields or modify the table structure when the number of tags increases; only the corresponding tag encoding dictionary table and compressed encoding set need to be updated to directly insert new data.

[0074] Table 1

[0075]

[0076] As shown in Table 2, Table 2 is a schematic table of storing image tags in an entity image processing method provided in an embodiment of this application. In Table 2, the entity image tag (image tag identifier, entity image value, such as (1, XXX01), (2, YYY01), (3, ZZZ01)) of the entity with entity identifier Xxxx01 is stored in dimension fragment 01 of configuration dimension 1; the entity image tag (image tag identifier, entity image value, such as (1, XXX03)) of the entity with entity identifier Xxxx03 is stored in dimension fragment 01 of configuration dimension 1; the entity image tag (image tag identifier, entity image value, such as (1, XXX01), (2, YYY02), (3, ZZZ03)) of the entity with entity identifier Xxxx02 is stored in dimension fragment 02 of configuration dimension 1; the entity image tag (image tag identifier, entity image value, such as (4, UUU01)) of the entity with entity identifier Xxxx01 is stored in dimension fragment 01 of configuration dimension 2.

[0077] Table 2

[0078]

[0079] like Figure 2 As shown, Figure 2 This is a logical diagram illustrating the storage of image tags in an entity image processing method provided in an embodiment of this application. Figure 2 In this context, multiple configuration dimensions include at least configuration dimension D_A and configuration dimension D_B. Configuration dimension D_A includes D_A dimension shard 1, D_A dimension shard 2, D_A dimension shard... and D_A dimension shard n, while configuration dimension D_B includes D_B dimension shard 1 and D_B dimension shard... It can be understood that all entity profile tags for all entities within the same dimension shard are evenly distributed. Furthermore, in subsequent processes (except the initialization phase), the number of shards for each configuration dimension is determined by the retrieval popularity (or data update frequency) of the configuration profile tags within that dimension. The higher the retrieval popularity (update frequency), the more dimension shards the current configuration dimension has. That is, the number of dimension shards changes dynamically with the popularity (frequency).

[0080] like Figure 3 As shown, Figure 3 This is a logical diagram illustrating the storage of image tags in another entity image processing method provided in an embodiment of this application. Figure 3In this application, D_A dimension shard 1 and D_A dimension shard ... are stored on physical node 1, D_A dimension shard 2, D_A dimension shard n and D_B dimension shard 1 are stored on physical node 2, and D_B dimension shard ... are stored on physical node ... . It is understandable that in this application, entity profile tags of the same dimension for the same entity are evenly distributed across several dimension shards, and these dimension shards are located on different physical nodes. This avoids the impact of high-frequency access to a single node when the profile service frequently queries tags of a certain configured dimension, thus reducing single points of failure. Alternatively, when the profile service queries entity profile tags for a certain entity in a subsequent process, the entity profile tags of the same dimension for that entity exist on a single physical node and logical shard, which can improve query efficiency.

[0081] like Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the processing of entity images in an entity image processing method provided in an embodiment of this application. Figure 4 First, data engineers can configure profile templates A, B, and ... . Profile template A can include configuration dimension A and configuration dimension ..., profile template B can include configuration dimension A, configuration dimension B, and configuration dimension ..., and profile template ... can include configuration dimension .... Configuration dimension A can include configuration profile tag A, configuration profile tag B, configuration profile tag C, and configuration profile tag ..., configuration dimension B can include configuration profile tag L, configuration profile tag M, and configuration profile tag ..., and configuration dimension ... can include configuration profile tag X, configuration profile tag Y, and configuration profile tag ... This allows for the establishment of relationships in the form of "profile template - configuration dimension - configuration profile tag" and storage in the metadata database.

[0082] Next, entity datasets corresponding to multiple entities to be processed can be obtained from data sources (such as database 1, database 2, etc.). Based on the tag calculation rules in each configured profile tag, entity profile tags for each entity dataset corresponding to multiple entities to be processed are obtained under the configured dimension. This enables the storage of entity profile tags in a tag database, thereby providing data support for subsequent entity profile queries. It should be noted that the computation scheduling in this application can not only calculate the entity profile tags corresponding to the entities to be processed, but also realize the dynamic adjustment of subsequent profile tags.

[0083] Next, data users can query the entity profile of entity A from the profile template. Understandably, this is stored in the metadata database in the format of "profile template - configuration dimension - configuration profile tag". Therefore, the database can retrieve entity profile values ​​containing configuration dimension A and configuration dimension ..., ultimately displayed under configuration profile tag A, configuration profile tag B, configuration profile tag C, configuration profile tag X, configuration profile tag Y, and configuration profile tag ... This allows data users to clearly understand the entity profile of entity A.

[0084] Understandably, the configuration dimensions in this application are not simply for classification, but rather fundamentally reorganize the storage, calculation, and querying methods of entity profiles, enabling local calculations to replace global scanning, so that the subsequent process profile tag storage can be independently expanded or reduced, thus providing support for improving query efficiency.

[0085] It should be noted that after performing the operations of determining the entity profile label of the entity to be processed under the configuration dimension based on the entity to be processed, and storing the entity profile label to the dimension fragment position of the entity to be processed under the configuration dimension, this application can also perform steps S501-S504. For example... Figure 5 As shown, Figure 5 This is a flowchart illustrating the updating of fragment positions in an entity image processing method provided in an embodiment of this application. Figure 5 This includes steps S501-S504, which are detailed below:

[0086] S501: In response to the query frequency of entity profile tags under the target dimension being greater than the preset query frequency, or in response to the execution frequency of tag calculation rules in the configuration profile tags under the target dimension being greater than the preset execution frequency, obtain the target dimension weight corresponding to the target dimension based on the target dimension.

[0087] In this step, the target dimension can be any one of multiple configuration dimensions. The preset query frequency and preset execution frequency are pre-set and can be set according to needs in actual applications. If the query frequency of entity profile tags under the target dimension is greater than the preset query frequency, it can be understood that the entity profile tags under the target dimension are frequently queried. If the execution frequency of the tag calculation rules in the configured profile tags is greater than the preset execution frequency, it can be understood that the tag calculation rules are used to perform calculations frequently, indicating that there are significant changes in the entity dataset.

[0088] It should be noted that before performing the operation of obtaining the target dimension weight based on the target dimension, this application can also obtain the number of entity profile tags corresponding to multiple configuration dimensions. The number of entity profile tags corresponding to a configuration dimension can be understood as the total number of entity profile tags existing under that configuration dimension.

[0089] Next, based on the number of dimension fragments corresponding to the target dimension, the inverse of the number of fragments can be obtained. This inverse can be understood as the inverse proportion of the number of dimension fragments, used to balance the impact of the number of fragments on the dimension weight, preventing excessive expansion of dimensions that have already been fragmented. Furthermore, the number of entity profile tags corresponding to multiple configuration dimensions and the number of entity profile tags corresponding to the target dimension can be calculated to obtain the proportion of entity profile tags corresponding to the target dimension. This proportion represents the percentage of profile tags included in the target dimension compared to the total number of profile tags included in all configuration dimensions.

[0090] Specifically, this application uses a dimension weight calculator to calculate the target dimension weight based on the query frequency of entity profile tags under the target dimension, the inverse of the number of shards corresponding to the target dimension, and the proportion of entity profile tags corresponding to the target dimension. The query frequency represents the degree of usage of the target dimension in the business; a higher query frequency indicates greater attention to the dimension and a potential need for more storage resources. Thus, this application can dynamically calculate dimension weights, encouraging high-frequency access and large-volume dimensions to receive more shards, while preventing already over-sharded dimensions from continuing to expand.

[0091] In one feasible implementation, the target dimension weights can be obtained through formula (4), which is as follows:

[0092] Formula (4)

[0093] in, The target dimension weights corresponding to the configuration dimension k are represented. The query frequency of entity profile tags under configuration dimension k represents the configuration frequency. The proportion of entity profile tags corresponding to configuration dimension k. The reciprocal of the number of partitions representing the number of partitions corresponding to the configuration dimension k. , and The adjustment index is trained using historical data. Among them, Generally, a value greater than 1 indicates its importance in amplifying query frequency (popularity). It is usually between 0 and 1, which indicates that multiple partitions should be made for the data volume dimension; It is usually between 0 and 1, and is used to prevent excessive fragmentation.

[0094] S502: Calculate and process the weight judgment interval based on the interval threshold and the dimension weights corresponding to the multiple configuration dimensions respectively.

[0095] In this step, the average weights corresponding to multiple configuration dimensions are calculated to obtain the average weight for each configuration dimension. The interval threshold is preset and can be understood as half the width of the cache area for the average weight, typically 0.1-0.2. Then, the interval threshold and the average weight for each configuration dimension are calculated to obtain a first judgment weight and a second judgment weight, where the first judgment weight is less than the second judgment weight. Based on the first and second judgment weights, the weight judgment interval is determined.

[0096] In one feasible implementation, the weight judgment interval can be represented as follows: ,in The first judgment weight is represented. Characterizes the weight of the second judgment. Characterizing the threshold of the interval, It represents the average weight of multiple configuration dimensions.

[0097] It should be noted that before executing step S503, this application can also determine whether the target dimension weight is less than the first determination weight. If the target dimension weight is less than the first determination weight, the number of dimension fragments needs to be recalculated to reduce the number of fragments. It can also determine whether the target dimension weight is greater than the second determination weight. If the target dimension weight is greater than the second determination weight, the number of dimension fragments needs to be recalculated to increase the number of fragments.

[0098] S503: If the target dimension weight is less than the first judgment weight, or if the target dimension weight is greater than the second judgment weight, calculate the target dimension weight based on the interval threshold, the number of dimension fragments corresponding to the target dimension, the target dimension weight corresponding to the target dimension, and the dimension weights corresponding to the multiple configuration dimensions respectively, to obtain the number of target dimension fragments corresponding to the target dimension.

[0099] In this step, when the target dimension weight is less than the first judgment weight, the number of first dimension fragments corresponding to the target dimension can be obtained by using a fragment count calculator based on the interval threshold, the number of dimension fragments corresponding to the target dimension, the target dimension weight corresponding to the target dimension, and the dimension weights corresponding to multiple configuration dimensions. The number of first dimension fragments can be determined as the number of target dimension fragments.

[0100] Alternatively, if the target dimension weight is greater than the second judgment weight, the number of second dimension fragments corresponding to the target dimension can be obtained by using a fragment count calculator based on the interval threshold, the number of dimension fragments corresponding to the target dimension, the target dimension weight corresponding to the target dimension, and the dimension weights corresponding to multiple configuration dimensions. The number of second dimension fragments can be determined as the number of target dimension fragments.

[0101] In one feasible implementation, the number of target dimension fragments can be obtained through formulas (5) and (6), which are as follows:

[0102] Formula (5)

[0103] Formula (6)

[0104] in, The target dimension fragments of dimension k are represented in the next time step closest to the current time step. This represents the number of dimension partitions for dimension k at the current moment. Characterization sensitivity coefficient.

[0105] Understandably, this application assumes that the target dimension weight is less than the first judgment weight (i.e., ), can be adopted The number of target dimension fragments is obtained; in this application, when the target dimension weight is greater than the second judgment weight ( ), can be adopted The number of target dimension fragments is obtained; in this application, the target dimension weight is between the first judgment weight and the second judgment weight ( The number of dimension fragments for the target dimension can be directly determined as the number of dimension fragments for the target dimension.

[0106] S504: For each entity in the target dimension, calculate the target dimension fragment position of the entity based on the entity identifier corresponding to the entity, the dimension identifier corresponding to the target dimension, the number of target dimension fragments corresponding to the target dimension, the balance value corresponding to the target dimension, and the total number of dimension fragments in the multiple configuration dimensions excluding the target dimension.

[0107] In this step, the equilibrium value of the target dimension is used as a metric to measure the uniformity of the distribution of entity profile tags across the various dimensional slices within the target dimension. The target dimension slice position of the entity may be the same as the dimensional slice position of the entity within the target dimension, or it may be different from the dimensional slice position of the entity within the target dimension.

[0108] In one feasible implementation, the target dimension fragment position of the entity in the target dimension can be obtained by formula (7), which is as follows:

[0109] Formula (7)

[0110] in, This represents the target dimension partition position of entity i under configuration dimension k. Represents the total number of dimension shards corresponding to multiple configuration dimensions (including the number of target dimension shards for configuration dimension k). The dimension identifier representing the hashed configuration dimension k. The hashed entity identifier representing entity i. Characterizes the degree of balance in configuration dimension k. The maximum value representing the degree of balance across all configuration dimensions.

[0111] It should be noted that the balance of configuration dimensions is related to the search popularity of profile tags within those dimensions. Since we are currently in the adaptive phase following the initialization phase, then... Not equal to 0, this (The sum of the absolute values ​​of the heat (load) differences between all pairs of dimension fragments in configuration dimension k) / (2 × the number of dimension fragments in configuration dimension k² × average load), where the average load can be understood as the average value of the load corresponding to each dimension fragment in configuration dimension k.

[0112] The adaptive phase can be understood as follows: after automatic tag scheduling is initiated, the tag storage structure is adjusted in real time as the entity dataset changes and the query frequency of entity profile tags changes. The purpose of this phase is to maintain efficient querying of entity profiles by adjusting the tag storage structure and allocating storage resources in a timely and reasonable manner.

[0113] Thus, in this application, in environments with massive amounts of data, storage space utilization can be improved by dynamically adjusting the storage structure of entity profile tags, using dimensions as the basic unit. This enhances the query efficiency of the profile service and prevents high-frequency queries of entity profile tags in hot dimensions from causing a surge in storage node load or even crashes. In other words, this application can adapt to dynamic changes in business queries, solve the problems of hotspot sharding and uneven load distribution in related technical solutions, and ensure system stability and query performance.

[0114] It should be noted that after performing calculations based on the entity identifier corresponding to the entity, the dimension identifier corresponding to the target dimension, the number of target dimension shards corresponding to the target dimension, the load balancing value corresponding to the target dimension, and the total number of dimension shards excluding the target dimension among multiple configuration dimensions to obtain the target dimension shard position of the entity under the target dimension, this application can also execute steps S601-S604. For example... Figure 6 As shown, Figure 6 This is a flowchart illustrating the execution of an adjustment instruction in an entity portrait processing method provided in an embodiment of this application. Figure 6 This includes steps S601-S604, which are detailed below:

[0115] S601: Generate a portrait storage adjustment instruction based on the total number of dimension fragments corresponding to the multiple configuration dimensions, the number of target dimension fragments corresponding to the target dimension, and the total number of dimension fragments among the multiple configuration dimensions excluding the target dimension.

[0116] In this step, the number of target dimension fragments corresponding to the target dimension and the total number of dimension fragments across multiple configuration dimensions (excluding the target dimension) are calculated, along with the new total number of dimension fragments corresponding to the multiple configuration dimensions. Then, the total number of dimension fragments corresponding to the multiple configuration dimensions is compared with the new total number of dimension fragments corresponding to the multiple configuration dimensions.

[0117] If the total number of dimension shards corresponding to multiple configuration dimensions is greater than the total number of new dimension shards corresponding to multiple configuration dimensions, a shrinkage instruction is generated, and this shrinkage instruction is identified as a profile storage adjustment instruction. If the total number of dimension shards corresponding to multiple configuration dimensions is less than the total number of new dimension shards corresponding to multiple configuration dimensions, an expansion instruction is generated, and this expansion instruction is identified as a profile storage adjustment instruction. If the total number of dimension shards corresponding to multiple configuration dimensions is equal to the total number of new dimension shards corresponding to multiple configuration dimensions, no profile storage adjustment instruction is generated.

[0118] The shrink command indicates a reduction in the number of dimensional shards and adjusts the entity image tags to the reduced dimensional shards. The expand command indicates an increase in the number of dimensional shards and adjusts the entity image tags to the increased dimensional shards.

[0119] S602: Based on the total number of query requests corresponding to the target dimension in the first time period, obtain the total number of prediction requests corresponding to the target dimension in the second time period.

[0120] In this step, the first timeframe is earlier than the second timeframe. The first and second timeframes can be determined based on actual needs in practical applications. In this application, a time series prediction model can be used to predict the total number of query requests corresponding to the target dimension within the first timeframe, obtaining the predicted total number of requests corresponding to the target dimension within the second timeframe output by the model. This lays the groundwork for subsequently determining the real-time pressure of the configuration dimension.

[0121] S603: Calculate the dimensional pressure value based on the total number of prediction requests corresponding to the target dimension in the second time period, the total number of query requests corresponding to the target dimension in the second time period, the preset maximum number of requests corresponding to a single dimension shard in the target dimension, the number of target dimension shards corresponding to the target dimension, and the uniform value of image storage.

[0122] In this step, the preset maximum number of requests for a single shard in the target dimension can be understood as the maximum number of query requests that a single shard in the target dimension can receive; the image storage uniformity value characterizes the degree of unevenness in the storage of entity images in the shards of the target dimension. This dimensional pressure value is used to assist in making migration decisions for entity image tags.

[0123] In one feasible implementation, the dimensional pressure value can be obtained by formula (8), which is as follows:

[0124] Formula (8)

[0125] in, The numerical value representing the dimensional pressure of configuration dimension k. This represents the total number of query requests corresponding to configuration dimension k within the second time period. This represents the total number of prediction requests corresponding to configuration dimension k within the second time period. The preset maximum number of requests corresponding to a single dimension shard in configuration dimension k. The number of target dimension fragments corresponding to the configuration dimension k. The characterization image stores uniform values. . , and To adjust the parameters, and .Should =Average absolute difference of the number of data points configured in dimension k in each dimension shard / (2 × avg(number of data points configured in dimension k in each dimension shard)).

[0126] S604: Execute the image storage adjustment instruction based on the dimensional pressure value.

[0127] In this step, the larger the dimensional pressure value, the more urgent the expansion of the dimensional partitioning is, and vice versa.

[0128] like Figure 7 As shown, Figure 7 This is a flowchart illustrating the execution of adjustment instructions in another entity image processing method provided in this application embodiment. Figure 7 First, the decision-maker obtains the image storage adjustment instruction and acquires the dimensional pressure value of the target dimension. Next, if the decision-maker determines that the dimensional pressure value is greater than the preset pressure value, it determines that expansion is needed; if the decision-maker determines that the dimensional pressure value is not greater than the preset pressure value, it determines that expansion is not needed and a shrinkage instruction needs to be executed, entering the migration queue.

[0129] Next, if the decision-maker determines that the dimensional pressure value is greater than the first expansion pressure value, it directly executes the expansion command; if the decision-maker determines that the dimensional pressure value is between the first and second expansion pressure values, it delays the execution of the expansion command and continues detection; if the decision-maker determines that the dimensional pressure value is less than the second expansion pressure value, it enters the migration queue. The preset expansion pressure values ​​include both the first and second expansion pressure values, where the first expansion pressure value is greater than the second expansion pressure value, and the preset pressure value is less than the preset expansion pressure value. The preset pressure values ​​and the preset expansion pressure values ​​can be set according to actual needs in practical applications.

[0130] It's important to note that the migration queue stores low-priority tasks, waiting to be executed sequentially. The purpose of the shrinking instruction is to release data control; it has the lowest priority. Furthermore, the most urgent expansion task should interrupt currently executing low-priority tasks to execute immediately. In other words, interrupting an ongoing expansion task has no impact on current data access, as the data service's pointers to access data have not yet switched to the new shard.

[0131] Thus, this application employs a dual-decision system to prevent frequent scaling-up and scaling-down commands caused by environmental fluctuations. This ensures smooth scaling-up and scaling-down of dimensional shards, minimizing load fluctuations in the hardware environment. Furthermore, it enables independent scaling-up and scaling-down for sudden traffic spikes in specific dimensions, reducing operational costs and improving response efficiency.

[0132] like Figure 8 As shown, Figure 8This diagram illustrates the use of a state machine in an entity image processing method provided in an embodiment of this application. Figure 8 The diagram illustrates the process of the state machine in the decision maker performing expansion and contraction. Figure 8 (a) shows the expansion process implemented by double write. It should be understood that the expansion is achieved by adding shards on the basis of the original stable state, copying the entity profile tags to the new shard and the old shard, completing the double write, and obtaining the new stable state after the routing switch is completed. Figure 8 (b) shows the scaling-down process implemented by scaling down single write. It should be understood that the fragments are reduced and frozen based on the original stable state, so that entity profile tags are written into the old fragments, and a new stable state is obtained after the writing is completed and the routing switch is completed.

[0133] It should also be noted that in this application, the adjustment and storage of entity profile tags can be dynamically realized by establishing a mapping relationship table. As shown in Table 3, Table 3 is a schematic table of the mapping relationship table in an entity profile processing method provided in this application embodiment. In Table 3, the mapping relationship table includes a mapping header and entities whose storage locations need to be dynamically adjusted. The mapping header records the entity information that needs to adjust the sharding position in this adaptive phase. The mapping header includes a dimension identifier, priority, adaptive batch, execution flag, expansion / reduction task ID, processing flag, and processing time. Among them, the priority can be determined by the dimension pressure value. The larger the dimension pressure value, the higher the priority. The higher the priority, the earlier the expansion operation is executed. The adaptive batch is the sequence number formed after initiating the expansion or reduction instruction. The execution order of expansion and reduction depends first on the priority and then on the adaptive batch.

[0134] Understandably, this application associates logical shards with physical shards. During the migration of entity profile tags, the entity profile tags are written to the corresponding positions according to the mapping relationship table. The mapping element records the original landing point information (dimensional shard position) and the new landing point information (target dimension shard position) of each entity's entity profile tag, so as to be traversed sequentially during the data migration stage. In this way, this application separates the change of "logical shard mapping" from the migration of "physical data blocks" and decouples them through the "new and old shard mapping set" to achieve near-seamless data migration without affecting external access to dimension and tag data.

[0135] Table 3

[0136]

[0137] In one feasible implementation, the process of "executing the image storage adjustment instruction based on the dimensional pressure value" in this application can be specifically as follows:

[0138] like Figure 9 As shown, Figure 9 This is a flowchart illustrating the execution instructions in an entity image processing method provided in an embodiment of this application. Figure 9 If the dimensional pressure value is greater than the preset pressure value and also greater than the preset expansion pressure value, an expansion command is executed, with this command having the highest priority. Next, for each entity in the target dimension, if the entity's dimensional shard position in the target dimension differs from its target dimensional shard position, and if the entity's entity profile label is not a new entity profile label, then the entity's entity profile label is stored in the target dimensional shard position of the entity in the target dimension. Alternatively, for each entity in the target dimension, if the entity's dimensional shard position in the target dimension differs from its target dimensional shard position, and if the entity's entity profile label is a new entity profile label, then the entity's entity profile label is stored in both the dimensional shard position of the entity in the target dimension and the target dimensional shard position.

[0139] Specifically, first, the decision-maker initiates a scaling command. Then, it calls the scaling interface of the shard manager, which connects to the label database and adds new dimension shards in the label database. The number of new shards is equal to the number of dimension shards calculated in the adaptive phase minus the original number of dimension shards.

[0140] Next, the mapping set is read into the local cache through the expansion coordinator. It then iterates through each mapping element in the set, checking if the dimension shard position in the mapping element is the same as the target dimension shard position. If the dimension shard position is the same as the target dimension shard position (i.e., in the same dimension shard), and the entity's entity profile tag is the new entity profile tag, then the write data interface connected to the tag database in the shard manager is called to update the data in the existing shard. If the dimension shard position is the same as the target dimension shard position, and the entity's entity profile tag is not the new entity profile tag, then no operation is required, meaning the data remains unchanged.

[0141] If the dimension shard position and the target dimension shard position are in different dimension shards, and the entity's entity profile label is a new entity profile label that differs from both the dimension shard position and the target dimension shard position (i.e., it's a newly added dimension shard), the expansion coordinator starts two threads simultaneously, calling the write data interface in the shard manager to write the new entity profile label to both the dimension shard position and the target dimension shard position. This ensures that the current dimension data is not affected before the data pointer points to the new shard. If the dimension shard position and the target dimension shard position are in different dimension shards, and the entity's entity profile label is a new entity profile label that is the same as both the dimension shard position and the target dimension shard position (i.e., it's not a newly added dimension shard), then only one thread is started to write the new entity profile label to the target dimension shard position.

[0142] If the dimension sharding position and the target dimension sharding position are in different dimension shards, and the entity's entity profile label is not a new entity profile label, and is different from the dimension sharding position and the target dimension sharding position, then the expansion coordinator calls the write data interface in the sharding manager to write the new entity profile label to the newly added shard at the new landing point; if the dimension sharding position and the target dimension sharding position are in different dimension shards, and the entity's entity profile label is not a new entity profile label, and is the same as the dimension sharding position and the target dimension sharding position, then the expansion coordinator calls the write data interface in the sharding manager to write the new entity profile label to the original shard at the new landing point.

[0143] Finally, when the detector detects the end of the traversal and the completion of the data migration, it updates the dimension configuration set, directing the external data service to the updated data storage area. Furthermore, the detector calls the shard manager's data cleanup interface to remove data from the mapping set whose original and new endpoints are no longer in the same shard.

[0144] Thus, the expansion mechanism in this application allows new data to be written to both the original and new shards simultaneously. When data migration occurs, or when entity data is updated, it ensures that both the old and new shard sets contain the latest data. The entire process, including before and after data pointer switching, does not affect data access for external profiling services. Fine-grained differentiation is applied to the migration of existing data, avoiding a "one-size-fits-all" copy operation of entity data with unchanged endpoints, saving storage space, reducing data storage fragmentation, and simplifying migration steps to improve efficiency.

[0145] In one feasible implementation, the process of "executing the image storage adjustment instruction based on the dimensional pressure value" in this application can be specifically as follows:

[0146] like Figure 10 As shown, Figure 10 This is a flowchart illustrating the execution instructions in another entity profiling method provided in this application embodiment. Figure 10 If the dimensional pressure value is not greater than the preset pressure value, a scaling instruction is executed. For each entity in the target dimension, if the dimensional sharding position of the entity in the target dimension is different from the target dimensional sharding position, the entity profile tag of the entity is stored in the target dimensional sharding position of the entity in the target dimension.

[0147] Specifically, the decision-maker first initiates a reduction of the fragment area command. Then, it calls the freeze interface of the fragment manager, which connects to the tag database to modify the read-only configuration of the fragments to be deleted in all dimension fragments of the current configuration dimension, preventing the storage of new entity profile tags. The number of fragments to be deleted is equal to the original number of dimension fragments minus the number of dimension fragments calculated in the adaptive phase.

[0148] Next, the mapping set is read into the local cache by the shrinking coordinator. Then, iterates through each mapping element in the set and checks if the dimension shard position in the mapping element is the same as the target dimension shard position (i.e., whether they are the same shard). If the dimension shard position is the same as the target dimension shard position, and the current entity needs to have a new entity profile label written, the write data interface connected to the database in the shard manager is called to update the data in the remaining dimension shards. If the dimension shard position is the same as the target dimension shard position, and the current entity does not need to have a new entity profile label written, no operation is required, meaning the data remains unchanged.

[0149] If the dimension sharding location and the target dimension sharding location are different, the entity profile tag of the entity is written to the target dimension sharding location of the entity under the target dimension by calling the write data interface in the sharding manager in the shrink coordinator. The above process is repeated until the mapping set traversal is completed.

[0150] Finally, when the detector detects the end of the traversal and the completion of the data migration, it updates the dimension configuration set, causing the external data service to point to the updated data storage area. Furthermore, the detector calls the data cleanup interface of the shard manager to clear data in the mapping set whose original and new landing points are not in the same shard, and deletes the frozen shards to be deleted. Thus, in this application, the shards to be deleted are frozen to prevent the write data interface from writing data to them. Otherwise, when the migration is complete and the data service's data pointer points to the new shard set, some data would be lost along with the deletion of the shards to be deleted, thereby avoiding data impact on the external profiling service during data migration or entity data updates.

[0151] like Figure 11 As shown, Figure 11 This is a flowchart illustrating the entire process of entity image processing in an embodiment of this application. Figure 11 This application includes an initialization phase, an adaptation phase, and a data migration phase. In the initialization phase, steps S101-S105 are executed using a dimension weight calculator, a shard number calculator, a shard placement calculator, and a shard manager, thereby writing entity profile tags to the corresponding dimension shard positions. In the adaptation phase, steps S501-S505 are executed using a detector, a dimension weight calculator, a shard number calculator, and a shard placement calculator, thereby dynamically adjusting entity profile tags to the corresponding dimension shard positions. In the data migration phase, the expansion and contraction processes are executed using a decision-maker, a shard manager, and a data coordinator (expansion coordinator and contraction coordinator).

[0152] Thus, this application enables precise differentiation of business dimensions, as well as between frequently accessed and infrequently accessed dimensions. This allows for centralized storage of same-dimensional tags for the same entity and even distribution of same-dimensional tags across different entities. This reduces the number of cross-table or cross-region queries required to obtain a complete entity profile, and minimizes the impact of frequent queries on a few storage nodes, avoiding a double waste of storage space and computing resources. Furthermore, it allows for precise resource allocation and elastic scaling for hot dimensions, resulting in an overall even distribution of entity data and a centralized distribution of tag data across dimensions. This achieves an optimal balance between storage efficiency, query performance, and resource costs under ultra-large-scale data conditions.

[0153] like Figure 12 As shown, Figure 12 This is a flowchart illustrating the complete process of entity image processing in another entity image processing method provided in this application embodiment. Figure 12 First, under the action of a timer, the entity dataset of the entity is periodically processed using the tag calculation rules in the configuration profile tags in the metadata database to obtain the entity profile tags of the entity under the configuration dimension, and stored in the corresponding dimension shard position. Next, the detector checks whether the query frequency of the entity profile tags under the target dimension is greater than the preset query frequency, or whether the execution frequency of the tag calculation rules in the entity profile tags under the target dimension is greater than the preset execution frequency. If the query frequency of the entity profile tags under the target dimension is greater than the preset query frequency, or the execution frequency of the tag calculation rules in the entity profile tags under the target dimension is greater than the preset execution frequency, the dynamic storage engine is triggered (see steps S501-S505 above, the scaling process), thereby dynamically adjusting the storage of the entity profile tags to maintain the entity profile tags after the storage position adjustment in the tag database.

[0154] Thus, this application intelligently distributes entity profile tags according to configuration dimensions, ensuring that data for the same dimension tag of the same entity is stored in the same shard. Hot dimension tags are not subject to cross-shard queries, significantly improving the efficiency of calculating and querying derived tags within the same dimension. Queries are transformed from "full-database scanning" to "precise positioning," reducing latency by more than 10 times. This ensures even distribution of entity data across dimension shards, reducing the probability of high load on a single node and guaranteeing high availability. Furthermore, hot and cold dimensions are allocated different storage resources as needed, and the number of data shards is intelligently controlled, resulting in a significant reduction in storage costs and a substantial improvement in resource utilization while maintaining optimal performance. Scaling and scaling are completed automatically and losslessly in the background, with minimal impact on the physical environment. The profile query service remains completely unaware of these changes, and prediction-based automatic decision-making and scheduling require no human intervention, operating fully automatically and intelligently, thus improving the processing performance of entity profiles.

[0155] In summary, this embodiment uses dimensions as the basic unit, determines the number of dimension shards corresponding to each dimension, and then, for each entity to be processed and each dimension, determines the dimension shard position of the entity to be processed within that dimension based on the entity identifier of the entity to be processed, the dimension identifier of that dimension, and the total number of dimension shards across all dimensions. The entity profile tag of the entity to be processed within that dimension is then stored at that dimension shard position. Thus, compared to related technical solutions, this application not only stores entity profile tags based on dimensions but also clearly defines the specific shard position for each entity profile. This allows for centralized storage of profile tags of the same dimension for the same entity and even distribution of profile tags of the same dimension for different entities. This reduces the number of cross-table or cross-region queries required to query a complete entity profile and also reduces the frequency of frequent queries impacting a few storage nodes, thereby improving the processing performance of entity profiles. It achieves precise allocation and elastic scheduling of storage resources, fundamentally solving the efficiency, cost, and performance dilemmas of related technical solutions.

[0156] Based on the entity image processing method provided in the foregoing embodiments, this application also provides an entity image processing apparatus. Figure 13 This is a schematic diagram of a physical image processing device provided in an embodiment of this application. Figure 8 As shown, the entity image processing device includes:

[0157] The dimension entity acquisition unit 1301 is used to acquire multiple preset configuration dimensions, as well as multiple entities to be processed and the upper limit value of the shard memory, wherein the upper limit value of the shard memory includes the upper limit value of the memory of a single dimension shard, and the dimension shard is used to store entity profile tags.

[0158] The dimension weight acquisition unit 1302 is used to calculate and process the label information in the configuration dimension for each configuration dimension to obtain the dimension weight corresponding to the configuration dimension, wherein the label information is related to determining that the entity to be processed belongs to the configuration profile label under the configuration dimension.

[0159] The sharding quantity obtaining unit 1303 is used to calculate the number of dimension shards corresponding to each configuration dimension based on the dimension weight corresponding to the configuration dimension, the dimension weights corresponding to the multiple configuration dimensions respectively, and the upper limit value of the sharding memory.

[0160] The sharding position acquisition unit 1304 is used to calculate and process, for each entity to be processed and each configuration dimension, based on the entity identifier corresponding to the entity to be processed, the dimension identifier corresponding to the configuration dimension and the total number of dimension shards corresponding to the multiple configuration dimensions, to obtain the dimension sharding position of the entity to be processed under the configuration dimension.

[0161] The image tag storage unit 1305 is used to determine the entity image tag of the entity to be processed under the configuration dimension for each entity to be processed and each configuration dimension, and to store the entity image tag to the dimension fragment position of the entity to be processed under the configuration dimension.

[0162] In one feasible implementation, each of the configuration dimensions includes at least one configuration profile tag, wherein the configuration profile tag is a null value; the profile tag storage unit 1305 is specifically used for:

[0163] Based on the tag calculation rules in the configuration profile tags under this configuration dimension, the entity dataset in the entity to be processed is processed to obtain the profile tag identifier of the entity to be processed under this configuration dimension and the entity profile value corresponding to the profile tag identifier.

[0164] The image tag identifier and the entity image value corresponding to the image tag identifier are determined as the entity image tag of the entity to be processed in this configuration dimension.

[0165] In one feasible implementation, the device further includes:

[0166] The target weight acquisition unit is used to obtain the target dimension weight corresponding to the target dimension based on the target dimension in response to the query frequency of entity profile tags under the target dimension being greater than the preset query frequency, or in response to the execution frequency of tag calculation rules in the configuration profile tags under the target dimension being greater than the preset execution frequency, wherein the target dimension is any one of the plurality of configuration dimensions;

[0167] The judgment interval obtaining unit is used to perform calculation processing based on the interval threshold and the dimension weights corresponding to the multiple configuration dimensions respectively to obtain the weight judgment interval, wherein the weight judgment interval includes a first judgment weight and a second judgment weight, and the first judgment weight is less than the second judgment weight.

[0168] The target quantity acquisition unit is used to calculate the target dimension fragment number corresponding to the target dimension based on the interval threshold, the number of dimension fragments corresponding to the target dimension, the target dimension weight corresponding to the target dimension, and the dimension weights corresponding to the multiple configuration dimensions if the target dimension weight is less than the first judgment weight, or if the target dimension weight is greater than the second judgment weight.

[0169] The target location acquisition unit is used to calculate and process, for each entity in the target dimension, based on the entity identifier corresponding to the entity, the dimension identifier corresponding to the target dimension, the number of target dimension fragments corresponding to the target dimension, the balance value corresponding to the target dimension, and the total number of dimension fragments in the multiple configuration dimensions excluding the target dimension, to obtain the target dimension fragment position of the entity in the target dimension.

[0170] In one feasible implementation, the device further includes:

[0171] The tag quantity acquisition unit is used to acquire the number of entity profile tags corresponding to the multiple configuration dimensions respectively;

[0172] The tag percentage obtaining unit is used to obtain the reciprocal of the number of shards of the dimension corresponding to the target dimension based on the number of dimension shards corresponding to the target dimension; and to obtain the entity image tag percentage corresponding to the target dimension based on the number of entity image tags corresponding to the multiple configuration dimensions and the number of entity image tags corresponding to the target dimension.

[0173] The target weight acquisition unit is specifically used for:

[0174] The target dimension weight is obtained by calculating the query frequency of entity profile tags under the target dimension, the inverse of the number of shards of the dimension shards corresponding to the target dimension, and the proportion of entity profile tags corresponding to the target dimension.

[0175] In one feasible implementation, the device further includes:

[0176] The adjustment instruction generation unit is used to generate a portrait storage adjustment instruction based on the total number of dimension fragments corresponding to the multiple configuration dimensions, the number of target dimension fragments corresponding to the target dimension, and the total number of dimension fragments in the multiple configuration dimensions excluding the target dimension.

[0177] The request total number obtaining unit is used to obtain the predicted total number of requests corresponding to the target dimension in a second time period based on the total number of query requests corresponding to the target dimension in a first time period, wherein the first time period is earlier than the second time period;

[0178] The pressure value acquisition unit is used to calculate and process the total number of prediction requests corresponding to the target dimension during the second time period, the total number of query requests corresponding to the target dimension during the second time period, the preset maximum number of requests corresponding to a single dimension shard in the target dimension, the number of target dimension shards corresponding to the target dimension, and the uniform value of image storage to obtain the dimension pressure value, wherein the uniform value of image storage represents the degree of uneven storage of entity images in the dimension shards of the target dimension.

[0179] The adjustment instruction execution unit is used to execute the image storage adjustment instruction based on the dimensional pressure value.

[0180] In one feasible implementation, the image storage adjustment instruction includes an expansion instruction, and the adjustment instruction execution unit is specifically used for:

[0181] If the dimensional pressure value is greater than the preset pressure value, and the dimensional pressure value is greater than the preset expansion pressure value, then execute the expansion command;

[0182] Executing the expansion instruction includes:

[0183] For each entity in the target dimension, if the dimension sharding position of the entity in the target dimension is different from the target dimension sharding position, and if the entity's entity profile label is not a new entity profile label, then the entity profile label of the entity is stored in the target dimension sharding position of the entity in the target dimension.

[0184] For each entity in the target dimension, if the entity's dimension sharding position in the target dimension is different from the target dimension sharding position, and if the entity's entity profile label is a new entity profile label, then the entity profile label of the entity is stored in the entity's dimension sharding position in the target dimension and the target dimension sharding position.

[0185] In one feasible implementation, the image storage adjustment instruction includes a scaling down instruction, and the adjustment instruction execution unit is specifically used for:

[0186] If the dimensional pressure value is not greater than the preset pressure value, execute the reduction command;

[0187] Executing the shrinkage instruction includes:

[0188] For each entity in the target dimension, if the dimension sharding position of the entity in the target dimension is different from the target dimension sharding position, the entity profile tag of the entity is stored in the target dimension sharding position of the entity in the target dimension.

[0189] The entity image processing apparatus provided in this application embodiment has the same beneficial effects as the entity image processing method provided in the above embodiment, and can improve the processing performance of entity images, which will not be described in detail here.

[0190] This application also provides a computer device. The computer device includes:

[0191] A memory that stores computer programs.

[0192] A processor is used to execute a computer program in memory to implement some or all of the steps in the entity portrait processing method described in the foregoing embodiments.

[0193] This application also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the program implements some or all of the steps in the entity image processing method described in the foregoing embodiments.

[0194] It should be noted that the "first" and "second" in the names such as "first" and "second" (if they exist) mentioned in the embodiments of this application are only used as name identifiers and do not represent the first and second in order.

[0195] It should also be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and equipment embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0196] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing entity images, characterized in that, include: Obtain multiple preset configuration dimensions, as well as multiple entities to be processed and shard memory upper limits, wherein the shard memory upper limits include the upper limit of memory for a single dimension shard, and the dimension shard is used to store entity profile tags; For each configuration dimension, the dimension weight is calculated based on the tag information in that configuration dimension. The tag information is related to determining that the entity to be processed belongs to the configuration profile tag under that configuration dimension. For each configuration dimension, the number of dimension shards corresponding to that configuration dimension is calculated based on the dimension weight corresponding to that configuration dimension, the dimension weights corresponding to the multiple configuration dimensions respectively, and the upper limit of the shard memory. For each entity to be processed and each configuration dimension, the dimension fragment position of the entity to be processed under the configuration dimension is obtained by calculation based on the entity identifier corresponding to the entity to be processed, the dimension identifier corresponding to the configuration dimension, and the total number of dimension fragments corresponding to the multiple configuration dimensions. For each entity to be processed and each configuration dimension, based on the entity to be processed, determine the entity profile label of the entity under the configuration dimension, and store the entity profile label to the dimension fragment position of the entity to be processed under the configuration dimension.

2. The method according to claim 1, characterized in that, Each of the aforementioned configuration dimensions includes at least one configuration profile tag, wherein the configuration profile tag is a null value; determining the entity profile tag of the entity to be processed under the configuration dimension based on the entity to be processed includes: Based on the tag calculation rules in the configuration profile tags under this configuration dimension, the entity dataset in the entity to be processed is processed to obtain the profile tag identifier of the entity to be processed under this configuration dimension and the entity profile value corresponding to the profile tag identifier. The image tag identifier and the entity image value corresponding to the image tag identifier are determined as the entity image tag of the entity to be processed in this configuration dimension.

3. The method according to claim 2, characterized in that, After determining the entity profile label of the entity to be processed under the configuration dimension based on the entity to be processed, and storing the entity profile label at the dimension fragment position of the entity to be processed under the configuration dimension, the method further includes: If the query frequency of entity profile tags under the target dimension is greater than the preset query frequency, or if the execution frequency of tag calculation rules in the configuration profile tags under the target dimension is greater than the preset execution frequency, the target dimension weight corresponding to the target dimension is obtained based on the target dimension, wherein the target dimension is any one of the plurality of configuration dimensions; The weight judgment interval is obtained by calculating the interval threshold and the dimension weights corresponding to the multiple configuration dimensions respectively. The weight judgment interval includes a first judgment weight and a second judgment weight, and the first judgment weight is less than the second judgment weight. If the target dimension weight is less than the first judgment weight, or if the target dimension weight is greater than the second judgment weight, the target dimension weight is calculated based on the interval threshold, the number of dimension fragments corresponding to the target dimension, the target dimension weight corresponding to the target dimension, and the dimension weights corresponding to the multiple configuration dimensions, to obtain the number of target dimension fragments corresponding to the target dimension. For each entity in the target dimension, the target dimension fragment position of the entity is obtained by calculating based on the entity identifier corresponding to the entity, the dimension identifier corresponding to the target dimension, the number of target dimension fragments corresponding to the target dimension, the balance value corresponding to the target dimension, and the total number of dimension fragments in the multiple configuration dimensions excluding the target dimension.

4. The method according to claim 3, characterized in that, Before obtaining the target dimension weight corresponding to the target dimension based on the target dimension, the method further includes: Obtain the number of entity profile tags corresponding to each of the multiple configuration dimensions; Based on the number of dimension shards corresponding to the target dimension, obtain the reciprocal of the number of shards corresponding to the target dimension; and based on the number of entity profile tags corresponding to the multiple configuration dimensions and the number of entity profile tags corresponding to the target dimension, obtain the proportion of entity profile tags corresponding to the target dimension. The step of obtaining the target dimension weight corresponding to the target dimension based on the target dimension includes: The target dimension weight is obtained by calculating the query frequency of entity profile tags under the target dimension, the inverse of the number of shards of the dimension shards corresponding to the target dimension, and the proportion of entity profile tags corresponding to the target dimension.

5. The method according to claim 3, characterized in that, After calculating the target dimension fragment position of the entity under the target dimension based on the entity identifier corresponding to the entity, the dimension identifier corresponding to the target dimension, the number of target dimension fragments corresponding to the target dimension, the balance value corresponding to the target dimension, and the total number of dimension fragments excluding the target dimension among the multiple configuration dimensions, the method further includes: Based on the total number of dimension fragments corresponding to the multiple configuration dimensions, the number of target dimension fragments corresponding to the target dimension, and the total number of dimension fragments in the multiple configuration dimensions excluding the target dimension, a portrait storage adjustment instruction is generated. Based on the total number of query requests corresponding to the target dimension in the first time period, the total number of predicted requests corresponding to the target dimension in the second time period is obtained, wherein the first time period is earlier than the second time period; The dimension pressure value is obtained by calculating the total number of prediction requests corresponding to the target dimension during the second time period, the total number of query requests corresponding to the target dimension during the second time period, the preset maximum number of requests corresponding to a single dimension shard in the target dimension, the number of target dimension shards corresponding to the target dimension, and the uniform value of image storage. The uniform value of image storage represents the degree of uneven storage of entity images in the dimension shards of the target dimension. Based on the dimensional pressure value, execute the image storage adjustment instruction.

6. The method according to claim 5, characterized in that, The image storage adjustment instruction includes an expansion instruction, and executing the image storage adjustment instruction based on the dimensional pressure value includes: If the dimensional pressure value is greater than the preset pressure value, and the dimensional pressure value is greater than the preset expansion pressure value, then execute the expansion command; Executing the expansion instruction includes: For each entity in the target dimension, if the dimension sharding position of the entity in the target dimension is different from the target dimension sharding position, and if the entity's entity profile label is not a new entity profile label, then the entity profile label of the entity is stored in the target dimension sharding position of the entity in the target dimension. For each entity in the target dimension, if the entity's dimension sharding position in the target dimension is different from the target dimension sharding position, and if the entity's entity profile label is a new entity profile label, then the entity profile label of the entity is stored in the entity's dimension sharding position in the target dimension and the target dimension sharding position.

7. The method according to claim 5, characterized in that, The image storage adjustment instruction includes a scaling down instruction, and executing the image storage adjustment instruction based on the dimensional pressure value includes: If the dimensional pressure value is not greater than the preset pressure value, execute the reduction command; Executing the shrinkage instruction includes: For each entity in the target dimension, if the dimension sharding position of the entity in the target dimension is different from the target dimension sharding position, the entity profile tag of the entity is stored in the target dimension sharding position of the entity in the target dimension.

8. A physical image processing device, characterized in that, include: The dimension entity acquisition unit is used to acquire multiple preset configuration dimensions, as well as multiple entities to be processed and the upper limit value of the shard memory, wherein the upper limit value of the shard memory includes the upper limit value of the memory of a single dimension shard, and the dimension shard is used to store entity profile tags. The dimension weight acquisition unit is used to calculate and process the label information in each configuration dimension to obtain the dimension weight corresponding to the configuration dimension, wherein the label information is related to determining that the entity to be processed belongs to the configuration profile label under the configuration dimension. The sharding quantity acquisition unit is used to calculate the sharding quantity of each configuration dimension based on the dimension weight corresponding to the configuration dimension, the dimension weights corresponding to the multiple configuration dimensions respectively, and the sharding memory limit value. The sharding position acquisition unit is used to calculate and process, for each entity to be processed and each configuration dimension, based on the entity identifier corresponding to the entity to be processed, the dimension identifier corresponding to the configuration dimension, and the total number of dimension shards corresponding to the multiple configuration dimensions, to obtain the dimension sharding position of the entity to be processed under the configuration dimension. The image tag storage unit is used to determine the entity image tag of each entity to be processed under the configuration dimension based on the entity to be processed, and to store the entity image tag at the dimension fragment position of the entity to be processed under the configuration dimension.

9. A computer device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the entity image processing method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the entity portrait processing method according to any one of claims 1-7.

Images