Data aggregation method, related device and medium

Abstract

The disclosed embodiments provide a data aggregation method, related devices, and media. The method determines the data identifier (target data identifier) ​​with the largest number of associated data blocks and the data identifier (second data identifier) ​​with the second largest number of associated data blocks from the data source, and determines the first number through the number of data blocks associated with the target data identifier and the number of data blocks associated with the second data identifier. Thus, the target data identifier that may cause data skew is split and scattered according to the first number, the data blocks originally associated with the target data identifier are divided into a first number of data block groups, and then each data block group is aggregated separately. In this way, the data blocks associated with the target data identifier can be divided into multiple data block groups of more appropriate size (number), and the sizes of the data blocks associated with other data identifiers in the data source are more uniform, avoiding the problem that the number of data blocks after the particularly large data is split still causes data skew.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a data aggregation method, related devices, and media. Background Art

[0002] In the distributed computing process, due to the uneven distribution of data, some nodes take on too many computing tasks, thus affecting the overall computing efficiency and stability. This phenomenon is called data skew.

[0003] To address data skew, prior art typically involves salting and then desalting the affected data to reduce the degree of skew. However, even simple salting and desalting operations can still lead to data skew when dealing with complex or large amounts of data, resulting in slow computational efficiency or program errors. Improving data computation efficiency is a pressing issue. Summary of the Invention

[0004] The embodiments of the present disclosure provide a data aggregation method, related devices, and media, aiming to improve data computing efficiency.

[0005] In a first aspect, an embodiment of the present disclosure provides a data aggregation method, the method comprising:

[0006] The data identifier with the most associated data blocks in the data source is used as the target data identifier;

[0007] determining a first number based on the number of data blocks associated with the target data identifier and the number of data blocks associated with a second data identifier, wherein the second data identifier is a data identifier associated with the second largest number of data blocks in the data source;

[0008] Grouping the data blocks associated with the target data identifier to obtain the first number of data block groups;

[0009] Aggregating each of the data block groups respectively to obtain the first number of new data block groups;

[0010] Re-determine the first number according to a preset condition, and jump to grouping the data blocks associated with the target data identifier to obtain the first number of data block groups;

[0011] When the number of the obtained new data block group is one, the target data identifier is associated with the new data block group to obtain target aggregated data.

[0012] In a second aspect, an embodiment of the present disclosure provides a data aggregation device, including:

[0013] an identifier selection unit, configured to use the data identifier associated with the most data blocks in the data source as the target data identifier;

[0014] a first number determining unit, configured to determine a first number based on the number of data blocks associated with the target data identifier and the number of data blocks associated with a second data identifier, wherein the second data identifier is a data identifier associated with the second largest number of data blocks in the data source;

[0015] a grouping unit, configured to group the data blocks associated with the target data identifier to obtain the first number of data block groups;

[0016] an aggregation calculation unit, configured to aggregate each of the data block groups to obtain the first number of new data block groups;

[0017] a second number determining unit, configured to redetermine the first number according to a preset condition, and jump to grouping the data blocks associated with the target data identifier to obtain the first number of data block groups;

[0018] The aggregation output unit is configured to associate the target data identifier with the new data block group when the number of the obtained new data block group is one, so as to obtain target aggregated data.

[0019] Optionally, the identification selection unit is specifically configured to:

[0020] Obtaining the data identifier associated with the most data blocks and the second data identifier in the data source;

[0021] When the ratio between the number of data blocks corresponding to the data identifier associated with the most data blocks and the number of data blocks corresponding to the second data identifier is greater than a preset threshold, the data identifier associated with the most data blocks is used as the target data identifier.

[0022] Optionally, the first number determining unit is specifically configured to:

[0023] The first number is determined according to a ratio of the number of the data blocks associated with the target data identifier to the number of the data blocks associated with the second data identifier.

[0024] Optionally, the grouping unit is specifically configured to: obtain the first number of different sub-identifiers; combine each of the sub-identifiers with the target data identifier to obtain the first number of sub-target data identifiers; and associate the data block groups with the sub-target data identifiers one by one;

[0025] The aggregation calculation unit is specifically configured to aggregate each of the data block groups through different data partitions based on the sub-target data identifier to obtain the first number of new data block groups.

[0026] Optionally, the aggregation calculation unit is specifically configured to:

[0027] For each of the data block groups, obtaining data features of each data block in the data block group;

[0028] According to the data feature, at least two data blocks belonging to the same category are aggregated to obtain the first number of new data block groups.

[0029] Optionally, the second number determining unit is specifically configured to:

[0030] Obtaining a first data block group having the largest number of data blocks and a second data block group having the second largest number of data blocks from the first number of new data block groups;

[0031] The first number is re-determined according to the number of data blocks in the first data block group and the number of data blocks in the second data block group.

[0032] Optionally, the second number determining unit is specifically configured to:

[0033] Obtaining the number of digits in the first number;

[0034] If the number of digits is greater than one, subtract one from the number of digits, and redetermine the first number based on the number of digits after subtraction;

[0035] When the number of digits is equal to one, the first number is determined to be one.

[0036] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data aggregation method in the first aspect is implemented.

[0037] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer executes the computer program, the data aggregation method as in the first aspect is implemented.

[0038] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, comprising a computer program or computer instructions, wherein the computer program or the computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer program or the computer instructions from the computer-readable storage medium, and the processor executes the computer program or the computer instructions, so that the computer device performs the data aggregation method as in the first aspect.

[0039] In the implementation of the present disclosure, the data identifier (target data identifier) ​​with the largest number of associated data blocks and the data identifier (second data identifier) ​​with the second largest number of associated data blocks are determined from the data source, and the first number is determined based on the number of data blocks associated with the target data identifier and the number of data blocks associated with the second data identifier. Thus, the target data identifier that may cause data skew is split and scattered according to the first number, the data blocks originally associated with the target data identifier are divided into the first number of data block groups, and then each data block group is aggregated separately. In this way, the data blocks associated with the target data identifier can be divided into multiple data block groups with more appropriate sizes (numbers), and the sizes of the data blocks associated with other data identifiers in the data source are more uniform, avoiding the problem that the number of data blocks after the data is split is still large and still causes data skew. After the data block groups obtained after the split and scattered are aggregated separately, the split data block groups need to be reintegrated into the data blocks associated with the target data identifier. In this process, the present disclosure merges the various data block groups by gradually merging. In each merging process, the first number is re-determined to re-divide the first number of data block groups, and each data block group is aggregated separately. The above process is repeated until the data block group is finally merged into one data block group, that is, the split data block group is reintegrated into the data block associated with the target data identifier. By gradually merging, when performing aggregation calculations on complex data or data with a huge amount of data, data skew caused by the huge amount of data caused by the one-time merging of the split data block group can be avoided. The solution disclosed in this disclosure can ensure that when performing distributed aggregation calculations on complex data or large data volumes, data skew will not be caused by the excessive number of data blocks associated with individual data identifiers, thereby improving the efficiency of aggregation calculations.

[0040] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A system architecture diagram for a data aggregation method according to an embodiment of the present disclosure;

[0042] Figure 2 A schematic diagram of a data aggregation process according to an example of the present disclosure;

[0043] Figure 3 is a flow chart of a data aggregation method according to an embodiment of the present disclosure;

[0044] Figure 4 for Figure 3Flowchart of determining target data identification in step 310;

[0045] Figure 5 for Figure 3 Flowchart of associating sub-goal data identification after step 330;

[0046] Figure 6 for Figure 3 Flowchart of respectively aggregating the data block groups in step 340;

[0047] Figure 7 for Figure 3 Flowchart of a first embodiment of redetermining the first number in step 350;

[0048] Figure 8 for Figure 3 A flowchart of a second embodiment of redetermining the first number in step 350;

[0049] Figure 9 is a module diagram of a data aggregation device according to an embodiment of the present disclosure;

[0050] Figure 10 According to the embodiment of the present disclosure Figure 3 The terminal structure diagram of the data aggregation method shown;

[0051] Figure 11 According to the embodiment of the present disclosure Figure 3 The server structure diagram of the data aggregation method shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical methods and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not intended to limit the present disclosure.

[0053] It should be noted that although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown in the flowcharts. The terms "first," "second," and the like in the specification, claims, and drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0054] In the description of the embodiments of the present disclosure, unless otherwise expressly defined, words such as setting, installing, and connecting should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the embodiments of the present disclosure in combination with the specific content of the technical solution. In the embodiments of the present disclosure, words such as "further", "exemplarily" or "optionally" are used to indicate examples, illustrations or descriptions, and should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions. The use of words such as "further", "exemplarily" or "optionally" is intended to present related concepts in a specific way.

[0055] Before further explaining the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are explained. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0056] Data skew: In distributed computing, data is unevenly distributed, causing some nodes or partitions to bear too much computing tasks, thus affecting overall computing efficiency and stability. This phenomenon is common in big data processing frameworks and can significantly degrade system performance.

[0057] Key-value storage is a simple and efficient data storage method that stores data in a database as key-value pairs. Each key is a unique identifier that uniquely identifies a specific value. This storage model is ideal for applications that require fast lookup, update, and delete operations. Due to its high performance, high concurrency, and easy scalability, it is widely used in various distributed systems.

[0058] Salting: For a key-value pair, a random number is inserted to split the key into multiple sub-keys, and the value corresponding to the key is also split into multiple sub-values.

[0059] Salt reduction: This refers to merging multiple sub-keys obtained by salting back into the original key, and integrating the sub-values ​​corresponding to the sub-keys into one value and associating it with the original key.

[0060] To address data skew, existing technologies typically salt and then desalt the skewed data to reduce its severity. However, existing solutions typically only address skewed keys, without a gradual desalting step. This approach can only address data skew in very simple computing scenarios. However, for complex scenarios involving large amounts of data and severe skew in individual keys, current solutions will still experience data skew, leading to slow computational efficiency or program errors. Improving data computation efficiency is a pressing issue.

[0061] Based on this, the embodiments of the present disclosure provide a data aggregation method, related devices and media, which determine the data identifier (target data identifier) ​​with the largest number of associated data blocks and the data identifier (second data identifier) ​​with the second largest number of associated data blocks from the data source, and determine the first number through the number of data blocks associated with the target data identifier and the number of data blocks associated with the second data identifier. Thus, the target data identifier that may cause data skew is split and scattered according to the first number, and the data blocks originally associated with the target data identifier are divided into a first number of data block groups, and then each data block group is aggregated separately. In this way, the data blocks associated with the target data identifier can be divided into multiple data block groups with more appropriate sizes (numbers), and the sizes of the data blocks associated with other data identifiers in the data source are more uniform, avoiding the problem that the number of data blocks after the data is split is particularly large and still causes data skew. After the data block groups obtained after the split and scattered are aggregated separately, it is also necessary to reintegrate the split data block groups into data blocks associated with the target data identifier. In this process, the embodiment of the present disclosure merges each data block group in a step-by-step merging manner. In each merging process, the number of data block groups after the current merge (new first number) is determined based on the data block group with the largest number of data blocks and the data block group with the second largest number of data blocks in the data block groups that were merged and aggregated last time. The first number of data block groups is re-determined based on the new first number, and each data block group is aggregated separately. The above process is repeated until it is finally merged into one data block group, that is, the split data block group is reintegrated into the data block associated with the target data identifier. By step-by-step merging, in the process of performing aggregation calculations on complex data or data with a huge amount of data, data skew caused by the huge amount of data caused by the one-time merging of the split data block groups can be avoided. Through the solution of the present disclosure, it can be ensured that when performing distributed aggregation calculations on complex data or large amounts of data, data skew will not be caused by the excessive number of data blocks associated with individual data identifiers, thereby improving the efficiency of aggregation calculations.

[0062] System architecture and scenario description of the application of the embodiments of the present disclosure

[0063] Figure 1 1 is a system architecture diagram of a data aggregation method according to an embodiment of the present disclosure, which includes an object terminal 140, the Internet 130, a gateway 120, a server 110, and the like.

[0064] The server 110 is a computer system that can provide distributed computing services and data aggregation processing services to the target terminals 140. Compared to the target terminals 140, the server 110 has higher requirements in terms of stability, security, and performance. The server 110 can also be a database that can provide data acquisition services to the target terminals 140. The server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). The server 110 can also communicate with the Internet 130 via wired or wireless means to exchange data.

[0065] Gateway 120, also known as a gateway or protocol converter, implements network interconnection at the transport layer and is a computer system or device that performs a conversion function. A gateway acts as a translator between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent by a client terminal 140 to a server 110 are sent through gateway 120 to the corresponding server 110. Messages sent by a server 110 to a client terminal 140 are also sent through gateway 120 to the corresponding client terminal 140.

[0066] The object terminal 140 is a device used to display the results of distributed computing or data aggregation processing for viewing by the subject. The subject can enter a data set requiring data aggregation processing and / or enter commands into the object terminal 140 to access database data and distribute it to different data partitions for distributed computing. The object terminal 140 includes various forms, such as desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, dedicated terminals, and digital tablets. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices connected via a local area network and sharing a common display device for collaborative operation constitute a terminal. The terminal can also communicate with the Internet 130 via wired or wireless means to exchange data.

[0067] During data aggregation, the server 110 can perform the aggregation, with the target terminal 140 acquiring and displaying the results of the server 110's execution. Alternatively, the server 110 and the target terminal 140 can perform the aggregation together. Alternatively, the server 110 can provide the required data to multiple target terminals 140, which then perform data aggregation calculations as distributed computing nodes. In this case, one server 110 can correspond to multiple computing nodes (data partitions), or one server 110 can correspond to one computing node (data partition).

[0068] The embodiments of the present disclosure can be applied in various scenarios, such as network data analysis scenarios. Figure 2 The data aggregation process is shown in Figure 1. Assume that in this example, network data from various regions needs to be analyzed. The data source collected for analysis is the network data from each region. The key is the region, such as Shanghai, Shenzhen, and Guangzhou, and the value is the specific data of the region (key).

[0069] Find the key with the largest number of values ​​from the data sources for each region and determine if this key has data skew. Then find the key' with the second largest number of values ​​from the data source. Determine the first number based on the ratio of the number of values ​​in key to key'. Assuming the number of values ​​in key is 10,000 times the number of values ​​in key', use 1000 as the first number.

[0070] Reference Figure 2 Execute operation 1: Randomly split a key into 1000 parts. Generate a random number and combine it with the key to form a new identifier, obtaining 1000 sub-keys. For example, split the original key into 0001key, 0002key, 0003key, and finally 1000key. Then, randomly split the original value into 1000 parts and associate them with each of the 1000 sub-keys. This completes the decoupling of the original skewed key-value pairs.

[0071] Perform operation ②: Aggregate the value of each sub-key for the first time. Send keys 0001 to 1000 to different data partitions, and aggregate the value of each sub-key separately.

[0072] Perform operation 3: Randomly split the 1000 subkeys into 100 parts. By generating a random number and combining it with the key to form a new identifier, the number of subkeys is reduced from 1000 to 100. The 100 subkeys are 001key, 002key, 003key, through 100key. The values ​​corresponding to the 1000 subkeys after the first aggregation in operation 2 are randomly split into 100 parts as a whole, and then associated with each of the 100 subkeys one by one. This completes the merging of the 1000 subkeys.

[0073] Perform operation 4: Aggregate the value of each sub-key for the second time. Send keys 001 to 100 to different data partitions and aggregate the value of each sub-key separately.

[0074] Perform operation 5: Randomly split the 100 subkeys into 10 parts. By generating a random number and combining it with the key to form a new identifier, the number of subkeys is reduced from 100 to 10. The 10 subkeys are 01key, 02key, 03key, through 10key. The values ​​corresponding to the 100 subkeys after the second aggregation in operation 4 are randomly split into 10 parts as a whole, and then associated with each of the 10 subkeys one by one. This completes the merging of the 100 subkeys.

[0075] Perform operation 6: Aggregate the value of each sub-key for the third time. Send keys 01 to 10 to different data partitions and aggregate the value of each sub-key separately.

[0076] Perform operation 7: Finally, restore the 10 subkeys to a single key. Combine the values ​​corresponding to the 10 subkeys after the third aggregation in operation 6 and associate them with the key. This completes the restoration of the final key.

[0077] Execute operation ⑧: Aggregate the final key value for the fourth time to the final aggregation result.

[0078] In the above example, the key with the largest number of associated values ​​and the key' with the second largest number of associated values ​​are determined from the data source, and the first number is determined based on the ratio of the number of values ​​between key and key'. The key that may cause data skew is thus split and scattered according to the first number, the value associated with the original key is divided into the first number of sub-values, and each sub-value is then aggregated separately. This allows the value associated with the original key, which has a large amount of data, to be divided into multiple sub-values ​​of more appropriate size (number), making the sizes of the values ​​associated with other keys in the data source more uniform, and avoiding the problem of data skew caused by the number of data blocks after the data is split. After the data block groups obtained after the split and scattered are aggregated separately, the split data block groups need to be reassembled into data blocks associated with the target data identifier. In this process, this example merges the data block groups in a step-by-step manner, and a new first number is re-determined during each merge. The first number of sub-keys is re-determined based on the new first number, and the values ​​corresponding to each sub-key are aggregated separately. The above process is repeated until it is restored to a single key. By gradually merging, when performing aggregation calculations on complex or large data volumes, we can avoid data skew caused by the large amount of data remaining after merging the split data blocks all at once. This example solution ensures that when performing distributed aggregation calculations on complex or large data volumes, data skew caused by the large number of data blocks associated with individual data identifiers is avoided, thereby improving the efficiency of aggregation calculations.

[0079] It should be understood that the above content only illustrates some application scenarios of the present disclosure. The business scenarios to which the present disclosure can be applied may include but are not limited to the specific embodiments listed above.

[0080] General description of the embodiments of the present disclosure

[0081] It should be emphasized that the embodiments of the present disclosure can be applied to a variety of application scenarios, such as various distributed computing scenarios, big data analysis and statistics in various industries, such as regional network data analysis, Internet basic resource statistical analysis and other scenarios. In the related art, the problem of data skew usually occurs in the above scenarios. In order to solve the problem of data skew, it is usually adopted to add salt and then desalt the data with skew problems to reduce the degree of data skew. However, simple salting and desalting operations will still have data skew problems when facing complex data or huge amounts of data, resulting in slow data calculation efficiency or program errors. Some embodiments of the present disclosure provide a data aggregation method, related devices and media, which are intended to improve data calculation efficiency.

[0082] The data aggregation method is a method that aggregates the data before using it for related processing or analysis in scenarios where a large amount of data needs to be applied.

[0083] The data aggregation method of the embodiment of the present disclosure may be executed on a server, or on a terminal, or may be executed partially on a server and partially on a terminal.

[0084] like Figure 3 As shown, according to one embodiment of the present disclosure, the data aggregation method includes:

[0085] Step 310: Use the data identifier associated with the most data blocks in the data source as the target data identifier;

[0086] Step 320: Determine a first number based on the number of data blocks associated with the target data identifier and the number of data blocks associated with the second data identifier, wherein the second data identifier is the data identifier associated with the second largest number of data blocks in the data source;

[0087] Step 330: Group the data blocks associated with the target data identifier to obtain a first number of data block groups;

[0088] Step 340: Aggregate each data block group to obtain a first number of new data block groups;

[0089] Step 350: re-determine the first number according to a preset condition, and jump to grouping the data blocks associated with the target data identifier to obtain a first number of data block groups;

[0090] Step 360: When the number of the obtained new data block group is one, associate the target data identifier with the new data block group to obtain target aggregated data.

[0091] Steps 310 to 330 are briefly described below.

[0092] In step 310 , the data source refers to a collection of multiple data extracted from a database or collected from a target object, which requires data aggregation calculation.

[0093] A data identifier is an identifier used to identify specific data. One data identifier can be associated with multiple data blocks. A data identifier is equivalent to the key in a key-value pair. A data block is the specific data indicated by a data identifier. A data block has only one corresponding data identifier, and a data block is equivalent to the value in a key-value pair.

[0094] The target data identifier is the data identifier with the largest number of associated data blocks in the data source and where data skew will occur. There can be one or more data identifiers with the largest number of associated data blocks.

[0095] In step 320, the second data identifier refers to the data identifier with the second largest number of associated data blocks in the data source. There may be one or more second data identifiers.

[0096] The first number is determined by the number of data blocks associated with the target data identifier and the number of data blocks associated with the second data identifier, and is used to indicate the number of grouped data blocks associated with the target data identifier. The first number can be obtained by converting the ratio between the number of data blocks associated with the target data identifier and the number of data blocks associated with the second data identifier, or can be determined by a multiple of the difference between the number of data blocks associated with the target data identifier and the number of data blocks associated with the second data identifier.

[0097] In step 330, the data blocks associated with the target data identifier refer to the data blocks pointed to by the target data identifier. It should be noted that the new data block groups obtained after grouping and aggregation are still associated with the target data identifier.

[0098] In some examples, the data blocks may be grouped by evenly dividing the data blocks according to their original order to obtain a first number of data block groups having the same number of data blocks. In other examples, the data blocks may be grouped by randomly shuffling the data blocks to obtain a first number of data block groups having the same number of data blocks. In other examples, the number of data blocks in each grouped data block group may vary, but the difference between the maximum and minimum number of data blocks cannot exceed a preset threshold. The preset threshold may be determined based on experience or historical data statistics.

[0099] In step 340, aggregating each data block group separately refers to aggregating the data blocks within each data block group, merging data blocks with similar or identical features. Since the data blocks within each data block group are aggregated, the number of aggregated data block groups remains unchanged and remains the first number.

[0100] In step 350, the preset condition refers to a rule set based on the first number of new data block groups for determining the first number of data block groupings for the next time, which will be described in detail in subsequent embodiments.

[0101] When the new first number is determined and the process jumps to step 330 to regroup the data blocks, the data blocks associated with the target data identifier refer to the sets of data blocks in each group of the first number of new data block groups obtained after the last grouping and aggregation.

[0102] In step 360, when the number of new data block groups reaches one, it indicates that all data aggregation has been completed for the data blocks associated with the target data identifier. At this point, the key in the key-value pair (target aggregated data) is still the target data identifier, while the value corresponds to the data block obtained by performing a single breakup and multiple step-by-step merging using the data aggregation method disclosed herein. The target aggregated data refers to the data after the aggregation calculation is completed for the data blocks associated with the target data identifier.

[0103] In the embodiment of steps 310 to 360 above, the data identifier (target data identifier) ​​with the largest number of associated data blocks and the data identifier (second data identifier) ​​with the second largest number of associated data blocks are determined from the data source, and the first number is determined based on the number of data blocks associated with the target data identifier and the number of data blocks associated with the second data identifier. The target data identifier that may cause data skew is thus split and scattered according to the first number, the data blocks originally associated with the target data identifier are divided into the first number of data block groups, and each data block group is then aggregated separately. This allows the data blocks associated with the target data identifier to be divided into multiple data block groups of more appropriate size (number), and the sizes of the data blocks associated with other data identifiers in the data source are more uniform, thus avoiding the problem that a particularly large number of data blocks still causes data skew after the data is split. After the data block groups obtained after the split and scattered are aggregated separately, the split data block groups need to be reassembled into data blocks associated with the target data identifier. In this process, the embodiment of the present disclosure merges the various data block groups by gradually merging. During each merging process, the first number is re-determined to re-divide the first number of data block groups, and each data block group is aggregated separately. The above process is repeated until it is finally merged into a data block group, that is, the split data block group is gradually reintegrated into the data blocks associated with the target data identifier. By gradually merging, in the process of performing aggregation calculations on complex data or data with a huge amount of data, data skew caused by the huge amount of data caused by the one-time merger of the split data block group can be avoided. Through the solution of the embodiment of the present disclosure, it can be ensured that when performing distributed aggregation calculations on complex data or large data volumes, data skew will not be caused by the excessive number of data blocks associated with individual data identifiers, thereby improving the efficiency of aggregation calculations.

[0104] The above is a general description of steps 310 to 360 . The specific implementation process of steps 310 to 350 will be described in detail below.

[0105] In one embodiment, referring to Figure 4 , step 310 includes:

[0106] Step 410: Obtain the data identifier and the second data identifier associated with the most data blocks in the data source;

[0107] Step 420: When the ratio between the number of data blocks corresponding to the data identifier associated with the most data blocks and the number of data blocks corresponding to the second data identifier is greater than a preset threshold, use the data identifier associated with the most data blocks as the target data identifier.

[0108] In step 410 , the data identifier associated with the most data blocks may be considered as the data identifier corresponding to the largest amount of data in the data source.

[0109] In step 420, the preset threshold is used to indicate whether the difference between the two quantities is too large. If the preset threshold is exceeded, it indicates that the number of data blocks corresponding to the data representation with the most associated data blocks is much larger than the number of data blocks corresponding to the second data identifier, indicating data skew. The preset threshold can be set based on experience or obtained through statistical analysis of historical data.

[0110] The data identifier with the largest number of associated data blocks in the data source does not necessarily experience data skew. In the embodiments of steps 410 and 420 described above, by setting a threshold and comparing it with the ratio between the number of data blocks corresponding to the data identifier with the largest number of associated data blocks and the number of data blocks corresponding to the second data identifier, it is possible to accurately determine whether the data identifier with the largest number of associated data blocks is likely to experience data skew, and the data identifier likely to experience data skew is used as the target data identifier.

[0111] In one embodiment, step 320 includes: determining a first number based on the ratio of the number of data blocks associated with the target data identifier to the number of data blocks associated with the second data identifier. The ratio is obtained by dividing the number of data blocks associated with the target data identifier by the number of data blocks associated with the second data identifier. Exemplarily, the first number can be determined based on the number of digits of the ratio. For example, when the ratio is three digits, the first number is set to 100. Alternatively, the ratio is directly used as the first number. For example, when the ratio is 1000, the first number is 1000. In this embodiment, by determining the first number based on the ratio, the number of groups of data blocks associated with the target data identifier can be better determined based on the size between the target data identifier and other data identifiers in the data source, so that the size of the data blocks associated with the target data identifier and the data blocks associated with other data identifiers in the data source after grouping is more uniform, thereby improving the computational efficiency when performing data aggregation on the data source as a whole.

[0112] In another embodiment, a multiple-number comparison table can be established based on experience. First, the multiple of the difference between the number of data blocks associated with the target data identifier and the number of data blocks associated with the second data identifier is determined. The multiple range to which this multiple belongs is determined from the comparison table. Finally, a corresponding number is found and used as the first number for grouping the data blocks.

[0113] In one embodiment, referring to Figure 5 , after step 330, including:

[0114] Step 510: Obtain a first number of different sub-identifiers;

[0115] Step 520: Combine each sub-identifier with the target data identifier to obtain a first number of sub-target data identifiers;

[0116] Step 530: Associate the data block groups with the sub-target data identifiers one by one.

[0117] Step 340 includes:

[0118] Step 540: Aggregate each data block group through different data partitions based on the sub-target data identifier to obtain a first number of new data block groups.

[0119] In step 510, the sub-identifier refers to an identifier used to combine with the target data identifier to form a new sub-target data identifier. The sub-identifier can be generated by a random number or set according to requirements.

[0120] In step 520, combining the sub-identifier with the target data identifier refers to concatenating the two identifiers to obtain a new identifier.

[0121] For example, assuming that the current first number is 1000, the sub-identifiers generated at this time are 0001, 0002, 0003 to 1000 respectively. Assuming that the target data identifier is key, the final first number of sub-target data identifiers are 0001key, 0002key, 0003key to 1000key respectively.

[0122] In step 530, associating the data block groups with the sub-target data identifiers one by one means establishing a one-to-one association relationship between the first number of data block groups and the first number of sub-target data identifiers. The sub-target data identifiers and data block groups can be randomly assigned. For example, a pair of sub-target data identifiers and data block groups is used as a key, and each data block in the data block group is used as the value of the sub-target data identifier to form a key-value structure.

[0123] In step 540, data partitions refer to computing nodes in distributed computing. Different data partitions may be deployed in different servers or terminals, or a server or terminal may be deployed with several data partitions.

[0124] By associating different data block groups with a sub-target data identifier, multiple data block groups can be simultaneously allocated to different data partitions. The data partitions recognize that the sub-target data identifiers are different, and thus aggregate calculations can be performed on the allocated data block groups.

[0125] In the embodiment of the present application, by grouping the data blocks corresponding to the target data identifier and associating the data block groups with the sub-target data identifiers, multiple key-value pairs with smaller memory usage can be obtained. The values ​​corresponding to each key (sub-target data partition) are then aggregated using different data partitions, thereby improving the efficiency of the aggregation calculation.

[0126] In one embodiment, referring to Figure 6 , step 340 includes:

[0127] Step 610: For each data block group, obtain data features of each data block in the data block group;

[0128] Step 620: Aggregate at least two data blocks belonging to the same category according to data features to obtain a first number of new data block groups.

[0129] In step 610, a data feature refers to an identifier that can indicate the characteristics of a data block. For example, if the data block is a key-value structured data, its data feature can be its identifier key. In another example, semantic recognition can be performed on the data block to extract the corresponding meaning of the data block, and its data feature can be the corresponding semantics.

[0130] In step 620, the category refers to the category for classifying data blocks based on data features. When the data features of two data blocks are the same or similar, the two data blocks can be considered to belong to the same category.

[0131] For example, each data block in the data block group may be clustered by K-means clustering to obtain multiple category clusters, and the data blocks in the same category cluster may be aggregated.

[0132] For example, when the data blocks are in a key-value structure, the data blocks with the same key are grouped together, and the values ​​corresponding to the key are aggregated.

[0133] Exemplarily, a semantic recognition model with preset values ​​is used to perform semantic recognition on each data block in the data block group, and data blocks with the same or similar semantics are aggregated.

[0134] Exemplarily, the Euclidean distance between every two data blocks is calculated, and the data blocks with a distance less than a preset threshold are classified as one class and aggregated.

[0135] In one embodiment, referring to Figure 7 , step 350 includes:

[0136] Step 710: Obtain a first data block group having the largest number of data blocks and a second data block group having the second largest number of data blocks from the first number of new data block groups.

[0137] Step 720: Re-determine the first number based on the number of data blocks in the first data block group and the number of data blocks in the second data block group.

[0138] In this embodiment, after the data blocks associated with the target data identifier are grouped and dispersed for the first time, during each re-grouping of the data blocks, the number of data block groups after the current grouping (a new first number) is determined based on the data block group with the largest number of data blocks and the data block group with the second largest number of data blocks in the new data block groups obtained from the previous grouping and aggregation. By gradually merging and performing aggregation calculations on the data blocks that were dispersed by the first grouping in this manner, when performing aggregation calculations on complex data or data with a huge amount of data, data skew caused by the huge amount of data resulting from the one-time merging of the split data block groups can be avoided, thereby improving the efficiency of the aggregation calculation.

[0139] In another embodiment, step 350 includes:

[0140] Step 810: Obtain the number of digits in the first number;

[0141] Step 820: If the number of digits is greater than one, subtract one from the number of digits, and redetermine the first number based on the number of digits after subtraction.

[0142] Step 830: When the number of digits is equal to one, determine the first number to be one.

[0143] In step 810 , the digit refers to the position occupied by each digit in the first number. For example, if the first number is 1000, the number of digits is 4.

[0144] In step 820, subtracting one from the number of digits in the previous first number means subtracting one from the number of digits in the previous first number. For example, if the previous first number is 1000 and the number of digits is 4, and the number of digits is subtracted by one to 3, then the newly determined first number is a three-digit number, such as 100.

[0145] In step 830, if the number of digits is already 1, it means that the data blocks that were first grouped and scattered have been merged to the final stage. At this time, the new first number is directly set to 1 to complete the final merger.

[0146] In this embodiment, after the data blocks associated with the target data identifier are first grouped and dispersed, each time the data blocks are regrouped, the number of digits in the previous first number is used to determine the number of data block groups after the current grouping (the new first number). In this way, the data blocks that were first grouped and dispersed are gradually merged and aggregated. When performing aggregate calculations on complex data or data with a huge amount of data, data skew caused by the huge amount of data remaining after the split data block groups are merged at one time can be avoided, thereby improving the efficiency of aggregate calculations.

[0147] Description of the apparatus and device of the present disclosure

[0148] It is to be understood that, although the steps in the above-mentioned flowcharts are shown in sequence according to the arrow representations, these steps are not necessarily performed in sequence according to the order represented by the arrows. Unless otherwise specified in the present embodiment, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0149] Figure 9 This is a schematic diagram of the structure of a data aggregation device 900 provided in an embodiment of the present disclosure. The data aggregation device 900 includes:

[0150] An identifier selection unit 910 is configured to use the data identifier associated with the most data blocks in the data source as the target data identifier;

[0151] A first number determining unit 920 is configured to determine a first number based on the number of data blocks associated with the target data identifier and the number of data blocks associated with the second data identifier, wherein the second data identifier is the data identifier associated with the second largest number of data blocks in the data source;

[0152] A grouping unit 930 is configured to group the data blocks associated with the target data identifier to obtain a first number of data block groups;

[0153] an aggregation calculation unit 940, configured to aggregate each data block group to obtain a first number of new data block groups;

[0154] A second number determining unit 950 is configured to redetermine the first number according to a preset condition, and jump to grouping the data blocks associated with the target data identifier to obtain a first number of data block groups;

[0155] The aggregation output unit 960 is configured to associate the target data identifier with the new data block group when the number of the obtained new data block group is one, to obtain target aggregated data.

[0156] Optionally, the identification selection unit 910 is specifically configured to:

[0157] Obtain the data identifier and the second data identifier associated with the most data blocks in the data source;

[0158] When the ratio between the number of data blocks corresponding to the data identifier associated with the most data blocks and the number of data blocks corresponding to the second data identifier is greater than a preset threshold, the data identifier associated with the most data blocks is used as the target data identifier.

[0159] Optionally, the first number determining unit 920 is specifically configured to:

[0160] The first number is determined according to a ratio of the number of data blocks associated with the target data identifier to the number of data blocks associated with the second data identifier.

[0161] Optionally, the grouping unit 930 is specifically configured to: obtain a first number of different sub-identifiers; combine each sub-identifier with a target data identifier to obtain a first number of sub-target data identifiers; associate the data block groups with the sub-target data identifiers one by one;

[0162] The aggregation calculation unit 940 is specifically configured to aggregate each data block group through different data partitions based on the sub-target data identifier to obtain a first number of new data block groups.

[0163] Optionally, the aggregation calculation unit 940 is specifically configured to:

[0164] For each data block group, obtain data features of each data block in the data block group;

[0165] At least two data blocks belonging to the same category are aggregated according to data features to obtain a first number of new data block groups.

[0166] Optionally, the second number determining unit 950 is specifically configured to:

[0167] Obtaining a first data block group having the largest number of data blocks and a second data block group having the second largest number of data blocks from the first number of new data block groups;

[0168] The first number is re-determined according to the number of data blocks in the first data block group and the number of data blocks in the second data block group.

[0169] Optionally, the second number determining unit 950 is specifically configured to:

[0170] Get the number of digits in the first number;

[0171] If the number of digits is greater than one, the number of digits is reduced by one, and the first number is re-determined based on the number of digits after reduction by one;

[0172] When the number of digits is equal to one, the first number is determined to be one.

[0173] Reference Figure 10 , Figure 10 The following is a block diagram of the structure of a terminal for implementing the data aggregation method according to an embodiment of the present disclosure. The terminal includes: a radio frequency (RF) circuit 1010, a memory 1015, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. It will be understood by those skilled in the art that Figure 10 The terminal structure shown does not constitute a limitation on the mobile phone or computer, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0174] The RF circuit 1010 may be used for receiving and sending signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is sent to the processor 1080 for processing. In addition, the designed uplink data is sent to the base station.

[0175] The memory 1015 may be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 1015 .

[0176] The input unit 1030 may be configured to receive input digital or character information and generate key signal input related to the settings and function control of the content terminal. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032 .

[0177] The display unit 1040 may be configured to display input information or provided information and various menus of the content terminal. The display unit 1040 may include a display panel 1041.

[0178] The audio circuit 1060 , the speaker 1061 , and the microphone 1062 may provide an audio interface.

[0179] In this embodiment, the processor 1080 included in the terminal can execute the data aggregation method of the previous embodiment.

[0180] The terminals of the embodiments of the present disclosure include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The embodiments of the present disclosure can be applied to various scenarios, including but not limited to content recommendation, data screening, etc.

[0181] Figure 11 A structural block diagram of part of a server for implementing the data aggregation method of an embodiment of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 1122 (for example, one or more processors) and memories 1132, and one or more storage media 1130 (for example, one or more mass storage devices) for storing application programs 1142 or data 1144. Among them, the memories 1132 and the storage media 1130 may be temporary storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1122 may be configured to communicate with the storage medium 1130 to execute a series of instruction operations in the storage medium 1130 on the server.

[0182] The server may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0183] The central processor 1122 in the server may be configured to execute the data aggregation method according to the embodiment of the present disclosure.

[0184] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store program code, and the program code is used to execute the data aggregation method of each of the aforementioned embodiments.

[0185] The present disclosure also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, so that the computer device implements the above-mentioned data aggregation method.

[0186] The terms "first," "second," "third," "fourth," and the like (if any) in the specification of the present disclosure and the accompanying drawings are used to distinguish between similar contents and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present disclosure described herein, for example, can be implemented in orders other than those illustrated or described herein. In addition, the terms "comprises" and "comprising," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0187] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated content, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and following associated content is in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0188] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to exclude the number itself, and above, below, within, etc. are understood to include the number itself.

[0189] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0190] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0191] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0192] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0193] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0194] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A data aggregation method, characterized in that: The method comprises: The data identifier with the most associated data blocks in the data source is used as the target data identifier; determining a first number based on the number of data blocks associated with the target data identifier and the number of data blocks associated with a second data identifier, wherein the second data identifier is a data identifier associated with the second largest number of data blocks in the data source; Grouping the data blocks associated with the target data identifier to obtain the first number of data block groups; Aggregating each of the data block groups respectively to obtain the first number of new data block groups; Re-determining the first number according to a preset condition, and jumping to grouping the data blocks associated with the target data identifier to obtain the first number of data block groups; wherein, re-determining the first number according to the preset condition includes: during each re-grouping of the data blocks, determining the new first number after the current grouping based on the data block group with the largest number of data blocks and the data block group with the second largest number of data blocks in the new data block groups obtained by the previous grouping and aggregation; When the number of the obtained new data block group is one, the target data identifier is associated with the new data block group to obtain target aggregated data.

2. The method according to claim 1, characterized in that The step of taking the data identifier associated with the most data blocks in the data source as the target data identifier includes: Obtaining the data identifier associated with the most data blocks and the second data identifier in the data source; When the ratio between the number of data blocks corresponding to the data identifier associated with the most data blocks and the number of data blocks corresponding to the second data identifier is greater than a preset threshold, the data identifier associated with the most data blocks is used as the target data identifier.

3. The method according to claim 1, characterized in that The determining the first number according to the number of the data blocks associated with the target data identifier and the number of the data blocks associated with the second data identifier includes: The first number is determined according to a ratio of the number of the data blocks associated with the target data identifier to the number of the data blocks associated with the second data identifier.

4. The method according to claim 1, wherein After grouping the data blocks associated with the target data identifier to obtain the first number of data block groups, the method further includes: obtaining the first number of different sub-identifiers; combining each of the sub-identifiers with the target data identifier to obtain the first number of sub-target data identifiers; and associating the data block groups with the sub-target data identifiers one by one; Then, respectively aggregating each of the data block groups to obtain the first number of new data block groups includes: respectively aggregating each of the data block groups through different data partitions based on the sub-target data identifier to obtain the first number of new data block groups.

5. The method according to claim 1 or 4, characterized in that The step of respectively aggregating each of the data block groups to obtain the first number of new data block groups includes: For each of the data block groups, obtaining data features of each data block in the data block group; According to the data feature, at least two data blocks belonging to the same category are aggregated to obtain the first number of new data block groups.

6. The method according to claim 1, characterized in that The re-determining the first number according to a preset condition includes: Obtaining a first data block group having the largest number of data blocks and a second data block group having the second largest number of data blocks from the first number of new data block groups; The first number is re-determined according to the number of data blocks in the first data block group and the number of data blocks in the second data block group.

7. The method according to claim 1, characterized in that The re-determining the first number according to a preset condition includes: Obtaining the number of digits in the first number; If the number of digits is greater than one, subtract one from the number of digits, and redetermine the first number based on the number of digits after subtraction; When the number of digits is equal to one, the first number is determined to be one.

8. A data aggregation device, characterized in that: include: an identifier selection unit, configured to use the data identifier associated with the most data blocks in the data source as the target data identifier; a first number determining unit, configured to determine a first number based on the number of data blocks associated with the target data identifier and the number of data blocks associated with a second data identifier, wherein the second data identifier is a data identifier associated with the second largest number of data blocks in the data source; a grouping unit, configured to group the data blocks associated with the target data identifier to obtain the first number of data block groups; an aggregation calculation unit, configured to aggregate each of the data block groups to obtain the first number of new data block groups; a second number determining unit configured to redetermine the first number based on a preset condition and jump to grouping the data blocks associated with the target data identifier to obtain the first number of data block groups; wherein, redetermining the first number based on the preset condition comprises: during each regrouping of the data blocks, determining the new first number after the current grouping based on the data block group with the largest number of data blocks and the data block group with the second largest number of data blocks in the new data block groups obtained by the previous grouping and aggregation; The aggregation output unit is configured to associate the target data identifier with the new data block group when the number of the obtained new data block group is one, so as to obtain target aggregated data.

9. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data aggregation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the data aggregation method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program or computer instructions, characterized in that The computer program or the computer instructions are stored in a computer-readable storage medium, and the processor of the computer device reads the computer program or the computer instructions from the computer-readable storage medium. The processor executes the computer program or the computer instructions, so that the computer device performs the data aggregation method according to any one of claims 1 to 7.

Images