Data processing method, device, readable storage medium and program product

Abstract

Embodiments of the present application provide a data processing method, device, readable storage medium and program product. The method comprises: obtaining, according to a first instruction, a value of a first source field of to-be-processed data from a source table; performing a hash algorithm on the value of the first source field to obtain a hash value, and determining a first data block to which the to-be-processed data belongs according to the hash value; and sending the to-be-processed data to a summary node corresponding to the first data block, so that the summary node summarizes a plurality of to-be-processed data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, readable storage medium, and program product. Background Technology

[0002] In big data and high-concurrency digital business scenarios, data such as payment and transfer transactions are growing exponentially, with daily data processing volumes reaching hundreds of millions or even billions. This data, as raw data, can be stored in raw data tables.

[0003] In some application scenarios, not only are raw data details needed, but also statistical analysis of the raw data from different dimensions is required to generate structured summary reports, providing accurate data support for risk verification, data analysis, and related subsequent decision-making.

[0004] Currently, in the process of summarizing massive amounts of raw data, there are problems such as long processing time and low resource utilization from the summarization process to writing the summarization results into the database. Summary of the Invention

[0005] This application provides a data processing method, device, readable storage medium, and program product to improve the problems of long processing time and low resource utilization in the process of summarizing massive amounts of raw data and writing the summarization results to the database.

[0006] In a first aspect, embodiments of this application provide a data processing method, including:

[0007] According to the first instruction, retrieve the value of the first source field of the data to be processed from the source table;

[0008] A hash algorithm is performed on the value of the first source field to obtain a hash value, and the first data block to which the data to be processed belongs is determined based on the hash value;

[0009] The data to be processed is sent to the aggregation node corresponding to the first data block, so that the aggregation node can aggregate multiple data to be processed.

[0010] Secondly, embodiments of this application provide a data processing method, including: receiving information of multiple pieces of data to be processed, wherein the information of the data to be processed includes the content of at least some fields of the data to be processed and the identifier of the source table where the data to be processed is located; wherein, a first data block determined by the hash value of the value of the first source field of the data to be processed is sent to a summary node;

[0011] At least one target table and at least one summary field in each target table are determined according to the configuration file. The configuration file includes the identifier of the target table, the identifier of the summary field, and the summary algorithm corresponding to each summary field. The summary algorithm summarizes the contents of some fields of the data to be processed.

[0012] For each summary field, the data to be processed is summarized according to the corresponding summary algorithm, and the summary result is stored in the local storage space.

[0013] In response to the fulfillment of the second preset condition, the summary results in the local storage space are written into the target database for storage.

[0014] Thirdly, embodiments of this application provide a data processing apparatus, including:

[0015] The first acquisition unit is used to acquire the value of the first source field of the data to be processed from the source table according to the first instruction;

[0016] An execution unit is configured to perform a hash algorithm on the value of the first source field to obtain a hash value, and determine the first data block to which the data to be processed belongs based on the hash value;

[0017] The first sending unit is used to send the data to be processed to the aggregation node corresponding to the first data block, so that the aggregation node can aggregate multiple data to be processed.

[0018] Fourthly, embodiments of this application provide a data processing apparatus, including:

[0019] The first receiving unit is used to receive information about multiple pieces of data to be processed. The information about the data to be processed includes the content of at least some fields of the data to be processed and the identifier of the source table where the data to be processed is located. The first data block determined by the hash value of the first source field of the data to be processed is sent to the aggregation node.

[0020] The first determining unit is configured to determine at least one target table and at least one summary field in each target table according to a configuration file. The configuration file includes the identifier of the target table, the identifier of the summary field, and the summary algorithm corresponding to each summary field. The summary algorithm summarizes the contents of some fields of the data to be processed.

[0021] The summarization unit is used to summarize the data to be processed for each summary field according to the corresponding summary algorithm, and store the summary results in the local storage space.

[0022] The writing unit is used to write the summary results in the local storage space into the target database for storage in response to the fulfillment of a second preset condition.

[0023] Fifthly, embodiments of this application provide a data processing device, including: a memory and a processor;

[0024] The memory stores computer-executed instructions;

[0025] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect, the second aspect, and / or various possible implementations of the first and second aspects as described above.

[0026] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect, the second aspect, and / or various possible implementations of the first and second aspects described above.

[0027] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect, the second aspect, and / or various possible implementations of the first and second aspects described above.

[0028] The data processing method, device, readable storage medium, and program product provided in this application retrieve the value of the first source field of the data to be processed from the source table after receiving a first instruction. Using a hash algorithm, a hash value is obtained based on the value of the first source field to determine the first data block to which the data to be processed belongs. The data to be processed is then sent to the aggregation node corresponding to the first data block for aggregation processing. By hashing according to the aggregation dimension, data of the same dimension is accurately pushed to the same aggregation node, avoiding intermediate data transmission across nodes, reducing the storage volume and cost of intermediate data. This allows data to be processed with the same first source field value to belong to the same data block and be transmitted to the same aggregation node. Data of the same aggregation dimension is processed centrally on a single node, reducing communication latency and data synchronization time between nodes. The aggregation node does not need to wait for intermediate data from other nodes and can directly perform aggregation calculations on the locally centralized data of the same dimension, shortening the overall time from aggregation to data entry. Simultaneously, it avoids redundant calculations when different nodes process data of the same dimension, improving the utilization of computing resources and increasing computing efficiency. In addition, the above solution avoids primary key conflicts caused by different summary nodes writing the same value of the first source field as the primary key to the database in parallel when writing summary results to the database. This reduces the time taken for summary nodes to write to the database, thereby improving the overall efficiency of summarizing data and storing it in the database.

[0029] In addition, this scheme distributes data relatively evenly to each data block and aggregation node through hashing, which at least partially solves the problem of some nodes being overloaded due to uneven data splitting in related technologies, and alleviates the processing lag caused by overload to a certain extent. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0031] Figure 1 A schematic diagram of the system architecture in which the data processing method provided in this application is applied;

[0032] Figure 2 A flowchart illustrating the data processing method provided in this application;

[0033] Figure 3 A schematic diagram of some configuration files provided for this application;

[0034] Figure 4 A flowchart illustrating the data processing method provided in this application;

[0035] Figure 5 A schematic diagram of the data processing apparatus provided in this application;

[0036] Figure 6 A schematic diagram of the data processing apparatus provided in this application;

[0037] Figure 7 A schematic diagram of the data processing device provided in this application.

[0038] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0040] In related technologies, solutions for aggregating massive amounts of raw data mainly employ two approaches: preprocessing hash mapping aggregation and multi-level hierarchical aggregation. The preprocessing hash mapping aggregation approach uses mapping rules such as consistent hashing to partition and categorize the raw data according to preset dimensions (e.g., users), distributing the data to designated distributed nodes. During data aggregation, only the corresponding nodes need to be located for local aggregation, eliminating the need to traverse the entire set of data shards. This effectively reduces invalid queries and improves basic aggregation efficiency, making it a commonly used optimization technique in the industry. However, this approach suffers from uneven data sharding and poor dimension adaptability. Fixed hash sharding rules cannot adapt to the data distribution differences in real-world business scenarios, easily leading to data skew. For example, if the data volume of shards corresponding to high-frequency merchants or other business entities is too large, it can cause severe task tailing problems, lengthening the overall aggregation time.

[0041] Multi-level hierarchical aggregation schemes first summarize the original sharded data across multiple dimensions and write it to an intermediate table. Then, based on the intermediate table, aggregation is performed layer by layer, statistical dimensions are simplified, and the data volume is converged to obtain the final summary result. This can effectively reduce the lock waiting time when multiple processes update the same data. However, multi-level hierarchical aggregation schemes suffer from high resource overhead and insufficient concurrency performance. This scheme relies on multiple intermediate tables to complete hierarchical aggregation. Multiple read and write operations to the database will significantly increase disk input / output and network overhead. The intermediate result tables will also consume a large amount of storage resources, resulting in a long overall aggregation chain and high time overhead.

[0042] In summary, both of the above methods suffer from time-consuming and inefficient processes when summarizing massive amounts of raw data and writing the summarized results into the database.

[0043] Figure 1 A schematic diagram of the system architecture in which the data processing method provided in this application is applied is shown, such as... Figure 1 As shown, the system architecture includes a source database cluster 11, a data sharding terminal 12, a data aggregation terminal 13, and a target database cluster 14. The source database cluster 11 can be a distributed database cluster, including multiple sharded database nodes such as sharded database nodes 111, 112, ..., and 11N, used to store business sharded data. The data sharding terminal 12 can read the sharded data from the source database cluster and perform data preprocessing; determine the corresponding aggregation node for the data, and send the data to the data aggregation terminal 13, which can include multiple aggregation nodes (such as aggregation node 1, 2, ..., and aggregation node K). Each aggregation node can aggregate the data allocated to it according to a preset aggregation algorithm and cache the aggregation results. When the aggregation results meet certain conditions, the aggregation results of multiple data entries are written to the target database cluster, which includes databases 141, 142, ..., and 14M. Here, N, M, and K are integers greater than 2.

[0044] Figure 2 A flowchart illustrating the data processing method provided in this application is shown below. Figure 2 As shown, the method includes the following steps:

[0045] S201: In response to the first instruction, retrieve the value of the first source field of the data to be summarized from the source table.

[0046] In some examples, at least a portion of the data processing method is executed by the data sharding end. The data sharding end may include at least one sharding node. A sharding node may be a service processing unit that performs source data reading, hash calculation, and data distribution operations; specifically, it may be a data sharding service instance deployed in a business service cluster. The first instruction may be a trigger signal used to trigger the data aggregation process. The first instruction can be generated based on a scheduled task or issued by the user. In some examples, the aforementioned source table may be a physical table that stores original data records. The aforementioned original business data may include, for example, transaction details data, business flow data, etc. In some examples, the aforementioned source table may be a logical source table storing original business data records. A logical source table is a complete data table viewed uniformly from a business perspective, possessing a unified field structure and business meaning, but not corresponding to a single physical storage medium. The data in the logical source table is distributed across multiple physical storage units; merging the data from all physical storage units yields the complete data of the logical source table. For example, a transaction details table is composed of multiple physical sub-tables in multiple databases at the underlying level; this transaction details table is a logical source table. After the logical source table is split into databases, the data for each part is stored in different database shards. For example, one part of the logical source table's data is stored in the first database, and another part is stored in the second database. In some application scenarios, each database shard can be further divided into multiple sub-tables for storage.

[0047] The data to be processed can be the raw data in the source table that needs to be statistically processed, such as transaction records generated within a certain period of time. The first source field can be a field taken from the source table and used for hash calculation; the first source field can be associated with the summary dimension corresponding to the target table. The first source field and the summary dimension of the target table can have the same name or different names, and the correspondence between them is established through the mapping relationship in the configuration file. For example, if the summary business requirement is to calculate the total transaction amount of each user by user identifier, then the summary dimension is user identifier. The data splitting end can read the target table summary dimension as "user identifier" from the configuration file, and determine the corresponding "user identifier" field in the source table according to the preset mapping relationship. At this time, "user identifier" is the first source field. When the data splitting end reads each transaction record from the source table, it extracts the "user identifier" value of that record, performs a hash calculation on that value, and sends the transaction records with the same user identifier to the same summary node for processing. If the summary business requirement is to calculate the total daily transaction amount of each user by two dimensions: user identifier and transaction date, then the summary dimension is a combination of the user identifier and transaction date fields. The data splitting end can determine the "user ID" field and "transaction date" field in the source table according to the mapping relationship in the configuration file. At this time, "user ID" and "transaction date" together constitute the first source field.

[0048] The data sharding end can extract the "user ID" and "transaction date" values ​​from each transaction record read from the source table. After combining these values, a hash calculation is performed, and transaction records with the same user ID and transaction date are sent to the same aggregation node for processing. In this illustration, the aggregation node groups and aggregates data according to both the user ID and transaction date. Multiple transactions from the same merchant on the same day are accumulated on the same aggregation node, generating a single aggregation record with the user ID and transaction date as the combined primary key. In this way, by configuring different combinations of aggregation dimensions, data aggregation at different granularities can be achieved.

[0049] Upon receiving the first instruction, the data segmentation endpoint can access the source table using one or more segmentation nodes, read one or more records of data to be processed stored within the source table, and extract the value of the first source field contained in each record. For each record of data to be processed, the value of its first source field can be the record content corresponding to the first source field of that data in the source table. The value of the first source field can be used as input data for a hash algorithm to calculate a hash value to group the data to be processed. The value of the first source field can be at least one of the following: text type, numeric type, or encoded identifier type. The value of the first source field includes, but is not limited to: business identity codes such as organization identifiers and user identifiers; classification identifiers such as payment channel codes, etc.

[0050] In some scenarios, the aforementioned source table can be stored in a single database, and the data sharding end can use a sharding node to read the values ​​of the first source field of multiple data to be processed in the source table from that single database.

[0051] In some scenarios, the aforementioned source table can be a logical source table, and the data sharding end can use one or more sharding nodes to read the values ​​of the first source field of multiple data to be processed from multiple sharded databases.

[0052] S202: Perform a hash algorithm on the value of the first source field to obtain a hash value, and determine the first data block to which the data to be processed belongs based on the hash value.

[0053] A hash algorithm is an algorithm used to transform input data of arbitrary length (such as strings, numbers, or binary data) into a fixed-length output value through a specific mathematical function. The output value after passing through the algorithm is usually called a hash value, hash code, or digest value. The same input data will always produce the same hash value after being processed by the same hash algorithm.

[0054] For any piece of data to be processed, the splitting node can invoke a preset hash algorithm, then input the value of the first source field extracted from the current data into the hash algorithm to complete the calculation, and output the corresponding hash value. The splitting node determines the data block to which the current data belongs by matching the hash value with a pre-divided hash interval. The hash algorithm can be of various types, including but not limited to: MD5, SHA, and CRC32 algorithms. The appropriate hash algorithm can be selected according to the specific application scenario.

[0055] For multiple pieces of data to be processed, if the values ​​of their corresponding first source fields are the same, their corresponding hash values ​​are also the same, and therefore, their corresponding data blocks are also the same. Multiple pieces of data with the same first source field value can be aggregated into the same data block, while pieces of data with different first source field values ​​can be aggregated into different data blocks. For illustration, the first source field is "M_id". Assume that a piece of data to be processed read from the source table has the value of the "M" field as "MCHNT_001". The splitting node uses this string "MCHNT_001" as input to the hash algorithm. After calculating the hash value for "MCHNT_001", it outputs a 32-bit integer hash value, for example, 0x7A3B9C1D (decimal representation is 2050705437). The splitting node then takes the modulo of this hash value with the total number of data blocks (assuming 100), resulting in 37, thus obtaining the data block number to which the data to be processed belongs is 37.

[0056] In some implementations, the first source field includes multiple fields of the source table, and a hash value is obtained by performing a hash algorithm on the value of the first source field, including:

[0057] The values ​​of multiple fields are combined, and a hash algorithm is applied to the combined value to obtain the hash value.

[0058] In some examples, the first source field can include multiple fields. In these examples, the splitting node can first combine the values ​​of multiple fields, and then use the combined value as input to the hash algorithm. Illustratively, the splitting node concatenates the values ​​of multiple first source fields in a preset order, inserting a preset delimiter between adjacent values ​​to generate a combined string. The delimiter can be an underscore "_", a vertical bar "|", a comma ",", or any other character that will not be confused with the field values ​​themselves.

[0059] For example, if the first source fields are "M" and "t", and a record has "MCHNT_001" for "M" and "20260721" for "t", the concatenated string is "MCHNT_001_20260721". The splitting node uses this string as input to the hash algorithm.

[0060] Since the first source field includes multiple fields from the source table, the data partitioning end can combine the values ​​of these fields and then execute a hash algorithm, expanding the input dimension of the hash calculation from a single field to a combination of multiple fields. When hashing based on a single field, if the value distribution of that field is uneven, the data may be concentrated in a few data blocks, causing some aggregation nodes to have high loads. However, when hashing by combining the values ​​of multiple fields, even if the value distribution of one field is relatively concentrated, the differences in the values ​​of other fields will make the overall distribution of the combined value relatively dispersed, thus making the data distribution more even across different data blocks and contributing to load balancing among multiple aggregation nodes. Furthermore, this method does not require changing the number of data block partitions or the underlying implementation of the hash algorithm. When it is necessary to adjust the data distribution criteria, only the field combination of the first source field needs to be changed in the configuration file, and the data partitioning end can then execute hash calculations according to the new combination without modifying the code or redeploying the service, reducing the operational costs required to adjust the data distribution strategy. In addition, the granularity of data aggregation can be adjusted by changing the number of combined fields.

[0061] In some implementations, retrieving the value of the first source field of the data to be processed from the source table includes:

[0062] Based on the distributed storage method of the source table, determine the corresponding database sharding and / or table sharding;

[0063] Read the value of the first source field of at least one piece of data to be processed concurrently from the sharded database and / or sharded table.

[0064] Distributed storage refers to the strategy for distributing data in a source table at the physical storage level, including at least one of database sharding and table sharding strategies. Database sharding involves distributing data from a source table across multiple physical databases based on the values ​​of specific fields. For example, data might be distributed across database A and database B based on the hash value of the merchant ID, with database A storing data where the merchant ID hash value falls within a certain range and database B storing data where the hash value falls within another range. Table sharding involves distributing data from a source table across multiple physical sub-tables within the same physical database based on the values ​​of specific fields. For example, data might be distributed across a sub-table for July 1, 2026, and another sub-table for July 2, 2026, based on the transaction date. A database shard refers to a single physical database instance used to store a portion of the logical source table's data in a distributed storage approach. The data in the logical source table can be split into multiple parts, each stored in a separate database shard. The data from all physical database shards is then merged to form the complete dataset of the logical source table. In some cases, different database shards can be deployed on different physical servers. Table partitioning refers to the further splitting of data within a database shard into multiple physical sub-tables. In some cases, a logical source table can be partitioned into multiple sub-tables within a single database shard based on time (e.g., by day, by month) or by the value range of a specific field. For example, the transaction details table in database A can be further partitioned into multiple sub-tables such as "Transaction Details_20260701" and "Transaction Details_20260702" based on the transaction date, with each sub-table constituting a partition.

[0065] Before reading data from the source table, the data sharding end can obtain the distributed storage method information of the source table by reading a preset configuration file or calling a metadata service. This distributed storage method information includes at least whether the source table uses database sharding, the fields used for sharding, and the number of databases. It also includes whether each database is further divided into tables, the fields used for table division, and the identification information of each table. For example, if the source table is a transaction details table, its distributed storage method could be to distribute the data across 8 physical databases at the database level according to the hash value of the merchant ID, and at the table level, distribute the data into one physical sub-table per day according to the transaction date. After learning this distribution method by reading the configuration information, the data sharding end can determine which databases and tables to read data from.

[0066] The data sharding end determines the range of physical storage units involved in the source table based on the obtained distributed storage method information. This range includes the identification information of all database shards that need to be read, as well as the identification information of all tables that need to be read within each database shard.

[0067] If the source table in a distributed storage method uses only database sharding and not table sharding, then the data sharding endpoint determines the data tables in each database shard as the read target. If the source table in a distributed storage method uses both database sharding and table sharding, then the data sharding endpoint determines the corresponding set of sharded tables within each database shard. Taking a transaction details table as an example, if the table is distributed across 8 databases and each database shards into multiple daily tables, then the data sharding endpoint determines all 8 databases and the multiple daily tables within each database that meet the summary time range.

[0068] In some cases, the data sharding end can determine the range of tables to be read based on the time or data range of the aggregation task, without having to read all tables. For example, if the aggregation task only involves data from July 1, 2026 to July 7, 2026, the data sharding end will only determine the tables corresponding to these 7 days as the reading targets, and tables outside this range will not be included in this reading scope.

[0069] The data sharding endpoint can create multiple concurrent read tasks based on the number of databases and tables determined in step S502. Each concurrent read task is responsible for reading data from a table in a database shard or from a table in a database shard. The data sharding endpoint can distribute each concurrent read task to at least one sharding node for parallel execution.

[0070] During concurrent read operations, each read task reads data records from the corresponding database and table shards according to a preset batch read size (e.g., 1000 records per read), and extracts the value of the first source field from each data record. In some examples, the values ​​of the first source fields read by each sharding node through the read tasks are transmitted to the master node at the data sharding end. The master node performs a hash calculation on the values ​​of the first source fields read by all read tasks, and determines and distributes data blocks based on the hash values ​​obtained from the hash calculation. In some examples, each sharding node can perform a hash algorithm on the values ​​of the first source fields read by the read tasks, and determine and distribute data blocks based on the hash values ​​obtained from the hash calculation. The above distribution refers to sending the values ​​of one or more fields used for summary calculations in the data to be processed to the corresponding summary node. It can be understood that the above one or more fields can also include all fields of the data to be processed, depending on the requirements.

[0071] Because the data sharding end determines the database shards and tables based on the distributed storage method of the source table, and concurrently reads the values ​​of the first source field from the determined database shards and tables, the concurrent read mechanism allows data from multiple database shards and tables to be read and processed simultaneously, without needing to read data from each database and table sequentially. When the source table is distributed across multiple physical databases and multiple physical sub-tables, the concurrent read method allows data read tasks to be executed simultaneously on multiple computing resources, reducing the overall read time caused by waiting for data to return from each database and table sequentially. Since the data sharding end determines the read range based on the actual distribution of the source table, the data ranges handled by each concurrent read task do not overlap (e.g., different tasks read different databases or different tables), which prevents tasks from repeatedly reading the same data records, thus avoiding, to some extent, the additional data transfer overhead caused by repeated reads.

[0072] The hash operation described above can aggregate different data to be processed from different sharded databases into the same data block based on the value of the first source field, and then perform aggregation of relevant dimensions.

[0073] Schematic, the data splitting end can determine one or more first source fields in the source table from the summary field of the source table according to a preset mapping relationship. The above mapping relationship can include the association between each summary field of the target table and at least one source field in the source table. Assuming that the summary field of the target table is "User ID", the source field associated with "User ID" can be determined as "M_id" according to the above mapping relationship.

[0074] S203: Send the data to be processed to the aggregation node corresponding to the first data block so that the aggregation node can aggregate multiple data to be processed.

[0075] The data aggregation endpoint may include at least one aggregation node. An aggregation node can be a service processing unit that performs the aggregation of data to be processed; specifically, it can be a data aggregation service instance deployed in a business service cluster. Here, the aggregation node may be the one corresponding to the first data block.

[0076] After the hash calculation and modulo operation described above, the splitting node obtains the data block number of the data to be processed. The data block indicated by the data block number is the first data block corresponding to the data to be processed. The splitting node can determine the summarizing node corresponding to the first data block based on the data block number.

[0077] In some examples, step S203 above includes:

[0078] Based on the first mapping relationship between data blocks and summary nodes, determine the summary node corresponding to the first data block;

[0079] The first mapping relationship includes the mapping relationship between multiple data blocks and their corresponding summary nodes.

[0080] The first mapping relationship described above stores multiple data block numbers and the corresponding summary nodes for each data block number. Each data block number corresponds to one data block. It is understood that different data block numbers can be associated with the same summary node or different summary nodes. In some examples, the first mapping relationship may also store the addresses (network addresses) corresponding to different summary nodes. Illustratively, a splitting node can determine that data block number 37 is associated with summary node A by querying the first mapping relationship. The splitting node can then send the data to be processed belonging to data block number 37 to summary node A according to the address of summary node A.

[0081] In some examples, the summary node can summarize at least one piece of data to be processed according to a preset summary algorithm, which may be pre-configured. Illustratively, the summary node can sum the values ​​of preset fields in multiple pieces of data to be processed belonging to data block number 37, or perform statistics on the total amount of data to be processed in that data block.

[0082] Because the first mapping relationship establishes a correspondence between data blocks and aggregation nodes, after the splitting node calculates the hash value and determines the first data block, it only needs to query this mapping relationship to obtain the address of the corresponding aggregation node, avoiding the overhead of repeatedly calculating the node address each time it distributes data. Since the first mapping relationship is independent of the hash calculation logic, when the aggregation end adds or removes aggregation nodes, it only needs to modify the corresponding entries between the relevant data blocks and aggregation nodes in the mapping relationship, without changing the hash algorithm's calculation method or the data block partitioning rules. This makes the adjustment cost in node scaling scenarios relatively low.

[0083] In this embodiment, after receiving the first instruction, the value of the first source field of the data to be processed is obtained from the source table. A hash algorithm is used to obtain the hash value based on the value of the first source field, determining the first data block to which the data to be processed belongs. The data to be processed is then sent to the aggregation node corresponding to the first data block for aggregation processing. By hashing according to the aggregation dimension, data of the same dimension is accurately pushed to the same aggregation node, avoiding intermediate data transmission across nodes, reducing the storage volume and cost of intermediate data. This allows data to be processed with the same first source field value to belong to the same data block and be transmitted to the same aggregation node. Data of the same aggregation dimension is processed centrally on a single node, reducing communication latency and data synchronization time between nodes. The aggregation node does not need to wait for intermediate data from other nodes and can directly perform aggregation calculations on the locally centralized data of the same dimension, shortening the overall time from aggregation to data entry. Simultaneously, it avoids redundant calculations when different nodes process data of the same dimension, improving the utilization of computing resources and increasing computational efficiency. In addition, the above solution avoids primary key conflicts caused by different summary nodes writing the same value of the first source field as the primary key to the database in parallel when writing summary results to the database. This reduces the time taken for summary nodes to write to the database, thereby improving the overall efficiency of summarizing data and storing it in the database.

[0084] In addition, this scheme distributes data relatively evenly to each data block and aggregation node through hashing, which at least partially solves the problem of some nodes being overloaded due to uneven data splitting in related technologies, and alleviates the processing lag caused by overload to a certain extent.

[0085] In some embodiments, prior to step S201 described above, the method further includes the following steps:

[0086] Get the configuration file;

[0087] The source table and the first source field in the source table associated with the corresponding summary dimension are determined based on the configuration file.

[0088] A configuration file can be a set of parameters stored in a structured format to define data processing rules. Configuration files can be stored in key-value pair format, tabular format, markup language format, or any other machine-readable format. Configuration files can be stored on local disks, distributed configuration centers, or remote configuration servers, and can be dynamically loaded by the sharding node at startup or during operation.

[0089] Before executing the data aggregation task, the sharding node first retrieves the configuration file from the configuration center or configuration storage medium. For example, the sharding node automatically loads the configuration file from the preset path corresponding to the configuration center or configuration storage medium upon startup; or, after receiving the first instruction, the sharding node retrieves the corresponding configuration file from the configuration center according to the configuration file identifier carried in the first instruction; or, after the user modifies the configuration parameters through the management interface, the configuration center generates a configuration file based on the modified configuration parameters and pushes it to the sharding node.

[0090] The configuration file can contain the following correspondences: source table and corresponding target table, summary dimensions in the target table, and the first source field in the source table corresponding to the summary dimension. For example, the configuration file may contain the following correspondence: "Target table summary field A corresponds to source table field X, target table summary field B corresponds to source table field Y". This correspondence means that when data needs to be summarized, the values ​​of field X and field Y in the source table should be extracted as the basis for determining the summary dimension to which the data belongs. The target table refers to the database table used to store the summary results. The target table can contain dimension fields and summary fields used to identify the summary dimensions. Summary dimension fields are used to record the summary dimensions, and summary fields are used to store the summary results. For example, if the business requirement is "summarize transaction amount based on user ID and transaction date", then the "user ID" field and the "transaction date" field in the target table are the dimension fields, which together constitute a set of summary dimensions.

[0091] Specifically, the data splitting endpoint can iterate through each corresponding record in the aforementioned mapping configuration items based on the obtained configuration file. For each corresponding record, it identifies the specified target table summary field and its corresponding source table field. Subsequently, the splitting node determines all these specified source table fields as the first source field. For example, suppose the configuration file contains the following mapping: the target table's "Merchant ID" summary field corresponds to the source table's "mchnt_id" field; the target table's "Transaction Date" summary field corresponds to the source table's "txn_date" field. Then, the splitting node will jointly determine the source table's "mchnt_id" field and "txn_date" field as the first source field.

[0092] It should be noted that the "first" in the first source field is only used to distinguish the multiple different sets of source fields involved in this embodiment, and does not indicate uniqueness in quantity or order. The number of fields included in the first source field depends on the number configured in the corresponding relationship.

[0093] Because the data sharding end retrieves a configuration file before processing data from the source table—a file that records the mapping between the source table identifier, the aggregation dimension, and the first source field—it can automatically determine the source of the data to be read and the fields used for hash calculation by parsing the configuration file, without needing to hardcode this information into the program code. When the source table involved in the aggregation task changes or the fields used for hash calculation need to be adjusted, only the content of the configuration file needs to be modified, without modifying the program code of the data sharding end or redeploying the service, thus reducing maintenance costs caused by changes in business requirements or data source adjustments. Simultaneously, because the configuration file records the correspondence between the aggregation dimension and the first source field, the data sharding end can automatically determine the corresponding hash field according to the grouping method required by the aggregation business. When the aggregation dimension changes (e.g., from aggregation by merchant number to aggregation by region), no modification to the program logic is required; only the mapping between the aggregation dimension and the first source field in the configuration file needs to be adjusted to allow the data sharding end to perform data partitioning operations according to the new dimension, thus improving the flexibility of this solution in dealing with changes in aggregation dimensions.

[0094] In some implementations, the configuration file includes at least one of the following:

[0095] At least one identifier for the target table; the storage address of the target table;

[0096] The identifier of the source table; the storage address of the source table;

[0097] The summary dimension corresponding to the target table;

[0098] The identifier of at least one summary field in the target table;

[0099] Each summary field corresponds to a summary algorithm, and each summary algorithm references at least one first source field in the source table.

[0100] Filtering rules are used to filter at least one piece of data to be processed in the source table.

[0101] The target table identifier refers to the unique identification information used to identify the target table. The target table identifier can be the target table's name (e.g., "Daily User Summary Table"), the target table's number, or a unique code for the target table in the data dictionary. The target table's storage address refers to the access address information of the target database, which includes at least the IP address or domain name of the server hosting the target database, the port number, and the name of the target database. The data splitting endpoint establishes a connection to the target database based on this storage address, thereby writing the summary results into the target table.

[0102] The source table identifier refers to the unique identification information used to identify the source table. The source table identifier can be the name of the source table (e.g., "Transaction Details Table"), the source table number, or a unique code for the source table in the data dictionary. The source table storage address refers to the access address information of the source database, which includes at least the IP address or domain name of the server where the source database resides, the port number, and the name of the source database. The data splitting endpoint establishes a connection with the source database based on this storage address to read the data to be processed.

[0103] The identifier of a summary field refers to the identification information of the data column in the target table used to store the summary calculation results. The identifier of a summary field can be the name or number of the data column, such as the "Total Transaction Amount" column in the target table. The summary algorithm refers to the calculation rules performed on the values ​​of a specific field in the source table. A summary algorithm includes at least one or more of the following: summation, counting, maximum value, minimum value, copying, or writing to a fixed value. Summation involves adding the values ​​of a specified field in each data record; counting involves counting the number of data records involved in the summary; maximum value means finding the maximum value of a specified field in each data record; minimum value means finding the minimum value of a specified field in each data record; and copying involves directly copying the value of a source field to the target field without performing mathematical operations.

[0104] A filtering rule is a set of conditions used to determine whether a single data record should be included in the summary processing. A filtering rule consists of one or more conditional expressions, each containing a field name, a comparison operator, and a reference value. Comparison operators include at least one of the following: greater than, less than, equal to, not equal to, greater than or equal to, and less than or equal to. For example, a filtering rule could be set to "transaction amount greater than 0 and transaction status equal to successful," meaning only data records that simultaneously meet both conditions will be included in the summary processing.

[0105] In some scenarios, the configuration file must include at least one source table identifier and its storage address, one target table identifier and its storage address, at least one summary field identifier, and the summary algorithm corresponding to each summary field. For example, to summarize data from a transaction details table to a merchant daily summary table, only the information for one source table and one target table needs to be configured.

[0106] In some scenarios, the configuration file contains identifiers for at least two source tables and their corresponding storage addresses, as well as an identifier and storage address for a target table. The configuration file records the mapping relationships between fields in each source table and the summary fields in the target table. For example, if data is summarized from both an online transaction table and an offline transaction table to the same overall transaction summary table, and the field structures of the two source tables are different but summarized to the same target table structure, the configuration file will configure the corresponding field mapping relationships and summarization algorithms for each source table.

[0107] In some scenarios, the configuration file contains an identifier for a source table and its storage address, as well as identifiers for at least two target tables and their respective storage addresses. The configuration file configures the mapping relationships between the summary dimensions and summary fields for each target table. For example, if a daily summary table grouped by user and a daily summary table grouped by region are simultaneously summarized from a transaction details table, and the summary dimensions and target fields of the two target tables are different, then the configuration file configures independent summary dimensions and summary algorithms for each target table.

[0108] In multi-source, multi-target scenarios, the configuration file contains identifiers for at least two source tables and their corresponding storage addresses, as well as identifiers for at least two target tables and their corresponding storage addresses. The configuration file records the many-to-many mapping relationships between each source table and each target table. For example, if online transaction tables and offline transaction tables are aggregated into a user daily summary table and a region daily summary table, respectively, the configuration file configures the corresponding aggregation dimension and aggregation algorithm for each pair of source and target tables.

[0109] In filtering scenarios, the configuration file includes filtering rules in addition to the information mentioned above. These filtering rules are used to filter data records during the reading phase. For example, if the filtering rule in the configuration file is set to "transaction type equals consumption", then only data records with the transaction type of consumption will be included in the summary processing. Filtering rules can be configured individually in the configuration file for any of the above scenarios.

[0110] Figure 3 Here is a schematic diagram of a configuration file, such as Figure 3As shown, the configuration file is written in a structured format, consisting of source table definition files, target table definition files, and summary rule files. Specifically, `Src_table.yml` defines the identifier, field structure, field attributes, and data distribution rules of the source table; `Des_table1.yml` and `Des_table2.yml` define the identifier, field structure, field attributes, and data distribution rules of each target table, respectively. All the table definition files use key-value pair format, describing the table's metadata information through `table_usage_index` (the table's unique index identifier), `table_usage_tp` (the table's usage type), and `Cols` (a list of fields, including field names, field types, and primary key identifiers).

[0111] SUM.yml is the core aggregation algorithm configuration file, used to associate source and target tables and define the complete aggregation execution logic. This file references the source table via the `src` field and one or more target tables via the `des` field. It configures field-level aggregation algorithms for each target table (e.g., `col1:sum(src_col1)` means summing the `src_col1` field of the source table and filling it into the `col1` field of the target table). Furthermore, SUM.yml specifies the aggregation dimension and first source field used for hash calculations via `Hash_cols`, defines data filtering rules via `Filter_CON` (logical OR relationships between filters, and logical AND relationships within the same filter), and sets the maximum number of records in memory cache and the number of records in a single commit via `MaxBuffSize` and `CommitNum`, respectively.

[0112] The aforementioned files together constitute a complete configuration file, which can be stored in the configuration management center or shared storage media and accessed by both the data sharding end and the aggregation node. After configuration, the data sharding end reads SUM.yml to obtain the complete aggregation task definition, including where to read data, how to filter and shard the data, where to send it, what algorithm to use for aggregation, and which target tables to write to. The aggregation node primarily reads the content associated with Des (target table structure and aggregation algorithm) as well as MaxBuffSize / CommitNum. The data sharding end and the aggregation node collaborate to complete the entire aggregation task by reading the same configuration file.

[0113] Because the configuration file contains source table identifiers and storage addresses, target table identifiers and storage addresses, summary field identifiers and corresponding summary algorithms, summary dimensions, and filtering rules, a single configuration file can completely describe all elements of a summary task. By configuring the mapping relationship between multiple source tables and the same target table in the configuration file, data can be read from multiple source tables and summarized to the same target table. Similarly, by configuring multiple target tables and their corresponding summary dimensions and algorithms in the configuration file, summary can be performed separately according to the configuration of each target table after reading the source data once, eliminating the need to repeatedly read the source data multiple times. This configuration-based approach means that when switching between different summary scenarios, only the information combination in the configuration file needs to be adjusted, without modifying the code or redeploying the service, thus improving the adaptability of this solution to various summary scenarios. Furthermore, because the configuration file contains filtering rules, data records are filtered according to the filtering rules during the data reading phase. Data records that do not meet the conditions will not be transmitted to the summary node, reducing network transmission overhead and the computational resource consumption of the summary node. Moreover, when the filtering conditions change, only the configuration file needs to be modified, not the code, further reducing the maintenance complexity when adjusting the filtering conditions.

[0114] In some examples, the configuration file is set by the user through a visual configuration interface; other methods include:

[0115] In response to an update operation on the configuration file, update one or more of the first source field, summary dimension, and summary algorithm in the configuration file.

[0116] The data sharding endpoint or configuration management platform presents a visual configuration interface to the user. This interface graphically displays the parameters in the current configuration file. Users can modify one or more of the primary source fields, aggregation dimensions, and aggregation algorithms recorded in the configuration file using the editing functions within the visual configuration interface. For example, users can modify the field name, add a new primary source field entry, or delete an existing primary source field entry in the primary source field input box of the above interface.

[0117] For example, users can modify the values ​​of dimension fields, add new dimension fields, or delete existing dimension fields in the summary dimension configuration area of ​​the interface. Another example is that users can modify the algorithm type corresponding to a summary field in the summary algorithm drop-down menu or input box, such as changing the summation algorithm to a counting algorithm. Users can send the updated configuration parameters to the configuration management center or shared storage medium by clicking the save or submit button in the visual configuration interface.

[0118] The configuration management center or shared storage medium receives updated configuration parameters submitted by the user and updates the corresponding configuration items in the configuration file with the modified values. When performing subsequent aggregation tasks, the data sharding end reads the updated configuration file from the configuration management center or shared storage medium. Since the configuration file has been updated by the user, the data sharding end obtains the latest version of the configuration parameters. The data sharding end performs corresponding data reading, hash calculation, and aggregation processing operations based on the updated first source field, aggregation dimension, or aggregation algorithm.

[0119] Because users can update the primary source field, aggregation dimension, and aggregation algorithm through a visual configuration interface, and the updated configuration parameters are saved to the configuration management center or shared storage medium, the data sharding end can automatically take effect by reading the updated configuration file when executing subsequent tasks, without requiring modification of program code or redeployment of services. This approach reduces the complexity of configuration update operations, allowing users without programming skills to complete configuration adjustments through the interface, while also reducing downtime caused by configuration changes.

[0120] In some embodiments, the method further includes:

[0121] Based on the first association relationship, the data to be processed with the value of the first source field being the first value is sent to the first aggregation node; wherein, the first association relationship is determined based on the following steps:

[0122] A first value is determined based on the value of the first source field corresponding to each of the multiple historical pending data read from the source table; the number of historical pending data corresponding to the first value satisfies a first preset condition.

[0123] Based on the first data block corresponding to the historical hash value of the first value, a first association relationship is established between the first value and the first summary node; wherein, the first summary node is the summary node corresponding to the first data block.

[0124] The first association refers to the correspondence between a specific value of the first source field and the aggregation node. This association is used to instruct the data splitting end to directly send the data to the aggregation node corresponding to that specific value when the first source field value of the data to be processed equals a certain specific value during subsequent data distribution. The first association is stored in the local storage space or shared storage medium of the data splitting end in the form of a mapping table.

[0125] The first value refers to a specific value among multiple different values ​​of the first source field appearing in the historical unprocessed data, where the number of historical unprocessed data corresponding to this specific value meets a first preset condition. The first preset condition can mean that the number of data corresponding to this value ranks among the top of all different values ​​(e.g., top 10), or it can mean that the number of data corresponding to this value exceeds a preset threshold (e.g., more than 1 million records). For example, if the first source field is the "user identifier" field, and the number of data records corresponding to "user A" in the historical unprocessed data is 5 million, ranking first among all users, then "merchant A" can be determined as the first value.

[0126] Historical pending data refers to data records that have been read and processed from the source table by the data splitting end before the current aggregation task is executed. Historical pending data can be data read in the previous batch of aggregation tasks, or data read in previous batches of aggregation tasks.

[0127] Historical hash value refers to the hash calculation result obtained by the data splitting end after performing a hash algorithm on the value of the first source field during the historical data processing process.

[0128] The first summary node refers to the summary node corresponding to the first data block mentioned above.

[0129] When executing the historical summary task, the data splitting end reads each data record from the source table, extracts the value of the first source field from that record, and increments the cumulative count of that field value in its local counter. After the historical summary task is completed, the data splitting end obtains the number of historical data to be processed corresponding to each different value of the first source field.

[0130] For example, when the first source field is the "user identifier" field, the data splitting end counts that there are 5 million historical data records corresponding to "user A", 3 million historical data records corresponding to "user B", and 1 million historical data records corresponding to "user C".

[0131] The data segmentation end sorts or filters the above statistical results and determines the first value as the F different values ​​of the number of historical pending data. F can be configured by the user or the configuration system. For example, when F=10 is configured, the merchant IDs of the top 10 data items are determined as the first value; or the different values ​​of the number of historical pending data exceeding a preset threshold are determined as the first value. For example, when the threshold is configured as 1 million, all user identifiers with more than 1 million data items are determined as the first value.

[0132] For example, if "User A" has 5 million data entries, "User B" has 3 million data entries, and "User C" has 1 million data entries, then the top two, "User A" and "User B," can be designated as the first value. Alternatively, "User A," with over 2 million data entries, can be designated as the first value.

[0133] The data splitting end can query the historical hash value corresponding to the first value from the historical processing records. Since the value of the same first source field produces the same result in each hash calculation, the data splitting end can obtain the historical hash value by querying the historical hash record or by re-performing the hash calculation at the current moment.

[0134] The data partitioning end obtains the data block division rules, which indicate the correspondence between hash value ranges and data block identifiers. For example, hash values ​​from 0 to 1000 correspond to data block 01, hash values ​​from 1001 to 2000 correspond to data block 02, and so on.

[0135] The data splitting end matches the historical hash value of the first value with the data block partitioning rule to determine the hash value range to which the historical hash value belongs, and determines the data block corresponding to the hash value range as the first data block.

[0136] The data segmentation end performs a hash algorithm on the merchant ID value of "User A" to obtain the historical hash value "12345". According to the data block partitioning rules, the hash value "12345" falls within the hash value range of data block "05". Therefore, the data segmentation end determines the first data block as data block "05".

[0137] The data splitting end obtains the first mapping relationship between data blocks and summary nodes. This first mapping relationship indicates the address of the summary node corresponding to each data block. The data splitting end can use the determined first data block identifier as an index to query this first mapping relationship, obtain the summary node corresponding to the first data block, and determine the summary node as the first summary node.

[0138] The data splitting end queries the first mapping relationship between data blocks and summary nodes, determines that the summary node corresponding to data block "05" is "summary node 3", and then determines "summary node 3" as the first summary node.

[0139] The data splitting end will associate and store the first value (e.g., "User A") determined in the above process with the first aggregation node (e.g., "Aggregation Node 3"), forming an association entry. This association entry records the correspondence between a specific value of the first source field and the aggregation node to which it should be sent.

[0140] The first association relationship can include multiple association relationship entries. Each association relationship entry can be stored in the form of key-value pairs, such as the key being the identifier of the first value and the value being the address information of the first summary node. For example, an association relationship entry can be recorded as "User A corresponds to summary node 3".

[0141] If multiple first values ​​are determined through historical data to be processed, the data splitting end establishes a relationship entry between each first value and the corresponding summary node.

[0142] The data splitting endpoint can store the established initial association in local storage or shared storage. When performing subsequent data distribution operations, for each piece of data to be processed, the endpoint first checks if the value of the first source field of that data exists in the first association. If it does, the endpoint directly sends the data to the corresponding aggregation node according to the association entry recorded in the first association record; if it does not exist, the endpoint determines the data block to which the data belongs and the corresponding aggregation node using the default hash routing method.

[0143] By determining the first value based on the data volume corresponding to different values ​​of the first source field in the historical data to be processed, and establishing a first association between the first value and the summary node corresponding to the first data block, this mechanism allows subsequent data to be processed corresponding to a specific value with a large data volume to bypass hash calculations and data block lookups, and instead be directly sent to the corresponding summary node according to the first association. This method reduces the number of hash calculations and mapping queries during data distribution. For a specific value with a large data volume that meets the first preset condition, each data record can skip the hash calculation step, reducing the computational overhead at the data splitting end to some extent. Furthermore, when it is necessary to adjust the summary node corresponding to a specific first value, only the mapping entry corresponding to that first value in the first association needs to be modified, without modifying the hash algorithm or data block partitioning rules, thus reducing the operational complexity required to adjust the data routing strategy. In addition, the first association is established based on historical data statistical results, and its establishment process does not depend on the real-time analysis of the current batch of data. It can be used directly when the current batch of data begins processing, reducing the preparation time before data distribution to some extent.

[0144] In some implementations, the method further includes:

[0145] In response to the second aggregation node satisfying the first preset condition, at least a portion of the data to be processed belonging to the second data block is sent to the third aggregation node; wherein, in the first mapping relationship between data blocks and aggregation nodes, the first data block is mapped to the first aggregation node.

[0146] The second aggregation node here is any aggregation node that meets the first preset condition. The first preset condition is used to determine whether the aggregation node is overloaded or has insufficient processing capacity. The specific content of the first preset condition can be configured by the user. In some cases, the first preset condition includes that the data processing latency of the aggregation node is greater than or equal to a preset latency threshold; in other cases, the first preset condition includes that the length of the pending data queue of the aggregation node exceeds a preset queue length threshold; in still other cases, the first preset condition includes that the CPU utilization or memory utilization of the aggregation node exceeds a preset resource utilization threshold. The second data block refers to a data block currently mapped to the second aggregation node. In the first mapping relationship between data blocks and aggregation nodes, the second data block currently has a corresponding relationship with the second aggregation node, that is, data records originally belonging to the second data block will be sent to the second aggregation node for processing. The second data block is a data block to be migrated after meeting the first preset condition, and part or all of the pending data of this data block will be rerouted to the third aggregation node. The third aggregation node refers to an aggregation node in the data processing system that serves as the migration target. The third aggregation node differs from the second aggregation node in that it currently does not meet the first preset condition, meaning it is currently in a relatively idle or low-load state. The third aggregation node has the capacity to handle additional data migrated from the second aggregation node.

[0147] During operation, the data sharding endpoint continuously or periodically acquires the operational status information of each aggregation node. In some scenarios, each aggregation node proactively reports its own operational status information to the data sharding endpoint; alternatively, the data sharding endpoint sends a status query request to each aggregation node, which responds by returning operational status information; or, the data sharding endpoint reads the operational status information of each aggregation node from a shared monitoring system or service registry. This operational status information includes at least data processing latency, the length of the pending data queue, and CPU utilization or memory utilization.

[0148] The data splitting end compares the acquired running status information of each aggregation node with the first preset condition. In some cases, if the data processing delay of a certain aggregation node is greater than or equal to a preset delay threshold, the data splitting end determines that the aggregation node meets the first preset condition. For example, if the preset delay threshold is 5 seconds and the current data processing delay of a certain aggregation node is 8 seconds, the data splitting end determines that the aggregation node meets the first preset condition.

[0149] In some cases, if the length of the pending data queue of a certain aggregation node exceeds a preset queue length threshold, the data splitting end determines that the aggregation node meets the first preset condition. For example, if the preset queue length threshold is 100,000 records and the current pending data queue length of a certain aggregation node is 150,000 records, the data splitting end determines that the aggregation node meets the first preset condition. The data splitting end can use one or more aggregation nodes that meet the first preset condition as second aggregation nodes.

[0150] In some implementations, the first preset condition includes one of the following: the number of multiple first historical pending data exceeds the number of multiple second historical pending data, which is greater than or equal to a preset number; wherein, the first historical pending data is pending data historically received by the second aggregation node, and the second historical pending data is pending data historically received by the third aggregation node.

[0151] The number of first historical pending data is greater than the first average, and the number of second historical pending data is less than the first average; the first average is determined by the total amount of third historical pending data processed by multiple summary nodes and multiple summary nodes; the generation time of the first historical pending data and the second historical pending data is earlier than the generation time of the third historical pending data.

[0152] The processing delay of the second aggregation node on the data to be processed is greater than or equal to the preset delay threshold, and the processing delay of the third aggregation node on the data to be processed is less than or equal to the preset delay threshold.

[0153] The preset quantity refers to a threshold used to compare the difference between the historical data processing volumes of two aggregation nodes. This threshold can be determined by system configuration. The preset latency threshold refers to a critical value used to determine whether a latency has occurred in the data processing of the aggregation node. This threshold can be determined by system configuration.

[0154] In some scenarios, the data splitting end obtains the first historical unprocessed data quantity of the second aggregation node and the second historical unprocessed data quantity of the third aggregation node. The data splitting end calculates the difference between the first historical unprocessed data quantity and the second historical unprocessed data quantity, and compares this difference with a preset quantity.

[0155] If the difference is greater than or equal to a preset quantity, the data splitting end determines that the second aggregation node meets the first preset condition. For example, if the preset quantity is set to 100,000 records, and the second aggregation node processes 500,000 records in the first preset time period, while the third aggregation node processes 300,000 records in the same time period, the difference between the two is 200,000 records, which is greater than or equal to the preset quantity of 100,000 records. Therefore, the data splitting end determines that the second aggregation node meets the first preset condition, and the third aggregation node can be used as the migration target node.

[0156] In other scenarios, the data splitting end calculates a first average based on the total amount of third historical data to be processed by multiple aggregation nodes within a third preset time period, combined with the number of aggregation nodes. The first average is calculated by dividing the total amount of third historical data to be processed by multiple aggregation nodes by the number of aggregation nodes.

[0157] The data splitting endpoint obtains the first historical number of pending data items from the second aggregation node and compares this number with a first average. If this number is greater than the first average, the data splitting endpoint determines that the load on the second aggregation node is higher than the system average. The data splitting endpoint also obtains the second historical number of pending data items from the third aggregation node and compares this number with a first average. If this number is less than the first average, the data splitting endpoint determines that the load on the third aggregation node is lower than the system average.

[0158] When the number of first historical data to be processed in the second aggregation node is greater than the first average, and the number of second historical data to be processed in the third aggregation node is less than the first average, the data splitting end determines that the second aggregation node meets the first preset condition.

[0159] For example, the system has 5 summary nodes: A1, B1, C1, D1, and E1. Within a first preset time period, summary node A1 processed 600,000 data entries, summary node B1 processed 500,000, summary node C1 processed 550,000, summary node D1 processed 200,000, and summary node E1 processed 150,000. The first average is 400,000 (2,000,000 ÷ 5). The data splitting endpoint determines that summary nodes A1 (600,000 entries) and B1 (500,000 entries) are both greater than the first average, while summary nodes D1 (200,000 entries) and E1 (150,000 entries) are both less than the first average. Therefore, the data splitting endpoint determines that summary node A1 or summary node B1 meets the first preset condition and can be used as the second summary node; summary node D1 or summary node E1 can be used as the third summary node.

[0160] In some cases, the first and second historical data to be processed were generated earlier than the third historical data to be processed. For example, the first and second historical data to be processed are taken from a first preset time period (such as the previous summary batch), while the third historical data to be processed is taken from a second preset time period (such as the batch before that). In this case, the calculation of the first mean can be based on the historical data of the earlier batch, while the benchmark for judging the differences between nodes is still based on the historical data of the most recent batch.

[0161] In other scenarios, the data splitting end obtains the processing delay data of the second and third aggregation nodes. Processing delay refers to the time elapsed from when the aggregation node receives the data to be processed to when it completes data processing.

[0162] The data splitting end compares the processing latency of the second aggregation node with a preset latency threshold. If the processing latency of the second aggregation node is greater than or equal to the preset latency threshold, the data splitting end determines that the second aggregation node is in a delayed state. The data splitting end compares the processing latency of the third aggregation node with a preset latency threshold. If the processing latency of the third aggregation node is less than the preset latency threshold, the data splitting end determines that the third aggregation node is in a normal state.

[0163] When the processing delay of the second aggregation node is greater than or equal to the preset delay threshold, and the processing delay of the third aggregation node is less than the preset delay threshold, the data splitting end determines that the second aggregation node meets the first preset condition.

[0164] For example, the preset delay threshold is set to 5 seconds. If the current data processing delay of the second aggregation node is 8 seconds (greater than 5 seconds) and the current data processing delay of the third aggregation node is 2 seconds (less than 5 seconds), then the data splitting end determines that the second aggregation node meets the first preset condition.

[0165] After the data splitting end determines that the second aggregation node meets the first preset condition according to at least one of the above methods, the data splitting end performs a data migration operation, changes the mapping target of at least one data block belonging to the second aggregation node to the third aggregation node, and sends the data to be processed belonging to the data block to the third aggregation node for processing.

[0166] Because the first preset condition includes multiple criteria such as differences in data volume between nodes, differences in node data volume compared to the system average, and processing latency, it allows for flexible determination of whether to trigger migration operations based on different dimensions of operational status data. For example, differences in data volume between nodes can trigger migration when there is a significant load difference between two nodes; differences in node data volume compared to the aforementioned average can trigger migration when individual nodes deviate from the average level; and processing latency can trigger migration when a node experiences a performance bottleneck. This improves the ability to detect various load imbalance scenarios to a certain extent. Furthermore, these multiple judgment methods can be combined, allowing migration to be triggered when any one of the conditions is met on the same node, reducing the likelihood of performance degradation due to load imbalances across multiple nodes.

[0167] After detecting that the second aggregation node meets the first preset condition, the data splitting end determines the data blocks currently mapped to the second aggregation node from the first mapping relationship, and selects at least one data block as the second data block. The data splitting end can select the second data block based on various selection strategies. In some cases, the data splitting end determines all data blocks mapped to the second aggregation node as the second data block. In other cases, the data splitting end selects one or more data blocks with the largest data volume as the second data block based on the size of each data block. In still other cases, the data splitting end selects one or more data blocks with the fastest growth rate as the second data block based on the growth rate of the unprocessed data volume corresponding to each data block.

[0168] To illustrate, if data blocks 01, 02, and 03 in the first mapping relationship are all mapped to the second summary node, where data block 01 corresponds to 500,000 records, data block 02 to 300,000 records, and data block 03 to 200,000 records, and the selection strategy is to select the data block with the largest data volume, then the data splitting end will determine data block 01 as the second data block.

[0169] The data splitting end selects one or more summary nodes from the data aggregation end, excluding the second summary node, as the third summary node. The third summary node should be capable of handling additional data.

[0170] The data splitting end can determine the third aggregation node in the following ways: determine the aggregation node with the lowest current load (e.g., the lowest data processing latency, the shortest queue of pending data, or the lowest resource utilization) as the third aggregation node; or, arbitrarily select one or more aggregation nodes from the aggregation nodes that do not currently meet the first preset condition as the third aggregation node; or, select one or more aggregation nodes from the available aggregation nodes as the third aggregation node according to the preset polling strategy or priority strategy.

[0171] After determining the third aggregation node, the data splitting end obtains the address information of the third aggregation node for subsequent data transmission.

[0172] The data splitting end can modify the first mapping relationship, changing the mapping target of the second data block from the second aggregation node to the third aggregation node.

[0173] The data splitting end performs an update operation on the mapping entry corresponding to the second data block in the first mapping relationship. Before the update, the mapping entry records the correspondence between the second data block and the second summary node; after the update, the mapping entry records the correspondence between the second data block and the third summary node.

[0174] After the first mapping relationship is updated, when the data splitting end encounters data to be processed belonging to the second data block during the subsequent data distribution process, it queries the updated first mapping relationship to obtain the aggregation node corresponding to the data block as the third aggregation node, and then sends the data to be processed to the third aggregation node.

[0175] For pending data belonging to the second data block that was read by the data splitting end before the mapping relationship was updated but has not yet been sent, the data splitting end sends it to the third aggregation node according to the updated first mapping relationship. For pending data belonging to the second data block that was read after the mapping relationship was updated, the data splitting end also sends it to the third aggregation node according to the updated first mapping relationship.

[0176] By remapping some data blocks originally belonging to the second aggregation node to the third aggregation node after detecting that the second aggregation node meets the first preset condition, this mechanism allows for partial migration of one or more data blocks when an aggregation node becomes overloaded. This ensures that the migration scope is controllable to some extent and reduces the amount of data rerouting due to load adjustments. Since the migration operation only requires modifying the mapping entry corresponding to the second data block in the first mapping relationship, without adjusting the hash algorithm or modifying the data block partitioning rules, the migration operation is completed as a configuration change and can be executed during the data splitting and aggregation processes without interrupting ongoing aggregation tasks. Because the amount of data in the migrated second data block may only be a portion of the total data volume handled by the aggregation node, the amount of data the second aggregation node needs to process is reduced after the migration. However, the remaining unmigrated data blocks are still processed by the second aggregation node, which alleviates processing latency and queue length issues to some extent. Meanwhile, the third aggregation node can still process the migrated data normally, achieving load balancing among multiple aggregation nodes to a certain extent.

[0177] In some embodiments, step S201 includes:

[0178] In response to the data to be processed meeting the filtering rules, the value of the first source field of the data to be processed is obtained from the source table.

[0179] When initiating a summary task, the data sharding end reads filtering rules from the configuration file. These filtering rules exist as structured configuration items in the configuration file, containing one or more conditional expressions. The data sharding end can parse the filtering rules into executable conditional judgment logic, storing it in local storage for subsequent judgment on each piece of data to be processed. The data sharding end can extract the field values ​​involved in the filtering rules from the data to be processed read from the source table. For each piece of data to be processed, the data sharding end can compare each extracted field value with each conditional expression in the filtering rule to determine whether each condition is true. The data sharding end can comprehensively determine whether the current data to be processed meets the filtering rules based on the logical combination of the conditional expressions in the filtering rules.

[0180] If multiple conditional expressions in the filtering rule are logical AND relations, the data splitting end will check whether each conditional expression is true one by one. If all of them are true, the current data to be processed is determined to meet the filtering rule. If any condition is false, the current data record is determined not to meet the filtering rule.

[0181] If multiple conditional expressions in the filtering rule are logical OR relational, the data splitting end will determine whether each conditional expression is true one by one. If any condition is true, the data to be processed is determined to meet the filtering rule; if none of them are true, the data to be processed is determined to not meet the filtering rule.

[0182] For example, if the filtering rule is set to "transaction amount greater than 0 and transaction status equal to successful", the data splitting end extracts the "transaction amount" and "transaction status" field values ​​from the current data to be processed. The data splitting end determines whether the transaction amount is greater than 0 and whether the transaction status is equal to "success". If both conditions are true, the data splitting end determines that the current data to be processed meets the filtering rule; if either condition is false, the current data to be processed does not meet the filtering rule.

[0183] In some examples, if the data to be processed does not meet the filtering rules, the data splitter will skip the current data to be processed.

[0184] The data splitting end does not perform the first source field extraction operation, hash calculation operation, and data distribution operation on data to be processed that does not meet the filtering rules. The data splitting end discards the data to be processed and continues to read the next data to be processed from the source table and filter it according to the filtering rules until all data to be processed has been processed.

[0185] Because the data splitting end only retrieves the value of the first source field from the source table when the data to be processed meets the filtering rules, it reduces the computational resources used by the data splitting end and also reduces the number of data records sent to the aggregation node, thus reducing the load on the aggregation node for receiving and processing data to some extent. Since the filtering rule determination occurs before the extraction of the first source field value, data records that do not meet the conditions do not need to be transmitted to the aggregation node, further reducing network bandwidth usage during data transmission.

[0186] In some implementations, the configuration file is also used to configure the summarization dimension, summarization field, and summarization algorithm corresponding to the same data to be processed in multiple target tables respectively; step S201 includes: for each target table, extracting the corresponding first source field value from the data to be processed and hashing it to obtain the first data block corresponding to the target table; step S202 includes: determining the corresponding summarization node based on each first data block, and sending the data to be processed to multiple summarization nodes respectively, so that the data to be processed is summarized at each summarization node based on the summarization algorithm of the summarization field.

[0187] When initiating a summary task, the data sharding endpoint reads a configuration file from the configuration management center or shared storage medium. This configuration file contains the summary dimensions, summary fields, and summary algorithms for the same data to be processed across multiple target tables. For example, the configuration file might contain the following settings: target table A's summary dimension is user identifier, the summary field is total transaction amount, and the summary algorithm is summation; target table B's summary dimension is location code, the summary field is the total transaction amount for that location, and the summary algorithm is summation. Under this configuration, the same data to be processed needs to be summarized separately according to both user identifier and location code dimensions, and written to target table A and target table B respectively. The data sharding endpoint establishes a connection with the source database based on the source table information determined in the configuration file and reads the data to be processed from the source table. This data to be processed can contain multiple fields, each with a corresponding value. The data sharding endpoint traverses each target table recorded in the configuration file. For the currently traversed target table, the data sharding endpoint determines the source table field corresponding to the summary dimension of that target table, and designates that source table field as the first source field for that target table. The data splitting end can extract the field value of the first source field from the data to be processed. If the summary dimension of the target table contains multiple fields, the data splitting end extracts the field values ​​of each of these multiple fields and combines them into a combined value according to a preset method.

[0188] For each target table, the data partitioning end performs a hash algorithm on the extracted first source field value to obtain the hash value corresponding to that target table. Based on this hash value, the first data block corresponding to that target table is determined. When different target tables have different aggregation dimensions, the extracted first source field is different, the calculated hash value is also different, and the determined first data block may also be different.

[0189] For example, for target table A, the data splitting end performs a hash algorithm on the first source field, user identifier "M001", to obtain a hash value H_A, and determines the first data block as data block 01. For target table B, the data splitting end performs a hash algorithm on the location code "R003" to obtain a hash value H_B, and determines the first data block as data block 03.

[0190] The data sharding end obtains the first mapping relationship between data blocks and summary nodes. Using the identifier of each first data block as an index, the data sharding end obtains the summary node corresponding to each first data block from the first mapping relationship. This then determines the summary node corresponding to each different target table.

[0191] The data splitting endpoint sends the same piece of data to be processed to the corresponding aggregation nodes of multiple target tables. Illustratively, the data splitting endpoint can copy the current data to be processed into multiple copies, each corresponding to a target table. For each copy of data, the data splitting endpoint generates a data message, which at least includes the identifier of the target table, the field values ​​related to the aggregation calculation in the data to be processed, and the identification information of the aggregation algorithm.

[0192] For each summary node, the data to be processed is summarized using the summary algorithm corresponding to the summary field of the target table in that summary node, resulting in the summary result corresponding to each summary field of each target table.

[0193] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the data processing method provided in the embodiments of this application, as shown below. Figure 4 As shown, the method includes:

[0194] S401: Receive information about multiple pieces of data to be processed. The information about the data to be processed includes the content of at least some fields of the data to be processed and the identifier of the source table where the data to be processed is located. The first data block determined by the hash value of the first source field of the data to be processed is sent to the aggregation node.

[0195] S402: Determine at least one target table and at least one summary field in each target table according to the configuration file. The configuration file includes the identifier of the target table, the identifier of the summary field, and the summary algorithm corresponding to each summary field. The summary algorithm summarizes the contents of some fields of the data to be processed.

[0196] S403: For each summary field, the data to be processed is summarized according to the corresponding summary algorithm, and the summary results are stored in the local storage space.

[0197] S404: In response to the fulfillment of the second preset condition, the summary results in the local storage space are written to the target database for storage.

[0198] The execution entity of the data processing method can be a data aggregation end, specifically one or more aggregation nodes within that end. Data to be processed is sent to the aggregation node based on its corresponding first data block; the first data block is determined based on the hash value corresponding to the value of the first source field in the data to be processed; the first source field is associated with the aggregation dimension of the target table. Based on the data to be processed, the corresponding first data block is determined, and based on the first data block, the corresponding aggregation node is determined. At least a portion of the data to be processed is sent to the corresponding aggregation node for aggregation. Please refer to [link / reference] for details. Figure 2 The description of the relevant parts of the illustrated embodiment will not be repeated here.

[0199] After the data splitting end completes data reading and hash block partitioning, it sends information about the data to be processed to the corresponding aggregation node. The aggregation node receives this information and parses out the contents of at least some fields of the data to be processed, as well as the identifier of the source table where the data to be processed resides.

[0200] Before reaching the aggregation node, the data to be processed has already been assigned to a first data block by the data splitting end based on the hash value of its first source field. This first data block is then sent to the current aggregation node based on the first mapping relationship between it and the aggregation node. Therefore, the information from multiple pieces of data to be processed received by the aggregation node all belong to the same first data block; that is, the hash values ​​of the first source field of these data pieces fall within the same data block range.

[0201] The aggregation node can receive information about data to be processed from one or more data sharding ends. When the data sharding end includes multiple sharding nodes, each sharding node reads different shards of the source table in parallel and sends the data to be processed to the corresponding aggregation node. The same aggregation node may receive data from at least one sharding node.

[0202] S402: Determine at least one target table and at least one summary field in each target table according to the configuration file; wherein, the configuration file includes the identifier of at least one target table, the identifier of at least one summary field in at least one target table, and the summary algorithm corresponding to each summary field, wherein the summary algorithm is used to summarize the data to be processed in the referenced source table.

[0203] Upon startup or before receiving the first piece of data, the aggregation node retrieves a configuration file from the configuration management center or shared storage medium. The configuration file records the identifiers of the source tables, at least one target table, at least one aggregation field in each target table, and the aggregation algorithm corresponding to each aggregation field. The aggregation algorithm uses the contents of certain fields in the data to be processed as input for calculation. The aggregation node performs a parsing operation on the configuration file to determine the target tables involved in this aggregation task, the aggregation fields to be written to each target table, and the aggregation algorithm corresponding to each aggregation field. The configuration file may record that target table A contains a "Total Transaction Amount" aggregation field and its corresponding summation algorithm, which uses the contents of the "Transaction Amount" field in the data to be processed as input for calculation. After parsing, the aggregation node determines that the summation result needs to be written to the "Total Transaction Amount" aggregation field of target table A.

[0204] For each piece of data to be processed received, the aggregation node performs aggregation calculations according to the aggregation algorithm corresponding to each aggregation field determined in step S402.

[0205] In some implementations, the above-mentioned summarization algorithm includes one or more of the following: the summarization algorithm includes at least one of the following: summing the values ​​of the fields to be summarized in multiple pieces of data to be processed, counting the received data to be processed, taking the maximum or minimum value of the values ​​of the fields to be summarized in multiple pieces of data to be processed, copying, and filling in a fixed value in the summary field.

[0206] The aggregation node extracts the content of the fields to be aggregated from the data to be processed, which are referenced by each aggregation algorithm. For aggregation fields that require summation, the aggregation node extracts the corresponding monetary value and adds it to the intermediate aggregation result corresponding to that aggregation field in local storage. For aggregation fields that require counting, the aggregation node increments the count value. For aggregation fields that require copying, the aggregation node directly fills the extracted values ​​of the fields to be aggregated into the corresponding intermediate aggregation result, etc.

[0207] The summary node maintains intermediate summary results for each summary field under each target table in local storage. The intermediate summary results in local storage are stored in key-value pair format, where the key is the value of the summary dimension (e.g., merchant number) and the value is the current cumulative result of each summary field under that dimension.

[0208] During the aggregation calculation process, the aggregation node can detect whether a second preset condition is met. In some implementations, the second preset condition includes one or more of the following: the number of aggregation results is greater than a preset quantity threshold; and the aggregation of multiple pending data within a preset time period is completed.

[0209] In some scenarios, the second preset condition can be that the number of data records corresponding to the intermediate summary results stored in the local storage space reaches a preset number threshold. For example, the preset number threshold is set to 10,000 records. When 10,000 different dimension grouping results have been accumulated in the local storage space, a write operation is triggered.

[0210] In some scenarios, the second preset condition can be the aggregation of multiple pending data items within a preset time period. For example, after the aggregation node has finished processing the received pending data, it receives a data transmission completion flag from the data splitting node. This flag indicates that all pending data within the preset time period has been transmitted, triggering a write operation. Illustratively, the preset time period could be 24 hours, etc.

[0211] When the second preset condition is met, the aggregation node reads the intermediate aggregation results of each aggregation field stored in the local storage space. Based on the target table identifier and target table structure recorded in the configuration file, the aggregation node constructs each intermediate aggregation result into data records conforming to the target table format. The aggregation node establishes a connection with the target database and writes the constructed data records into the corresponding target tables in the database in batches.

[0212] In these implementations, since the second preset condition includes two triggering scenarios: a cache quantity threshold and a data sending completion flag, the cache threshold mechanism can prevent the intermediate results from growing indefinitely and causing memory overflow, while the data sending completion flag can ensure that the results will not remain in memory after all data is aggregated. The two triggering conditions complement each other and improve the adaptability of the aggregation node to different data volumes to a certain extent.

[0213] Because the aggregation node stores the aggregation results in local storage and only writes them to the target database in batches when the second preset condition is met, this mechanism merges multiple aggregation results into a single batch write operation. Compared to performing a database write for each piece of data processed, this reduces the number of connections established with the target database and the frequency of write operations, thus lowering the overhead of database connection creation and release. Since the configuration file records the identifiers of the target table, the identifiers of the aggregation fields, and the aggregation algorithms corresponding to each aggregation field, the aggregation node can automatically determine the write target and calculation method of the aggregation results by parsing the configuration file. When it is necessary to adjust the aggregation fields or modify the aggregation algorithm, only the configuration file content needs to be modified. After the aggregation node reloads the configuration, it can perform aggregation according to the new configuration, which reduces the code modification and redeployment costs caused by changes in aggregation requirements to a certain extent.

[0214] In some implementations, step S401 includes:

[0215] At least one aggregation node receives its respective set of multiple data entries to be processed.

[0216] Step S403 above includes:

[0217] For each summary node, the summary algorithm corresponding to the target field is used to summarize the multiple received data to be processed, and the summary results are stored in the local storage space.

[0218] The data aggregation end can include multiple aggregation nodes. The source table can include multiple rows of data to be processed. The values ​​of the first source field in these multiple rows of data are not completely the same, their hash values ​​are also not completely the same, and consequently, the first data blocks to which these multiple rows of data belong are also not completely the same. At least one splitting node in the data splitting end can send the multiple rows of data to be processed in the source table to their respective aggregation nodes in parallel.

[0219] Each of the aforementioned aggregation nodes can receive multiple pending data entries. The aggregation node can run the aggregation algorithm corresponding to the target field in the target table, and then use the aggregation algorithm to aggregate the multiple pending data entries. In some examples, the aggregation node can group the multiple pending data entries according to different values ​​of the aggregation dimension, and each group can include at least one pending data entry. For each group, the aggregation node can use the aggregation algorithm corresponding to the aggregation field to aggregate the multiple pending data entries. Illustratively, the first aggregation node S1 can receive multiple pending data entries S11, S12, S13, and S14. The aggregation dimension is the user's total transaction amount. Among the multiple pending data entries, S11 and S12 correspond to user U1, and S12 correspond to user U2. The first data block corresponding to the respective hash values ​​of users U1 and U2 is the same. The first aggregation node's aggregation dimension is the user. Therefore, the multiple pending data entries can be grouped according to users U1 and U2. The first group includes pending data S11 and S12 corresponding to user U1, and the second group includes pending data S13 and S14 corresponding to user U2. The first aggregation node can sum up the transaction amount of the pending data S11 and S12 in the first group to obtain the total transaction amount for user U1. Similarly, the first aggregation node can aggregate the transaction amounts of the pending data S13 and S14 in the second group to obtain the total transaction amount for user U2.

[0220] The first aggregation node can cache the above aggregation results in its local storage space. When the second preset condition is met, it writes the multiple locally cached aggregation results into the database for storage.

[0221] Because multiple aggregation nodes independently receive their respective data to be processed and perform aggregation calculations in parallel, there is no need for aggregation nodes to share memory resources or coordinate calculation progress. Each aggregation node only processes the data it receives, and the data blocks processed by different aggregation nodes do not overlap. This parallel processing method allows the overall data processing capability of the aggregation task to increase accordingly with the number of aggregation nodes. When the data volume is large, compared with the single-node processing method, it can shorten the overall aggregation task processing time to a certain extent. At the same time, since each aggregation node independently completes aggregation calculations and performs batch writes in its own local storage space, there is no need for additional communication or synchronization operations between different aggregation nodes, which reduces the coordination overhead in the parallel processing process to a certain extent.

[0222] Figure 5 A schematic diagram of the data processing apparatus provided in this application is shown below. Figure 5 As shown, the data processing apparatus 50 provided in this embodiment includes:

[0223] The first acquisition unit 501 is used to acquire the value of the first source field of the data to be processed from the source table according to the first instruction;

[0224] Execution unit 502 is used to perform a hash algorithm on the value of the first source field to obtain a hash value, and determine the first data block to which the data to be processed belongs based on the hash value;

[0225] The first sending unit 503 is used to send the data to be processed to the aggregation node corresponding to the first data block, so that the aggregation node can aggregate multiple data to be processed.

[0226] In one possible implementation, the first acquisition unit 501 is further configured to:

[0227] Get the configuration file;

[0228] The source table and the first source field in the source table associated with the corresponding summary dimension are determined based on the configuration file.

[0229] In one possible implementation, the configuration file includes at least one of the following:

[0230] At least one identifier for the target table; the storage address of the target table;

[0231] The identifier of the source table; the storage address of the source table;

[0232] The summary dimension corresponding to the target table; the first source field corresponding to the summary dimension;

[0233] The identifier of at least one summary field in the target table;

[0234] The aggregation algorithm corresponding to each aggregation field, and the identifier of at least one field to be aggregated in the source table referenced by each aggregation algorithm;

[0235] Filtering rules are used to filter at least one piece of data to be processed in the source table.

[0236] In one possible implementation, the first source field includes multiple fields from the source table, and the execution unit 502 is further used for:

[0237] The values ​​of multiple fields are combined, and a hash algorithm is applied to the combined value to obtain the hash value.

[0238] In one possible implementation, the execution unit 502 is further configured to: determine the summary node corresponding to the first data block based on the first mapping relationship between the data block and the summary node;

[0239] The first mapping relationship includes the mapping relationship between multiple data blocks and their corresponding summary nodes.

[0240] In one possible implementation, the first acquisition unit 501 is further configured to: determine the corresponding database shard and / or table shard based on the distributed storage method of the source table;

[0241] Read the value of the first source field of at least one piece of data to be processed concurrently from the sharded database and / or sharded table.

[0242] In one possible implementation, the device 50 further includes a second transmitting unit (not shown in the figure), the second transmitting unit being used for:

[0243] Based on the first association relationship, the data to be processed with the value of the first source field being the first value is sent to the first aggregation node; wherein, the first association relationship is determined by the second sending unit based on the following steps:

[0244] A first value is determined based on the value of the first source field corresponding to each of the multiple historical pending data read from the source table; the number of historical pending data corresponding to the first value satisfies a first preset condition.

[0245] Based on the first data block corresponding to the historical hash value of the first value, a first association relationship is established between the first value and the first summary node; wherein, the first summary node is the summary node corresponding to the first data block.

[0246] In one possible implementation, the device 50 further includes a third transmitting unit (not shown in the figure), which is used for:

[0247] In response to the second aggregation node satisfying the first preset condition, at least a portion of the data to be processed belonging to the second data block is sent to the third aggregation node; wherein, in the first mapping relationship between data blocks and aggregation nodes, the first data block is mapped to the first aggregation node.

[0248] In one possible implementation, the first precondition includes at least one of the following:

[0249] The number of first historical pending data exceeds the number of second historical pending data, which is greater than or equal to a preset number; wherein, the first historical pending data is pending data received by the second aggregation node in the past, and the second historical pending data is pending data received by the third aggregation node in the past.

[0250] The number of first historical pending data is greater than the first average, and the number of second historical pending data is less than the first average; the first average is determined by the total amount of third historical pending data processed by multiple summary nodes and multiple summary nodes; the generation time of the first historical pending data and the second historical pending data is earlier than the generation time of the third historical pending data.

[0251] The processing delay of the second aggregation node on the data to be processed is greater than or equal to the preset delay threshold, and the processing delay of the third aggregation node on the data to be processed is less than or equal to the preset delay threshold.

[0252] In one possible implementation, the first acquisition unit 501 is further configured to: in response to the data to be processed satisfying the filtering rules, acquire the value of the first source field of the data to be processed from the source table.

[0253] In one possible implementation, the configuration file is also used to configure the summary dimensions, summary fields, and summary algorithms for the same data to be processed in multiple target tables, with different summary dimensions for the multiple target tables;

[0254] The execution unit 502 is further configured to: for each target table, extract the corresponding first source field value from the data to be processed and hash it to obtain the first data block corresponding to the target table;

[0255] The first transmitting unit 503 is further used for:

[0256] Based on each first data block, a corresponding summary node is determined, and the data to be processed is sent to multiple summary nodes so that the data to be processed is summarized at each summary node based on the summary algorithm of the summary field.

[0257] In one possible implementation, the configuration file is set by the user through a visual configuration interface; the device 50 also includes a configuration unit (not shown in the figure), which is used for:

[0258] In response to an update operation on the configuration file, update the aggregation dimensions and aggregation algorithm in the configuration file.

[0259] The data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0260] Figure 6 A schematic diagram of the data processing apparatus provided in this application is shown below. Figure 6 As shown, the data processing device 60 provided in this embodiment includes:

[0261] The first receiving unit 601 is used to receive information of multiple pieces of data to be processed. The information of the data to be processed includes the content of at least some fields of the data to be processed and the identifier of the source table where the data to be processed is located. The first data block determined by the hash value of the first source field of the data to be processed is sent to the aggregation node.

[0262] The first determining unit 602 is used to determine at least one target table and at least one summary field in each target table according to the configuration file. The configuration file includes the identifier of the target table, the identifier of the summary field, and the summary algorithm corresponding to each summary field. The summary algorithm summarizes the contents of some fields of the data to be processed.

[0263] The summary unit 603 is used to summarize the data to be processed according to the corresponding summary algorithm for each summary field and store the summary results in the local storage space.

[0264] The writing unit 604 is used to write the summary results in the local storage space to the target database for storage in response to the fulfillment of the second preset condition.

[0265] In one possible implementation, the aggregation algorithm includes at least one of the following:

[0266] Sum the values ​​of a specified field;

[0267] Count the number of records in the data to be processed;

[0268] Copy the value of the specified field;

[0269] Fill the specified field in the target table with a fixed value.

[0270] In one possible implementation, the first receiving unit 601 is further configured to:

[0271] At least one aggregation node receives its respective set of multiple data entries to be processed.

[0272] Summary unit 603 is further used for:

[0273] For each summary node, the summary algorithm corresponding to the target field is used to summarize the multiple received data to be processed, and the summary results are stored in the local storage space.

[0274] In one possible implementation, the second precondition includes one or more of the following:

[0275] The number of aggregated results exceeds a preset threshold;

[0276] Complete the aggregation of multiple pending data within a preset time period.

[0277] In one possible implementation, the summarization algorithm includes at least one of the following: summing the values ​​of the field to be summarized in multiple pieces of data to be processed, counting the received data to be processed, taking the maximum or minimum value of the field to be summarized in multiple pieces of data to be processed, copying, and filling in a fixed value in the summary field.

[0278] Figure 7 A schematic diagram of the data processing device provided in this application. Figure 7 As shown, the data processing device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 70.

[0279] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0280] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0281] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0282] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0283] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0284] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0285] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0286] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0287] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0288] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0289] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0290] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0291] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0292] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0293] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: According to the first instruction, retrieve the value of the first source field of the data to be processed from the source table; A hash algorithm is performed on the value of the first source field to obtain a hash value, and the first data block to which the data to be processed belongs is determined based on the hash value; The data to be processed is sent to the aggregation node corresponding to the first data block, so that the aggregation node can aggregate multiple data to be processed.

2. The method according to claim 1, characterized in that, Before retrieving the value of the first source field of the data to be processed from the source table, the process also includes: Get the configuration file; The source table and the first source field in the source table associated with the corresponding summary dimension are determined based on the configuration file.

3. The method according to claim 2, characterized in that, The configuration file includes at least one of the following: An identifier for at least one target table; the storage address of the target table; The identifier of the source table; the storage address of the source table; The summary dimension corresponding to the target table; the first source field corresponding to the summary dimension; The identifier of at least one summary field in the target table; The summarization algorithm corresponding to each of the summarization fields, and the identifier of at least one field to be summarized in the source table referenced by each summarization algorithm; Filtering rules are used to filter at least one piece of data to be processed in the source table.

4. The method according to claim 2, characterized in that, The first source field includes multiple fields of the source table, and the step of performing a hash algorithm on the value of the first source field to obtain a hash value includes: The values ​​of the multiple fields are combined, and a hash algorithm is applied to the combined value to obtain the hash value.

5. The method of claim 1, wherein, The step of determining the first data block to which the data to be processed belongs based on the hash value includes: Based on the first mapping relationship between data blocks and summary nodes, determine the summary node corresponding to the first data block; The first mapping relationship includes the mapping relationship between multiple data blocks and their corresponding summary nodes.

6. The method of claim 1, wherein, The step of retrieving the value of the first source field of the data to be processed from the source table includes: Based on the distributed storage method of the source table, determine the corresponding database shards and / or table shards for the source table; Read the value of the first source field of at least one piece of data to be processed concurrently from the database and / or table shards.

7. The method according to claim 1 or 6, characterized in that, Also includes: Based on the first association relationship, the data to be processed with the value of the first source field being the first value is sent to the first aggregation node; wherein, the first association relationship is determined based on the following steps: A first value is determined based on the value of the first source field corresponding to each of the multiple historical pending data read from the source table; the number of historical pending data corresponding to the first value satisfies a first preset condition. Based on the first data block corresponding to the historical hash value of the first value, a first association relationship is established between the first value and the first aggregation node; wherein, the first aggregation node is the aggregation node corresponding to the first data block.

8. The method of claim 1, wherein, The method further includes: In response to the second aggregation node satisfying the first preset condition, at least a portion of the data to be processed belonging to the second data block is sent to the third aggregation node; wherein, in the first mapping relationship between data blocks and aggregation nodes, the first data block is mapped to the second aggregation node.

9. The method of claim 8, wherein, The first preset condition includes at least one of the following: The number of first historical pending data exceeds the number of second historical pending data, which is greater than or equal to a preset number; wherein, the first historical pending data is pending data received historically by the second aggregation node, and the second historical pending data is pending data received historically by the third aggregation node; The quantity of the first historical pending data is greater than the first average, and the quantity of the second historical pending data is less than the first average; the first average is determined by the total amount of the third historical pending data processed by the multiple aggregation nodes; the generation time of the first historical pending data and the second historical pending data is earlier than the generation time of the third historical pending data. The processing delay of the second aggregation node on the data to be processed is greater than or equal to a preset delay threshold, and the processing delay of the third aggregation node on the data to be processed is less than or equal to the preset delay threshold.

10. The method of claim 3, wherein, The step of retrieving the value of the first source field of the data to be processed from the source table includes: In response to the data to be processed satisfying the filtering rules, the value of the first source field of the data to be processed is obtained from the source table.

11. The method of claim 3, wherein, The configuration file is also used to configure the summary dimensions, summary fields, and summary algorithms for the same data to be processed in multiple target tables; the summary dimensions of the multiple target tables are different. The step of performing a hash algorithm on the value of the first source field to obtain a hash value, and determining the first data block to which the data to be processed belongs based on the hash value, includes: For each target table, extract the corresponding first source field value from the data to be processed and hash it to obtain the first data block corresponding to the target table; Sending the data to be processed to the aggregation node corresponding to the first data block includes: Based on each first data block, a corresponding aggregation node is determined, and the data to be processed is sent to multiple aggregation nodes so that the data to be processed is aggregated at each aggregation node based on the aggregation algorithm of the aggregation field.

12. The method of claim 2 or 3, wherein, The configuration file is set by the user through a visual configuration interface; the method further includes: In response to an update operation on the configuration file, update the aggregation dimensions and aggregation algorithm in the configuration file.

13. A data processing method, characterized by, include: The system receives information about multiple pieces of data to be processed, the information of which includes the content of at least some fields of the data to be processed and the identifier of the source table where the data to be processed is located; wherein, the first data block determined by the hash value of the first source field of the data to be processed is sent to the aggregation node. At least one target table and at least one summary field in each target table are determined according to the configuration file. The configuration file includes the identifier of the target table, the identifier of the summary field, and the summary algorithm corresponding to each summary field. The summary algorithm summarizes the contents of some fields of the data to be processed. For each summary field, the data to be processed is summarized according to the corresponding summary algorithm, and the summary result is stored in the local storage space. In response to the fulfillment of the second preset condition, the summary results in the local storage space are written into the target database for storage.

14. The method of claim 13, wherein, The aggregation algorithm includes at least one of the following methods: Sum the values ​​of the fields to be summarized in multiple data entries; Count the received data to be processed; Take the maximum or minimum value of the field to be summarized in multiple data entries; copy; Enter fixed values ​​in the summary fields.

15. The method of claim 13, wherein, The information received from multiple pieces of data to be processed includes: At least one aggregation node receives its respective set of multiple data entries to be processed. For each target field, the data to be processed is summarized according to the corresponding summarization algorithm, and the summary result is stored in local storage space, including: For each of the aforementioned summary nodes, the summary algorithm corresponding to the target field is used to summarize the received multiple pieces of data to be processed, and the summary results are stored in the local storage space.

16. The method according to any one of claims 13-15, characterized in that, The second preset condition includes one or more of the following: The number of aggregated results exceeds a preset threshold; Complete the aggregation of multiple pending data within a preset time period.

17. A data processing apparatus, characterized by include: The first acquisition unit is used to acquire the value of the first source field of the data to be processed from the source table according to the first instruction; An execution unit is configured to perform a hash algorithm on the value of the first source field to obtain a hash value, and determine the first data block to which the data to be processed belongs based on the hash value; The first sending unit is used to send the data to be processed to the aggregation node corresponding to the first data block, so that the aggregation node can aggregate multiple data to be processed.

18. A data processing party device, characterized by include: The first receiving unit is used to receive information about multiple pieces of data to be processed. The information about the data to be processed includes the content of at least some fields of the data to be processed and the identifier of the source table where the data to be processed is located. The first data block determined by the hash value of the first source field of the data to be processed is sent to the aggregation node. The first determining unit is configured to determine at least one target table and at least one summary field in each target table according to a configuration file. The configuration file includes the identifier of the target table, the identifier of the summary field, and the summary algorithm corresponding to each summary field. The summary algorithm summarizes the contents of some fields of the data to be processed. The summarization unit is used to summarize the data to be processed for each summary field according to the corresponding summary algorithm, and store the summary results in the local storage space. The writing unit is used to write the summary results in the local storage space into the target database for storage in response to the fulfillment of a second preset condition.

19. An electronic device, comprising: include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-16.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-16.

21. A computer program product, characterised in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-16.

Images