Data fragmentation method and device, medium and program product

Abstract

The invention discloses a data fragmentation method and device, a medium and a program product, and relates to the technical field of financial science and technology. The data fragmentation method comprises the following steps: analyzing initial to-be-processed data to obtain fragmentation association features of the initial to-be-processed data; based on the total record number of the initial to-be-processed data and the target parallelism degree, calculating a single job node processing record number; when the current to-be-processed data meets a data fragment permission condition, determining an initial record sequence number of the current data fragment, and determining an end record sequence number of the current data fragment according to the fragment association feature and a single job node processing record number; and determining nodes matched with the current data fragment according to the starting record sequence number and the end record sequence number of the current data fragment, and executing data in parallel. According to the technical scheme provided by the embodiment of the invention, the uniformity of data fragmentation and the balance of node loads can be ensured, so that the data processing efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a data sharding method, device, medium and program product. Background Art

[0002] Existing financial batch job systems generally process all data at one node based on a predetermined data processing range, or split the data range according to fixed rules and then process it in parallel.

[0003] Traditional sharding relies on manual experience or simple rules, resulting in uneven data sharding, which ultimately leads to unbalanced node load and excessively long processing time for some nodes, greatly affecting the data processing efficiency of financial institutions. Summary of the Invention

[0004] The present invention provides a data sharding method, device, medium and program product to solve the problem of uneven sharding in existing data sharding methods, which leads to low data processing efficiency in financial institutions.

[0005] According to one aspect of the present invention, a data sharding method is provided, comprising:

[0006] Analyze the initial data to be processed to obtain shard correlation characteristics of the initial data to be processed;

[0007] Calculate the number of records that can be processed by a single job node based on the total number of records in the initial data to be processed and the target parallelism;

[0008] When the current data to be processed meets the data sharding permission conditions, the starting record sequence number of the current data shard is determined, and the ending record sequence number of the current data shard is determined based on the shard association characteristics and the number of records processed by a single job node;

[0009] According to the starting record sequence number and the ending record sequence number of the current data shard, the node that matches the current data shard is determined to execute the data in parallel.

[0010] According to another aspect of the present invention, there is provided a data sharding device, comprising:

[0011] A data analysis module is used to analyze the initial data to be processed and obtain the shard correlation characteristics of the initial data to be processed;

[0012] A module for calculating the number of records processed by a single job node is used to calculate the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism;

[0013] The module for determining the starting and ending record numbers is used to determine the starting record number of the current data shard when the currently processed data meets the data sharding permission conditions, and to determine the ending record number of the current data shard based on the shard association characteristics and the number of records processed by a single job node;

[0014] The data sharding module is used to determine the nodes that match the current data shard and execute the data in parallel based on the starting record sequence number and the ending record sequence number of the current data shard.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data sharding method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data sharding method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the data sharding method according to any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention obtains the shard association characteristics of the initial data to be processed by analyzing the initial data to be processed, and then calculates the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism. Then, when the current data to be processed meets the data sharding permission conditions, the starting record sequence number of the current data shard is determined, and the ending record sequence number of the current data shard is determined based on the shard association characteristics and the number of records processed by the single job node. Further, based on the starting record sequence number and the ending record sequence number of the current data shard, the node that matches the current data shard is determined to execute the data in parallel. In this solution, the data can be coarse-grainedly sharded based on the total number of records of the initial data to be processed and the target parallelism, and the coarse-grained data shards can be dynamically adjusted according to the correlation between the data through the shard association characteristics. While ensuring the uniformity of data sharding, data correlation is also taken into account to avoid related data being scattered in different data shards, ensuring the reliability of data processing, and solving the problem of uneven sharding in existing data sharding methods, which leads to low data processing efficiency in financial institutions. It can ensure the uniformity of data sharding and the balance of node load to improve data processing efficiency and accuracy.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A flowchart of a data sharding method provided in Example 1 of the present invention;

[0025] Figure 2 A flowchart of a data sharding method provided in Example 2 of the present invention;

[0026] Figure 3 A schematic diagram of data sharding logic provided in the second embodiment of the present invention;

[0027] Figure 4 A schematic diagram of the structure of a data sharding device provided in Embodiment 3 of the present invention;

[0028] Figure 5 A schematic structural diagram of an electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "initial", "target", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Example 1

[0032] Figure 1 This is a flow chart of a data sharding method provided in the first embodiment of the present invention. This embodiment is applicable to the case of uniformly sharding the data of a financial institution. The method can be executed by a data sharding device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. The electronic device may include but is not limited to a server and a computer. Figure 1 As shown, the method includes:

[0033] Step 110: Analyze the initial data to be processed to obtain shard association features of the initial data to be processed.

[0034] The initial data to be processed may be raw data that requires data sharding. The initial data to be processed may be data in a financial institution's business system. Shard association features may be used to reflect the correlation between data in the initial data to be processed. Shard association features may include, but are not limited to, user account numbers, transaction processing times, and transaction numbers.

[0035] In an embodiment of the present invention, initial data to be processed may be obtained first, and then parsed to determine shard association features of the initial data to be processed, so that data with the same shard association features in the initial data to be processed can be divided into one data shard.

[0036] Step 120: Calculate the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism.

[0037] The total number of records can be the total number of data items in the initial data to be processed. The target parallelism can be the preset data execution parallelism. The number of records processed by a single job node can be the number of data items that a single job node needs to process.

[0038] In an embodiment of the present invention, the number of data items of the initial data to be processed can be counted to obtain the corresponding total number of records, and then the quotient of the total number of records and the target parallelism is used as the number of records processed by a single job node.

[0039] Step 130: When the current data to be processed meets the data sharding permission conditions, determine the starting record sequence number of the current data shard, and determine the ending record sequence number of the current data shard based on the sharding association characteristics and the number of records processed by a single job node.

[0040] Among them, the current data to be processed may be part or all of the data in the initial data to be processed, that is, the data in the initial data to be processed that is currently being sharded. The data sharding permission condition is the condition under which the data set can be sharded. The data sharding permission condition may include but is not limited to the number of data shards obtained being less than the number of sharding operations corresponding to the target parallelism, and / or the number of data items currently remaining to be sharded being greater than a certain threshold. The number of sharding times corresponding to the target parallelism is equal to the difference between the target parallelism and 1. The current data shard may be a data set that is first divided from the current data to be processed. The starting record sequence number may be the sequence number of the starting data of the data shard in the initial data to be processed. The ending record sequence number may be the sequence number of the ending data of the data shard in the initial data to be processed.

[0041] In an embodiment of the present invention, before the initial data to be processed is sharded, the data sharding permission conditions are first obtained, and then it is determined whether the current data to be processed meets the data sharding permission conditions. When the current data to be processed meets the data sharding permission conditions, the starting record sequence number of the current data shard is determined, and then the record sequence number of the end data of the current data shard is preliminarily determined based on the starting record sequence number and the number of records processed by a single job node. If the record sequence number of the end data preliminarily determined based on the sharding association characteristics has no correlation with its next adjacent data, the record sequence number of the end data preliminarily determined is used as the end record sequence number of the current data shard. If the record sequence number of the end data preliminarily determined based on the sharding association characteristics has a correlation with its next adjacent data, the end record sequence number of the current data shard is determined based on the sharding association characteristics and the record sequence number of the end data preliminarily determined.

[0042] Optionally, when the current data to be processed is the initial data to be processed, the starting record sequence number of the current data shard is the data sequence number corresponding to the first data in the initial data to be processed. If the current data to be processed is not the initial data to be processed, the sequence number of the next data at the end of the previous data shard of the current data to be processed is used as the starting record sequence number of the current data shard.

[0043] Step 140: Determine the node that matches the current data shard and executes the data in parallel based on the starting record sequence number and the ending record sequence number of the current data shard.

[0044] The data executed in parallel by the node is the data in the current data shard.

[0045] In an embodiment of the present invention, according to the starting record number and the ending record number of the current data shard, the data corresponding to the starting record number and the ending record number, as well as the data between the two record numbers, can be queried from the initial data to be processed, and the queried data can be used as the node parallel execution data matching the current data shard.

[0046] The technical solution of the embodiment of the present invention obtains the shard association characteristics of the initial data to be processed by analyzing the initial data to be processed, and then calculates the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism. Then, when the current data to be processed meets the data sharding permission conditions, the starting record sequence number of the current data shard is determined, and the ending record sequence number of the current data shard is determined based on the shard association characteristics and the number of records processed by the single job node. Further, based on the starting record sequence number and the ending record sequence number of the current data shard, the node that matches the current data shard is determined to execute the data in parallel. In this solution, the data can be coarse-grainedly sharded based on the total number of records of the initial data to be processed and the target parallelism, and the coarse-grained data shards can be dynamically adjusted according to the correlation between the data through the shard association characteristics. While ensuring the uniformity of data sharding, data correlation is also taken into account to avoid related data being scattered in different data shards, ensuring the reliability of data processing, and solving the problem of uneven sharding in existing data sharding methods, which leads to low data processing efficiency in financial institutions. It can ensure the uniformity of data sharding and the balance of node load to improve data processing efficiency and accuracy.

[0047] Example 2

[0048] Figure 2 This is a flowchart of a data sharding method provided in the second embodiment of the present invention. This embodiment is specific based on the above embodiment and provides a specific optional implementation method for analyzing the initial data to be processed and obtaining the sharding correlation characteristics of the initial data to be processed. Figure 2 As shown, the method includes:

[0049] Step 210: Determine the job processing scope of the initial data to be processed.

[0050] The job processing scope may be the business scope corresponding to the initial data to be processed.

[0051] In the embodiment of the present invention, the initial data to be processed may be parsed to determine the business scope corresponding to the initial data to be processed, that is, the job processing scope.

[0052] Step 220: Determine the shard association characteristics of the initial data to be processed based on the job processing scope of the initial data to be processed.

[0053] In an embodiment of the present invention, a shard association feature that matches the job processing scope of the initial data to be processed can be determined based on the job processing scope of the initial data to be processed. Since the shard association feature is strongly associated with the job processing scope, when data is sharded based on the shard association feature, the reliability of business processing can be effectively guaranteed.

[0054] Optionally, when the job processing scope is different, the shard association characteristics matching the job processing scope may be different.

[0055] Step 230: Calculate the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism.

[0056] In an optional embodiment of the present invention, after calculating the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism, it can also include: when the current data to be processed does not meet the data sharding permission conditions, the current data to be processed is not sharded, and the sharding parameters of the current data to be processed are determined.

[0057] The sharding parameter may be a parameter describing the data range corresponding to the data sharding. For example, the sharding parameter may include but is not limited to the starting data of the data sharding and the ordinal number of the ending data corresponding to the initial data to be processed.

[0058] In an embodiment of the present invention, if the current data to be processed does not meet the data sharding permission conditions, it indicates that the current data to be processed is not suitable for re-sharding. The ordinal numbers of the starting data and the ending data of the current data to be processed corresponding to the initial data to be processed can be used as the sharding parameters of the current data to be processed, that is, it is automatically determined whether the data sharding operation needs to be performed without manual intervention. When the data sharding operation is not suitable, the current data to be processed is used as a data shard, and the corresponding sharding parameters are recorded to facilitate data positioning during subsequent maintenance.

[0059] In an optional embodiment of the present invention, before calculating the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism, it can also include: obtaining the historical processing records of the node; training a load prediction model based on the historical processing records of the node to predict the target parallelism through the trained load prediction model.

[0060] The node historical processing record may be a record of the node's historical processing data. The load prediction model may be a model for predicting the node's data processing capability.

[0061] In an embodiment of the present invention, the node that processes the data shard corresponding to the initial data to be processed can be determined, and the node historical processing record of the aforementioned node can be obtained, so as to use the node historical processing record and the parallelism of the node when processing historical data to train the load prediction model, so that the initial data to be processed can be input into the trained load prediction model, and the target parallelism can be predicted by the trained load prediction model. That is, this solution uses the load prediction model to autonomously predict the parallelism of data execution so that the node can achieve the best load distribution.

[0062] Step 240: When the current data to be processed meets the data sharding permission conditions, determine the starting record sequence number of the current data shard, and determine the ending record sequence number of the current data shard based on the sharding association characteristics and the number of records processed by a single job node.

[0063] In an optional embodiment of the present invention, when the current data to be processed meets the data sharding permission condition, determining the starting record sequence number of the current data shard may include: determining the current number of remaining records to be processed; when the difference between the total number of records of the initial data to be processed and the current number of remaining records to be processed is greater than the single shard record number threshold, and the shard sequence number of the current data shard is less than the preset sharding operation number, determining that the current data to be processed meets the data sharding permission condition; when the current data to be processed meets the data sharding permission condition, determining the starting record sequence number of the current data shard based on the end record sequence number of the previous data shard of the current data shard, or the starting record sequence number of the first data shard.

[0064] The current number of remaining records to be processed may be the number of data items in the current data to be processed. The single-shard record count threshold may be a preset upper limit for the number of data items in a single data shard. The preset number of sharding operations may be the number of sharding operations performed on the initial data to be processed. Assuming that the initial data to be processed needs to be divided into five data shards, the number of sharding operations is four, and the preset number of sharding operations is four. The first data shard may be the first data shard divided from the initial data to be processed.

[0065] In an embodiment of the present invention, the number of data items of the current data to be processed, that is, the number of records remaining to be processed, can be determined first, and the difference between the total number of records of the initial data to be processed and the number of records remaining to be processed can be calculated, and the difference can be compared with the single-shard record number threshold. If the difference is greater than the single-shard record number threshold, and the shard sequence number of the current data shard is less than the preset shard operation number, it is determined that the current data to be processed meets the data sharding permission condition. If the initial data to be processed has not yet been data sharded, the current data to be processed is the initial data to be processed, and the sequence number of the first data in the initial data to be processed, that is, the starting record sequence number of the first data shard, is used as the starting record sequence number of the current data shard. When the current data to be processed meets the data sharding permission condition and the initial data to be processed has been data sharded, the current data to be processed is part of the data in the initial data to be processed, and the end record sequence number of the previous data shard of the current data shard is determined, and the next record sequence number of the end record sequence number of the previous data shard of the current data shard is used as the starting record sequence number of the current data shard. This solution can automatically determine the sequence number of the data shard starting data of the current data to be processed in the initial data to be processed when the current data to be processed meets the data sharding permission conditions, so as to distinguish the current data shard from the previous data shard through the starting record sequence number of the current data shard.

[0066] Step 250: Determine the node that matches the current data shard and executes the data in parallel based on the start record sequence number and the end record sequence number of the current data shard.

[0067] In an optional embodiment of the present invention, the data sharding method may further include: when it is determined that the parallel processing task matching the initial data to be processed has failed to execute, retrying the failed subtask; when the failed subtask fails to retry, obtaining the data resharding feature, and adding the data resharding feature to the sharding association feature; returning to execute the operation of calculating the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism, so as to complete the resharding of the initial data to be processed.

[0068] A parallel processing task can be a task in which multiple nodes process multiple data shards of the initial data to be processed in parallel. A failed subtask can be a task in which a node failed to process the task. The data resharding feature can be a sharding association feature configured by operations personnel for a failed subtask.

[0069] In an embodiment of the present invention, a plurality of nodes execute parallel processing tasks that match the initial data to be processed in parallel. When it is determined that the parallel processing task that matches the initial data to be processed fails to execute, the execution task of the node where the task execution failed, i.e., the failed subtask, is identified, and the failed subtask is retried. Then, when the retry of the failed subtask fails, the data resharding feature is obtained to add the data resharding feature to the sharding association feature, and the operation of calculating the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism is returned, thereby completing the resharding of the initial data to be processed. The failure of the retry of the failed subtask is very likely caused by improper data sharding. Through the data resharding feature and the sharding association feature, the data sharding is adjusted in time to ensure the successful execution of the parallel processing task.

[0070] In an optional embodiment of the present invention, the data sharding method may further include: obtaining parallel processing results when it is determined that the parallel processing task matching the initial data to be processed is successfully executed; and performing consistency checking and data merging processing on the parallel processing results.

[0071] The parallel processing result may be a processing result of a parallel processing task.

[0072] In an embodiment of the present invention, if the parallel processing task matching the initial data to be processed is successfully executed, the parallel processing results of each node are further obtained, and the parallel processing results of all nodes are subjected to consistency verification and data merging processing, so as to ensure the accuracy, integrity and reliability of the data through consistency verification, and ensure the effectiveness of business decisions and data analysis. The parallel processing results of different related nodes are aggregated and merged to improve data quality and decision-making efficiency.

[0073] In a specific example, the data intelligent sharding and parallel processing system can include a data preprocessing module, a parallel execution module, and a monitoring and fault tolerance module. The data preprocessing module analyzes the job processing scope to obtain the characteristics of the initial data to be processed (such as data volume, type, and relevance), and determines the distribution of related data based on the characteristics of the initial data to be processed. The parallel execution module performs data sharding based on the target parallelism and the distribution of related data. The data sharding logic is as follows: Figure 3 The intelligent data sharding and parallel processing system supports multi-threaded / multi-process processing. When the data volume does not meet the requirements for further sharding, the node can run idle. The monitoring and fault tolerance module monitors task status in real time and automatically retries failed subtasks or reshards.

[0074] This solution can adjust the sharding granularity in real time based on data characteristics and resource status. It can also be combined with a load prediction model to predict node processing capabilities to optimize task allocation and further perform consistency verification and merging on the final parallel processing results. Distribute N records as evenly as possible to P shards to balance the load of each shard, maximize processing efficiency, and ensure the uniformity of data sharding; all records of the same account must be assigned to the same shard to avoid consistency errors caused by hot accounts in multiple shards and ensure the consistency of data shards. When the number of shards P changes, minimize the amount of data migration to ensure data stability. After sorting the initial data to be processed, the expected data volume of each shard is N / P. The initial data to be processed is sequentially divided into 1...P shards. If the account numbers of adjacent data are the same, they are divided into the same shard.

[0075] It has been proven that the data sharding method provided by this solution can significantly improve the processing efficiency of the batch operation system. When the degree of parallelism allows, for 500,000 records to be processed, the batch operation system time for the same scenario is reduced from 1 hour to 10 minutes, greatly improving the system's processing efficiency.

[0076] The technical solution of the embodiment of the present invention determines the job processing scope of the initial data to be processed, and thus determines the shard association characteristics of the initial data to be processed according to the job processing scope of the initial data to be processed, and then calculates the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism. Further, when the current data to be processed meets the data sharding permission conditions, the starting record sequence number of the current data shard is determined, and the end record sequence number of the current data shard is determined according to the shard association characteristics and the number of records processed by a single job node, so as to determine the node matching the current data shard to execute data in parallel according to the starting record sequence number and the end record sequence number of the current data shard. In this solution, the data can be coarse-grainedly sharded based on the total number of records of the initial data to be processed and the target parallelism, and the coarse-grained data shards can be dynamically adjusted according to the correlation between the data through the shard association characteristics. While ensuring the uniformity of data sharding, data correlation is also taken into account to avoid related data being scattered in different data shards, ensuring the reliability of data processing, and solving the problem of uneven sharding in existing data sharding methods, which leads to low data processing efficiency in financial institutions. It can ensure the uniformity of data sharding and the balance of node load to improve data processing efficiency and accuracy.

[0077] Example 3

[0078] Figure 4 This is a structural diagram of a data sharding device provided in the third embodiment of the present invention. Figure 4 As shown, the device includes:

[0079] The data analysis module 310 is used to analyze the initial data to be processed and obtain the shard correlation characteristics of the initial data to be processed;

[0080] A single job node processing record number calculation module 320 is used to calculate the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism;

[0081] The starting and ending record sequence number determination module 330 is used to determine the starting record sequence number of the current data shard when the current data to be processed meets the data sharding permission conditions, and to determine the ending record sequence number of the current data shard based on the shard association characteristics and the number of records processed by a single operation node;

[0082] The data sharding module 340 is used to determine the nodes matching the current data shard to execute data in parallel according to the starting record sequence number and the ending record sequence number of the current data shard.

[0083] The technical solution of the embodiment of the present invention obtains the shard association characteristics of the initial data to be processed by analyzing the initial data to be processed, and then calculates the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism. Then, when the current data to be processed meets the data sharding permission conditions, the starting record sequence number of the current data shard is determined, and the ending record sequence number of the current data shard is determined based on the shard association characteristics and the number of records processed by the single job node. Further, based on the starting record sequence number and the ending record sequence number of the current data shard, the node that matches the current data shard is determined to execute the data in parallel. In this solution, the data can be coarse-grainedly sharded based on the total number of records of the initial data to be processed and the target parallelism, and the coarse-grained data shards can be dynamically adjusted according to the correlation between the data through the shard association characteristics. While ensuring the uniformity of data sharding, data correlation is also taken into account to avoid related data being scattered in different data shards, ensuring the reliability of data processing, and solving the problem of uneven sharding in existing data sharding methods, which leads to low data processing efficiency in financial institutions. It can ensure the uniformity of data sharding and the balance of node load to improve data processing efficiency and accuracy.

[0084] Optionally, the data analysis module 310 is configured to determine a job processing scope of the initial data to be processed; and determine shard association characteristics of the initial data to be processed based on the job processing scope of the initial data to be processed.

[0085] Optionally, the starting and ending record sequence number determination module 330 includes a shard starting record sequence number determination unit, which is used to determine the current number of remaining records to be processed; when the difference between the total number of records of the initial data to be processed and the current number of remaining records to be processed is greater than the single-shard record number threshold, and the shard sequence number of the current data shard is less than the preset shard operation number, it is determined that the current data to be processed meets the data sharding permission condition; when the current data to be processed meets the data sharding permission condition, the starting record sequence number of the current data shard is determined based on the end record sequence number of the previous data shard of the current data shard, or the starting record sequence number of the first data shard.

[0086] Optionally, the data sharding device further includes a sharding parameter determination module for not sharding the current data to be processed when the current data to be processed does not meet the data sharding permission condition, and determining the sharding parameters of the current data to be processed.

[0087] Optionally, the data sharding device further includes a target parallelism prediction module for obtaining historical processing records of nodes; training a load prediction model based on the historical processing records of nodes to predict the target parallelism through the trained load prediction model.

[0088] Optionally, the data sharding device also includes a resharding module, which is used to retry the failed subtask when it is determined that the parallel processing task matching the initial data to be processed fails to execute; when the failed subtask fails to retry, obtain the data resharding feature and add the data resharding feature to the sharding association feature; return to execute the operation of calculating the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism, so as to complete the resharding of the initial data to be processed.

[0089] Optionally, the data sharding device further includes a data post-processing module for obtaining a parallel processing result when it is determined that the parallel processing task matching the initial data to be processed is successfully executed; and performing consistency check and data merging processing on the parallel processing result.

[0090] The data sharding device provided in the embodiment of the present invention can execute the data sharding method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0091] Example 4

[0092] Figure 5A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are provided for example only and are not intended to limit the implementation of the present inventions described and / or claimed herein.

[0093] like Figure 5 As shown, electronic device 10 includes at least one processor 11 and memory, such as ROM 12 and RAM 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor, and processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An I / O interface 15 is also connected to bus 14. ROM 12 is a read-only memory, RAM 13 is a random access memory, and I / O interface 15 is an input / output interface.

[0094] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0095] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors for running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the data sharding method.

[0096] In some embodiments, the data slicing method may be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data slicing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data slicing method in any other appropriate manner (e.g., by means of firmware).

[0097] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0099] In the context of the present invention, computer-readable storage medium can be a tangible medium that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device or any suitable combination of the foregoing.

[0100] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0101] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0102] A computing system may include clients and servers. The clients and servers are generally remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS servers.

[0103] The present application also discloses a computer program product, comprising a computer program that, when executed by a processor, implements the data sharding method provided in any of the embodiments of the present application. This program product and the data sharding method disclosed in each embodiment of the present application share the same inventive concept and are therefore not described in detail here.

[0104] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0105] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data sharding method, characterized in that: include: Analyze the initial data to be processed to obtain shard correlation characteristics of the initial data to be processed; Calculate the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism; When the current data to be processed meets the data sharding permission condition, determine the starting record sequence number of the current data shard, and determine the ending record sequence number of the current data shard based on the shard association characteristics and the number of records processed by the single operation node; According to the starting record sequence number and the ending record sequence number of the current data slice, a node matching the current data slice is determined to execute data in parallel.

2. The method according to claim 1, characterized in that Analyze the initial data to be processed to obtain the shard-related characteristics of the initial data to be processed, including: Determining a job processing scope of the initial data to be processed; Determine the shard association characteristics of the initial data to be processed according to the job processing scope of the initial data to be processed.

3. The method according to claim 1, characterized in that When the current data to be processed meets the data sharding permission conditions, the starting record sequence number of the current data shard is determined, including: Determine the number of records currently remaining to be processed; When the difference between the total number of records of the initial data to be processed and the current number of remaining records to be processed is greater than the single-shard record number threshold, and the shard sequence number of the current data shard is less than the preset shard operation number, it is determined that the current data to be processed meets the data sharding permission condition; When the current data to be processed meets the data fragmentation permission condition, the starting record sequence number of the current data fragment is determined according to the end record sequence number of the previous data fragment of the current data fragment or the starting record sequence number of the first data fragment.

4. The method according to claim 1, wherein After calculating the number of records to be processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism, the method further includes: When the current data to be processed does not meet the data sharding permission condition, the current data to be processed is not sharded, and sharding parameters of the current data to be processed are determined.

5. The method according to claim 1, wherein Before calculating the number of records to be processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism, the method further includes: Get the node's historical processing records; A load prediction model is trained according to the historical processing records of the node, so as to predict the target parallelism through the trained load prediction model.

6. The method according to claim 1, characterized in that Also includes: When it is determined that the parallel processing task matching the initial data to be processed fails to execute, retrying the failed subtask; When the failed subtask fails to be retried, a data resharding feature is obtained, and the data resharding feature is added to the sharding association feature; Return to execute the operation of calculating the number of records processed by a single job node based on the total number of records of the initial data to be processed and the target parallelism, so as to complete the resharding of the initial data to be processed.

7. The method according to claim 6, characterized in that Also includes: When it is determined that the parallel processing task matching the initial data to be processed is successfully executed, obtaining a parallel processing result; The parallel processing results are subjected to consistency checking and data merging processing.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data sharding method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data sharding method according to any one of claims 1 to 7 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the data sharding method according to any one of claims 1 to 7.

Images