Data task processing and segmenting method and related device

Abstract

The invention discloses a data task processing and segmenting method and a related device thereof. The data task processing method comprises the steps that a processing node receives at least one to-be-processed data task sent by a processing platform; for each to-be-processed data task, under the condition that the data volume of the to-be-processed data task meets a preset data volume requirement, the to-be-processed data task is segmented into a plurality of processing sub-data tasks by using a unique segmentation identifier, and the data volume difference between the processing sub-data tasks is smaller than a preset difference value; and performing concurrent processing on the plurality of processing sub-data tasks to complete the to-be-processed data task. According to the scheme, the real-time performance of data task processing can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data warehouse, and in particular to a data task processing and segmentation method and related devices thereof. Background Art

[0002] In the prior art, with the development and use of data processing platforms, data sharing between different platforms, different computing systems or different computing engines can be achieved. However, when processing very large amounts of data, it is limited by performance bottlenecks and resource bottlenecks, resulting in large delays in data processing, leading to a poor user experience. For example, for the data ferrying scenario of a single task with very large amounts of data, especially when the data is evenly distributed or unevenly distributed, the existing processing solutions are often limited by multiple bottlenecks such as performance bottlenecks in the processing link or resource bottlenecks of a single server, resulting in low real-time performance of the ferrying of very large amounts of data and large delays in data ferrying.

[0003] Therefore, there is an urgent need in the market for a data processing method to solve the above problems. Summary of the invention

[0004] The present application at least provides a data task processing and segmentation method and related devices, which can improve the real-time performance of data task processing.

[0005] The first aspect of the present application provides a data task processing method, which includes: a processing node receives at least one data task to be processed sent by a processing platform; for each data task to be processed, when the data volume of the data task to be processed meets the preset data volume requirement, using a unique splitting identifier to split the data task to be processed into a number of processing sub-data tasks, wherein the difference in data volume between each processing sub-data task is less than a preset difference value; and concurrently processing the several processing sub-data tasks to complete the data task to be processed.

[0006] Among them, the unique split identifier is the split key, and the split key represents the symbol for segmenting the data task to be processed; the data task to be processed is split into several processing sub-data tasks using the unique split identifier, including: in response to the split key in the data task to be processed meeting the preset condition, the data task to be processed is split using the split key to obtain several processing sub-data tasks.

[0007] Among them, the preset condition is that the split key is uniquely distributed; and / or the unique split identifier is a unique identifier, and the data task to be processed is split into several processing sub-data tasks using the unique split identifier, and also includes: in response to the split key in the data task to be processed not satisfying the preset condition, searching for the unique identifier of the original data corresponding to the data task to be processed, and using the dimension corresponding to the unique identifier to split the data task to be processed to obtain several processing sub-data tasks.

[0008] Among them, several processing sub-data tasks are processed concurrently to complete the data task to be processed, including: based on a preset data range, several processing sub-data tasks are combined to obtain a task execution group; the task execution group is processed concurrently to complete the data task to be processed, wherein the concurrent processing includes at least one of data reading, data flow, and data writing operations; and / or, the preset data volume requirement is that the data volume of the data task to be processed is greater than or equal to a processing threshold; and / or, the data task to be processed is the original data task received by the processing platform, or is a split sub-data task obtained by splitting the original data task by the processing platform.

[0009] The second aspect of the present application provides a data task splitting method, which includes: a processing platform determines the data volume of an original data task; in response to the data volume of the original data task satisfying a splitting condition, splits the original data task to obtain a number of split sub-data tasks; and respectively distributes the several split sub-data tasks to at least one processing node in a data processing system for processing, wherein, after receiving the split sub-data task sent by the processing platform, the processing node can split the sub-data task and concurrently process the split sub-data tasks.

[0010] Among them, the splitting conditions include at least one of the following: the current remaining resources of all processing nodes in the data processing system can process the data volume of the original data task, the data volume of the original data task is greater than or equal to the splitting threshold, the original data task is set with a target data processing rate and the target data processing rate is greater than the preset data processing rate, wherein the target data processing rate indicates that the original data task must be completed within the set time.

[0011] The original data task is segmented to obtain a number of segmented sub-data tasks, including: acquiring data association information of the original data task; in response to the data association information satisfying a first association condition, segmenting the original data task based on a library dimension to obtain a number of segmented sub-data tasks; in response to the data association information satisfying a second association condition, segmenting the original data task based on a data interval dimension to obtain a number of segmented sub-data tasks; wherein the first association condition indicates that the data association information corresponds to at least two data warehouses, and the data warehouse corresponds to at least two data tables; the second association condition indicates that the data association information corresponds to one data warehouse, and the data warehouse corresponds to one data table.

[0012] Among them, several split sub-data tasks are respectively assigned to at least one processing node in the data processing system for processing, including: using the current remaining resources of each processing node to make predictions, respectively, to obtain future resource prediction information of each processing node; determining the number of receiving tasks of each processing node based on each future resource prediction information; based on the number of receiving tasks of each processing node, assigning a corresponding number of split sub-data tasks to each processing node; and / or, the method also includes: in response to the data volume of the original data task not meeting the splitting conditions, directly sending the original data task to the processing node.

[0013] The third aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the data task processing method in the first aspect above, and / or to implement the data task segmentation method in the second aspect above.

[0014] The sixth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon. When the program instructions are executed by a processor, the data task processing method in the first aspect and / or the data task segmentation method in the second aspect are implemented.

[0015] In the above scheme, after receiving at least one data task to be processed sent by the processing platform, the processing node uses a unique splitting identifier to split the data task to be processed into several processing sub-data tasks when the data volume of the data task to be processed meets the preset data volume requirement, so that the difference in data volume between each processing sub-data task is less than the preset difference value, so as to achieve the effect of uniform splitting, and then concurrently processes several processing sub-data tasks to complete the data task to be processed. Regardless of whether the data in the data task is evenly distributed or unevenly distributed, it can be evenly split using a unique splitting identifier, so that the difference in data volume between each processing sub-data task will not be too large, so as to achieve the purpose of improving the performance of large data volume tasks. At the same time, the data task to be processed is split into smaller-level processing sub-data tasks and performed concurrently, which can improve the efficiency of task execution, thereby enabling data tasks to be processed concurrently more quickly to improve the real-time performance of data task processing.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0018] Figure 1 It is a flowchart of an embodiment of the data task segmentation method of the present application; Figure 2 It is a flowchart of an embodiment of the data task processing method of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of the data processing system of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of a data task processing device of the present application; Figure 5 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 6 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0019] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0020] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0021] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0022] At present, with the widespread development and use of data processing platforms, the performance bottleneck and real-time problems they face are becoming more and more severe. For example, as a specific example of a massive data processing platform, the big data computing platform (MaxCompute) provides distributed processing capabilities for TB (Tera Byte) / PB (Peta Byte)-level data with low real-time requirements, and is applied to data analysis, mining, business intelligence and other fields. Therefore, in order to give full play to the role of the big data computing platform, it is hoped that data can be shared with other different computing platforms, different computing systems or different computing engines. One of the challenges is the real-time performance of data, which guarantees the value of data.

[0023] See also Figure 1 , Figure 1This is a flow chart of an embodiment of the data task segmentation method of the present application. Specifically, it may include the following steps: Step S110: The processing platform determines the data volume of the original data task.

[0024] This application is mainly used in data ferrying scenarios, where data ferrying is to share data between different computing platforms, different computing systems or different computing engines through data middleware. In this scenario, the amount of data that usually needs to be ferryed is huge, so there are performance bottlenecks such as data extraction, processing, and delivery in the data ferrying process, which leads to delays in data ferrying, seriously affecting the experience of the target object and the efficiency of data ferrying. Therefore, in this application, the original data task with a huge amount of data is split into multiple subtasks to simplify the data task and improve the execution of the task. For the subtasks, the unique segmentation identifier is used for uniform segmentation and distribution, which can improve the performance of data processing and thus achieve the purpose of improving real-time performance.

[0025] In some embodiments, after receiving the original data task, the processing platform analyzes the original data task. First, the processing platform analyzes the data volume of the original data task to determine whether the data volume of the original data task is greater than or equal to the split threshold; then, it determines whether the current remaining resources of all processing nodes in the data processing system can process the data volume of the original data task, thereby determining whether to send the original data task directly to the processing node or send the split original data task to the processing node.

[0026] In addition, if the target data processing rate is also set in the original data task, the processing platform must also detect the target data processing rate to detect whether the target data processing rate corresponding to the original data task is greater than the preset data processing rate. Among them, the target data processing rate means that the original data task must be completed within the set time. For example, the data volume of the original data task is 10 million. Although the data volume has not reached the splitting threshold, the target data processing rate is set in the original data task, and the task must be completed within 1 minute. Therefore, in order to complete the task, the processing platform combines the current remaining resource information of all processing nodes in the data processing system to reasonably split the original data task to complete the task within 1 minute.

[0027] Step S120: In response to the data volume of the original data task satisfying the splitting condition, the original data task is split to obtain a plurality of split sub-data tasks.

[0028] Among them, the splitting conditions include at least one of the following: the current remaining resources of all processing nodes in the data processing system can process the data volume of the original data task, the data volume of the original data task is greater than or equal to the splitting threshold, and the original data task has a target data processing rate and the target data processing rate is greater than the preset data processing rate.

[0029] In some embodiments, the splitting condition can be set to the data volume of the original data task being greater than or equal to the splitting threshold. If the processing platform detects that the data volume of the original data task is greater than or equal to the splitting threshold, the processing platform needs to split the original data task to obtain a number of split sub-data tasks, thereby simplifying the original data task to improve execution efficiency. At this time, whether the current remaining resources of all processing nodes in the data processing system can handle the data volume of the original data task, the original data task needs to be executed. For example, if the current remaining resources of all processing nodes in the data processing system can handle the data volume of the original data task, the current remaining resources of all processing nodes can be fully mobilized, and a reasonable resource distribution method can be adopted to quickly complete the concurrent processing of several split sub-data tasks; if the current remaining resources of all processing nodes in the data processing system cannot handle the data volume of the original data task, the several split sub-data tasks obtained by splitting the original data task can be processed in batches, or the split sub-data tasks with higher priority among the several split sub-data tasks obtained can be given priority processing, or the operation object can be notified through an alarm, "This task exceeds the processing capacity of the data processing system. It is recommended to add new processing nodes to this data processing system, or stop the task!" and so on.

[0030] In addition, if the processing platform detects that the data volume of the original data task is less than the splitting threshold, the original data task does not need to be split and can be directly sent to the processing node.

[0031] In other embodiments, the splitting condition can be set as the data volume of the original data task is greater than or equal to the splitting threshold and the target data processing rate is greater than the preset data processing rate. If the processing platform detects that the data volume of the original data task is greater than or equal to the splitting threshold, and the original data task is also set with a target data processing rate and the target data processing rate is also greater than the preset data processing rate, in this case, the original data task is split and processed to obtain a number of split sub-data tasks.

[0032] In addition, if the data volume of the original data task is less than the splitting threshold and the target data processing rate is not set or the set target data processing rate is less than the preset data processing rate, the original data task does not need to be split and can be directly sent to the processing node.

[0033] In other embodiments, the splitting condition may be set to the data volume of the original data task that the current remaining resources of all processing nodes in the data processing system can process. If the processing platform detects that the data volume of the original data task is less than the splitting threshold, but the current remaining resources of all processing nodes in the data processing system are fully capable of processing the data volume of the original data task, then, in order to reasonably utilize resources, the processing platform may split the original data task to obtain a number of split sub-data tasks.

[0034] In other embodiments, if the original data task does not simultaneously meet the three splitting conditions: the current remaining resources of all processing nodes in the data processing system are able to process the data volume of the original data task, the data volume of the original data task is greater than or equal to the splitting threshold, and the original data task has a target data processing rate and the target data processing rate is greater than the preset data processing rate, that is, in response to the data volume of the original data task not meeting the splitting conditions, the original data task is directly sent to the processing node.

[0035] In some embodiments, when the original data task meets the splitting condition, the original data task may be split to obtain a plurality of split sub-data tasks. Specifically, the original data task may be split based on the library dimension or the data interval dimension to obtain a plurality of split sub-data tasks.

[0036] Therefore, before the original data task is segmented, it is necessary to determine whether to segment it in the library dimension or in the data interval dimension. Therefore, the data association information of the original data task can be obtained first. If the data association information satisfies the first association condition, the original data task is segmented based on the library dimension to obtain several segmentation sub-data tasks; if the data association information satisfies the second association condition, the original data task is segmented based on the data interval dimension to obtain several segmentation sub-data tasks. Among them, the first association condition indicates that the data association information corresponds to at least two data warehouses, and the data warehouse corresponds to at least two data tables. At this time, the original data task is segmented according to the table, so that each segmentation sub-data task obtained by the segmentation corresponds to a table; the second association condition indicates that the data association information corresponds to a data warehouse, and the data warehouse corresponds to a data table. At this time, the data interval dimension is used for segmentation. For example, if a data table corresponding to the original data task has a data volume of 100 million, it is segmented with a data interval of 100,000, and 1,000 segmentation sub-data tasks can be obtained.

[0037] In other embodiments, a data segmentation level can be set, and different data volumes are segmented using different data segmentation levels. For example, the segmentation threshold is set to 100 million, and after reaching the segmentation threshold condition, the data is segmented according to the data segmentation level: the first data segmentation level is a data task with a data volume of 100 million to 500 million (excluding 500 million), which uses a data interval dimension of 200,000 for segmentation; the second data segmentation level is a data task with a data volume of 500 million to 1 billion (excluding 1 billion), which uses a data interval dimension of 2 million for segmentation; the third data segmentation level is a data task with a data volume of 1 billion and above, which uses a data interval dimension of 20 million for segmentation. After the data volume of the original data task meets the segmentation threshold, the corresponding segmentation method can be selected for segmentation based on which data segmentation level its data volume meets.

[0038] In addition, the classification can also be carried out by quantitative data classification, which is not specifically limited here.

[0039] Step S130: Allocate a plurality of segmented sub-data tasks to at least one processing node in the data processing system for processing.

[0040] Among them, after receiving the split sub-data task sent by the processing platform, the processing node can split the sub-data task and perform concurrent processing on the split sub-data task.

[0041] In some embodiments, the obtained plurality of sliced ​​sub-data tasks may be randomly assigned to at least one processing node in the data processing system for processing, or the obtained plurality of sliced ​​sub-data tasks may be evenly assigned to at least one processing node in the data processing system for processing.

[0042] In other embodiments, considering that the processing capacity of each processing node may be different, in order to fully and reasonably utilize the resources of the data processing system, the sub-data tasks may be allocated according to the processing capacity of each processing node. Specifically, the current remaining resource situation of each processing node in the data processing system is first obtained, and then the current remaining resources of each processing node are used to make predictions to obtain the future resource prediction information of each processing node. Based on the future resource prediction information, the number of receiving tasks of each processing node is determined, and based on the number of receiving tasks of each processing node, the corresponding number of sub-data tasks are allocated to each processing node, so as to maximize the utilization of the fragmented resources of each processing node.

[0043] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of the data task processing method of the present application. Specifically, it may include the following steps: Step S210: The processing node receives at least one data task to be processed sent by the processing platform.

[0044] In some embodiments, the data task to be processed received by the processing node may be the original data task received by the processing platform. In this case, the processing node is a random one of multiple processing nodes in the data processing system whose current remaining resources can process the original data task; or it may be a split sub-data task obtained by the processing platform by splitting the original data task. In this case, the number of split sub-data tasks received by the processing node is within the maximum amount of data that the current remaining resources of the processing node can allow to be received.

[0045] Step S220: For each data task to be processed, if the data volume of the data task to be processed meets the preset data volume requirement, the data task to be processed is divided into several processing sub-data tasks using a unique segmentation identifier, wherein the data volume difference between the processing sub-data tasks is less than the preset difference value.

[0046] In some embodiments, the preset data volume requirement may be set based on historical experience. When the data volume of the data task to be processed meets the preset data volume requirement, the data task to be processed may be divided into several processing sub-data tasks using a unique split identifier, wherein the preset data volume requirement is that the data volume of the data task to be processed is greater than or equal to the processing threshold. Specifically, the unique split identifier may be a split key, which represents a symbol for segmenting the data task to be processed. In response to the split key in the data task to be processed meeting the preset condition, the data task to be processed is split using the split key to obtain several processing sub-data tasks. The preset condition may be that the split key is uniquely distributed.

[0047] In other embodiments, the split key in the data task to be processed is detected. If it is detected that the split key in the data task to be processed does not meet the preset condition, the unique identifier of the original data corresponding to the data task to be processed is found, and the data task to be processed is split using the dimension corresponding to the unique identifier to obtain a number of processing sub-data tasks. In this case, the unique split identifier is the unique identifier. The preset condition is that the split key exists or the split key is uniquely distributed.

[0048] Step S230: concurrently process a plurality of data processing subtasks to complete the data task to be processed.

[0049] In some embodiments, the processing node performs multi-threaded concurrent processing of sub-data tasks to improve task execution efficiency.

[0050] In other embodiments, a number of processing sub-data tasks can be combined based on a preset data range to obtain a task execution group; the task execution group is processed concurrently to complete the data task to be processed, so that data sharing between different computing platforms, different computing systems or different computing engines can be achieved, wherein concurrent processing includes at least one of data reading, data flow, and data writing operations. Specifically, the data task to be processed corresponds to a table of data, in which each row of data is set with a serial number, therefore, 300 rows of data can be selected as the preset data range, for example, serial numbers 1-300 rows are the first task execution group, serial numbers 301-600 rows are the second task execution group, etc. In addition, 300 rows of data can also be randomly selected as the task execution group. After obtaining the task execution group, the task execution group is used as the smallest concurrent unit, and the task execution group is used to be responsible for the data reading, data flow, and data writing of each processing sub-data task within its unit.

[0051] In addition, considering that the amount of data in the sub-data processing task may be large, in order to maximize the execution efficiency of the sub-data processing task, at this time, multi-threaded concurrency will be implemented for the execution operation of a certain operator in the sub-data processing task. For example, data reading will start multiple threads to implement concurrent reading, and data writing will implement thread-level concurrent writing, further improving the execution efficiency of the task.

[0052] In this application, the original data task is split into multiple sub-data tasks, and the sub-data tasks are split into task execution groups, all of which are intelligently analyzed and calculated, not just simply improving task performance without considering the complexity of task execution. Therefore, the data processing system will also conduct comprehensive calculation learning based on historical data, and output unique concurrent model parameters for each original data task type.

[0053] In this application, the original data task is split into multiple segmentation sub-data tasks to achieve task-level concurrency and improve data ferrying performance. The segmentation sub-data tasks are assigned to each processing node for concurrent execution, which can maximize the use of the resources of each processing node, break through the resource bottleneck of single processing node execution, and achieve unlimited performance improvement. The segmentation sub-data tasks are then segmented into processing sub-data tasks, and then a number of processing sub-data tasks are randomly formed into task execution groups. According to the multi-concurrency execution of the task execution group, the minimum granularity concurrency at the task level is achieved to improve the task execution efficiency. When the segmentation sub-data tasks are segmented, the uneven segmentation sub-data tasks are evenly segmented according to the unique segmentation identifier, and the processing performance of the uneven segmentation sub-data tasks can be improved. And the high-performance memory queue is used to decouple the concurrency of data reading and data writing, and the execution efficiency at the processing sub-data task level is improved. Finally, this solution can maximize the performance of a large data volume task, and achieve the purpose of continuously improving performance as the execution resources are improved.

[0054] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0055] See also Figure 3 , Figure 3 3 is a schematic diagram of a framework of an embodiment of a data processing system 300 of the present application. The data processing system 300 includes a processing platform 310 and a processing node 320. The processing platform 310 is used to detect and analyze the received raw data, and to segment the raw data according to the analysis results to obtain segmentation sub-data tasks, and to distribute the segmentation sub-data tasks to at least one processing node 320 in the data processing system 300 for processing. The processing node 320 is used to segment the received data task to be processed using a unique segmentation identifier to obtain a number of processing sub-data tasks, and to randomly combine the processing sub-data tasks according to a preset data range to be included in a task execution group, and then to concurrently process the task execution group to complete the data task to be processed, wherein the data task to be processed received by the processing node 320 can be the raw data task received by the processing platform 310, or the segmentation sub-data task obtained by the processing platform segmenting the raw data task. In this embodiment, the preset data range is set to 2. It can be understood that the preset data range can be set according to actual conditions, and no specific limitation is made here.

[0056] The processing nodes 320 may include a first processing node 3201, a second processing node 3202 and multiple processing nodes 320, and the number of processing nodes 320 in the data processing system 300 may be increased or decreased according to the data volume of the original data task to be executed.

[0057] See also Figure 4 , Figure 4 It is a schematic diagram of the framework of an embodiment of a data task processing device 400 of the present application. The data task processing device 400 includes: a receiving module 410, a splitting module 420 and a concurrent execution module 430. The receiving module 410 executes a processing node to receive at least one data task to be processed sent by a processing platform. The splitting module 420 executes, for each data task to be processed, when the data volume of the data task to be processed meets the preset data volume requirement, to split the data task to be processed into a plurality of processing sub-data tasks using a unique splitting identifier, wherein the data volume difference between each processing sub-data task is less than a preset difference value. The concurrent execution module 430 performs concurrent processing of a plurality of processing sub-data tasks to complete the data task to be processed.

[0058] In some embodiments, the splitting module 420 executes a unique splitting identifier as a splitting key, where the splitting key represents a symbol for segmenting the data task to be processed; the data task to be processed is split into a number of processing sub-data tasks using the unique splitting identifier, including: in response to the splitting key in the data task to be processed meeting a preset condition, the data task to be processed is split using the splitting key to obtain a number of processing sub-data tasks.

[0059] In some embodiments, the splitting module 420 executes a preset condition that the splitting key is uniquely distributed; and / or the unique splitting identifier is a unique identifier, and the data task to be processed is split into a number of processing sub-data tasks using the unique splitting identifier, and also includes: in response to the splitting key in the data task to be processed not satisfying the preset condition, searching for the unique identifier of the original data corresponding to the data task to be processed, and splitting the data task to be processed using the dimension corresponding to the unique identifier to obtain a number of processing sub-data tasks.

[0060] In some embodiments, the concurrent execution module 430 performs concurrent processing of several processing sub-data tasks to complete the data task to be processed, including: based on a preset data range, combining several processing sub-data tasks to obtain a task execution group; concurrently processing the task execution group to complete the data task to be processed, wherein the concurrent processing includes at least one of data reading, data flow, and data writing operations.

[0061] In some embodiments, the splitting module 420 executes a preset data volume requirement that the data volume of the data task to be processed is greater than or equal to a processing threshold; and / or, the data task to be processed is the original data task received by the processing platform, or is a split sub-data task obtained by splitting the original data task by the processing platform.

[0062] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of an electronic device 50 of the present application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other, and the processor 52 is used to execute program instructions stored in the memory 51 to implement the steps of any of the above data task segmentation method embodiments, or to implement the steps of any of the above data task processing method embodiments. In a specific implementation scenario, the electronic device 50 may include, but is not limited to: a microcomputer, a server, and in addition, the electronic device 50 may also include mobile devices such as a laptop computer and a tablet computer, which are not limited here.

[0063] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps of any of the above-mentioned data task segmentation method embodiments, or to implement the steps of any of the above-mentioned data task processing method embodiments. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0064] See also Figure 6 , Figure 6 1 is a schematic diagram of a computer-readable storage medium 60 of the present application. The computer-readable storage medium 60 stores program instructions 601 that can be executed by a processor, and the program instructions 601 are used to implement the steps of any of the above data task segmentation method embodiments, or to implement the steps of any of the above data task processing method embodiments.

[0065] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0066] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0067] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0068] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0069] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A data task processing method, characterized in that: include: The processing node receives at least one data task to be processed sent by the processing platform; For each of the data tasks to be processed, when the data volume of the data task to be processed meets the preset data volume requirement, the data task to be processed is divided into a plurality of processing sub-data tasks using a unique splitting identifier, wherein the data volume difference between the processing sub-data tasks is less than a preset difference value; The plurality of processing sub-data tasks are processed concurrently to complete the data task to be processed.

2. The method according to claim 1, characterized in that The unique segmentation identifier is a segmentation key, and the segmentation key represents a symbol for segmenting the data task to be processed; the method of using the unique segmentation identifier to segment the data task to be processed into a plurality of processing sub-data tasks includes: In response to the split key in the data task to be processed satisfying a preset condition, the data task to be processed is split using the split key to obtain the plurality of processing sub-data tasks.

3. The method according to claim 2, characterized in that The preset condition is that the segmentation key is uniquely distributed; And / or, the unique segmentation identifier is a unique identifier, and the method of using the unique segmentation identifier to segment the to-be-processed data task into a plurality of processing sub-data tasks further includes: In response to the split key in the data task to be processed not satisfying the preset condition, the unique identifier of the original data corresponding to the data task to be processed is searched, and the data task to be processed is split using the dimension corresponding to the unique identifier to obtain the several processing sub-data tasks.

4. The method according to claim 1, characterized in that: The concurrently processing the plurality of processing sub-data tasks to complete the data task to be processed includes: Based on a preset data range, a plurality of the processing sub-data tasks are combined to obtain a task execution group; Performing concurrent processing on the task execution group to complete the data task to be processed, wherein the concurrent processing includes at least one of data reading, data transfer, and data writing operations; And / or, the preset data volume requirement is that the data volume of the data task to be processed is greater than or equal to a processing threshold; And / or, the data task to be processed is an original data task received by the processing platform, or is a segmented sub-data task obtained by segmenting the original data task by the processing platform.

5. A data task segmentation method, characterized in that: include: The processing platform determines the data volume of the raw data task; In response to the data volume of the original data task satisfying a splitting condition, splitting the original data task to obtain a plurality of split sub-data tasks; Several of the split sub-data tasks are respectively assigned to at least one processing node in the data processing system for processing, wherein after receiving the split sub-data tasks sent by the processing platform, the processing node can split the split sub-data tasks and concurrently process the split sub-data tasks.

6. The method according to claim 5, characterized in that The splitting condition includes at least one of the following: the current remaining resources of all processing nodes in the data processing system are capable of processing the data volume of the original data task, the data volume of the original data task is greater than or equal to the splitting threshold, the original data task is set with a target data processing rate and the target data processing rate is greater than a preset data processing rate, wherein the target data processing rate indicates that the original data task must be completed within a set time.

7. The method according to claim 5, characterized in that The original data task is segmented to obtain a plurality of segmented sub-data tasks, including: Acquire data association information of the original data task; In response to the data association information satisfying a first association condition, segmenting the original data task based on a library dimension to obtain a plurality of segmented sub-data tasks; In response to the data association information satisfying a second association condition, segmenting the original data task based on a data interval dimension to obtain a plurality of segmented sub-data tasks; Among them, the first association condition indicates that the data association information corresponds to at least two data warehouses, and the data warehouses correspond to at least two data tables; the second association condition indicates that the data association information corresponds to one data warehouse, and the data warehouse corresponds to one data table.

8. The method according to claim 5, characterized in that The step of respectively allocating the plurality of segmentation sub-data tasks to at least one processing node in the data processing system for processing comprises: Using the current remaining resources of each processing node to perform prediction respectively, to obtain future resource prediction information of each processing node; Determining the number of receiving tasks of each processing node based on each future resource prediction information; Based on the number of received tasks of each processing node, a corresponding number of the segmented sub-data tasks are allocated to each processing node; And / or, the method further comprises: In response to the data volume of the original data task not satisfying the segmentation condition, directly sending the original data task to the processing node.

9. An electronic device, characterized in that: It includes a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the data task processing method described in any one of claims 1 to 4, and / or to implement the data task segmentation method described in any one of claims 5 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, the data task processing method described in any one of claims 1 to 4 is implemented, and / or the data task segmentation method described in any one of claims 5 to 8 is implemented.

Images