A data fragmentation processing method, apparatus and electronic device

By sorting and grouping data tasks according to execution time and processing them in the order of thread execution, the problem of uneven task allocation in traditional methods is solved, thereby improving the efficiency of multi-threaded processing and the overall performance of the system.

CN122195689BActive Publication Date: 2026-11-10CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202610678489.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-05-18
Publication Date
2026-11-10
Estimated Expiration
2046-05-18

AI Technical Summary

Technical Problem

Traditional data sharding methods prioritize the processing of time-consuming tasks, leading to uneven task allocation, widening the gap in processing time, increasing thread idle time, and reducing the efficiency of multi-threaded processing.

Method used

Multiple target data tasks are sorted by execution time, and each group of M tasks is assigned to M data shards. These shards are then executed by X threads in the order of execution time until all tasks are completed.

Benefits of technology

It achieves a balanced distribution of data tasks with different durations, reduces thread idle time, improves the overall utilization of multithreading, optimizes the distributed parallel processing flow, and reduces the total duration of data tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122195689B_ABST
    Figure CN122195689B_ABST
Patent Text Reader

Abstract

This application discloses a data sharding processing method, apparatus, and electronic device. The method includes: sorting multiple target data tasks according to the execution time of each target data task; grouping the sorted target data tasks into groups of M, and sequentially allocating the M target data tasks in each group to M target data shards; and using X target threads to sequentially execute the target data tasks contained in each target data shard according to their respective execution time, until all target data tasks contained in the M target data shards have been executed. Here, M and X are both positive integers. Implementing this application embodiment can evenly allocate data tasks of different durations based on data sharding, effectively improving the overall utilization rate of multi-threading, thereby helping to reduce the total processing time of data tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data fragmentation processing method, apparatus, and electronic device. Background Technology

[0002] Currently, with the development of multi-threaded parallel processing technology and distributed computing, data task sharding on various platforms (such as cloud platforms and local servers) has gradually become the norm. However, traditional data sharding methods often tend to prioritize processing time-consuming tasks, which not only squeezes out the processing priority of a large number of time-consuming tasks, but also easily widens the processing time gap between different data shards, leading to an imbalance in task allocation for different threads, and making threads more prone to idle time, thus increasing the total processing time of data tasks. Summary of the Invention

[0003] This application discloses a data sharding processing method, apparatus, and electronic device, which can evenly allocate data tasks of different durations based on data sharding, effectively improve the overall utilization of multi-threading, and thus help reduce the total processing time of data tasks.

[0004] The first aspect of this application discloses a data fragmentation processing method, including: For multiple target data tasks, sort them according to the execution time of each target data task; The sorted target data tasks are grouped into groups of M in sequence, and the M target data tasks in each group are then assigned to M target data slices in turn; where M is a positive integer. Using X target threads, the target data tasks contained in each target data slice are executed sequentially according to their respective execution time lengths, until all target data tasks contained in M ​​target data slices are completed; where X is a positive integer.

[0005] The second aspect of this application discloses a data fragmentation processing apparatus, comprising: The sorting unit is used to sort multiple target data tasks according to the execution time of each target data task. The data sharding unit is used to group the sorted target data tasks into groups of M in sequence, and then sequentially allocate the M target data tasks of each group to M target data shards; where M is a positive integer; An execution unit is configured to use X target threads to sequentially execute the target data tasks contained in each of the target data slices according to their respective execution time lengths, until all the target data tasks contained in the M target data slices have been executed; where X is a positive integer.

[0006] The third aspect of this application discloses an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor enables the processor to implement any of the data fragmentation processing methods disclosed in the first aspect of this application.

[0007] The fourth aspect of this application discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the data fragmentation processing methods disclosed in the first aspect of this application.

[0008] Compared with related technologies, the embodiments of this application have the following beneficial effects: In this embodiment, by sorting multiple target data tasks according to their execution time, the sorted target data tasks can be further grouped into groups of M, and each group of M target data tasks can be assigned to M target data shards. Based on this, X target threads can sequentially execute the target data tasks within each target data shard according to their execution time, until all target data tasks within the M target data shards are completed. Here, M and X are both positive integers. Therefore, implementing this embodiment can evenly distribute data tasks of different durations during data sharding and the actual execution of sharded data tasks, making each data shard contain more diverse tasks, avoiding the concentration of long-running tasks for priority processing, and minimizing the difference in the total processing time between all tasks in different data shards. This further balances task allocation across different threads, reduces thread idle time, effectively improves the overall utilization of multi-threading, optimizes the multi-threaded distributed parallel processing flow, helps reduce the total processing time of data tasks, and improves the overall efficiency of the platform or system. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating a data fragmentation processing method disclosed in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the execution of a fragmented data task using multiple threads, as disclosed in an embodiment of this application. Figure 3This is a flowchart illustrating another data fragmentation processing method disclosed in an embodiment of this application; Figure 4 This is a flowchart illustrating another data fragmentation processing method disclosed in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the method for calculating the total execution time of multiple threads, as disclosed in an embodiment of this application. Figure 6 This is a modular schematic diagram of a data fragmentation processing device disclosed in an embodiment of this application; Figure 7 This is a modular schematic diagram of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0011] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0012] This application discloses a data sharding processing method, apparatus, and electronic device, which can evenly allocate data tasks of different durations based on data sharding, effectively improve the overall utilization of multi-threading, and thus help reduce the total processing time of data tasks.

[0013] The following will be described in detail with reference to the accompanying drawings.

[0014] Please see Figure 1 , Figure 1 This is a flowchart illustrating a data fragmentation processing method disclosed in an embodiment of this application. Figure 1 As shown, the method may include the following steps: S102. For multiple target data tasks, sort them according to the execution time of each target data task.

[0015] In this embodiment, to achieve parallel processing of multiple target data tasks, data sharding can be used to allocate the multiple target data tasks to different target data shards. Multiple threads can then be invoked to process (i.e., execute) the target data tasks contained in each target data shard, thereby allocating the multi-threaded processing flow based on data sharding. The aforementioned target data tasks may include one or more task units that can be executed by threads in the data processing system.

[0016] In some embodiments, the parallel processing of the multiple target data tasks can be centralized, for example, by calling multiple target threads through a local server; in other embodiments, the parallel processing of the multiple target data tasks can also be distributed, for example, by deploying multiple target threads through a cloud platform.

[0017] It is understood that a single thread often requires different execution times when executing different target data tasks. These execution times can be preset according to different task types or estimated separately for specific target data tasks. No specific limitation is made in the embodiments of this application.

[0018] Based on this, the multiple target data tasks are sorted according to their execution time (e.g., sorted from longest to shortest execution time or from shortest to longest execution time, or sorted according to the time interval to which the execution time belongs). In subsequent steps, based on the sorted target data tasks, a relatively balanced data sharding can be achieved, so that the target data tasks contained in each target data shard are more diverse. This can avoid the risk of long-running tasks being concentrated on the same shard, which would crowd out other tasks and reduce the average response time of the data processing system.

[0019] S104. The sorted target data tasks are grouped into groups of M in order, and the M target data tasks in each group are then assigned to the M target data slices in turn; where M is a positive integer.

[0020] In this embodiment of the application, for multiple sorted target data tasks, they can be further grouped according to the number of target data shards to be allocated, so as to evenly distribute the multiple target data tasks to each target data shard.

[0021] For example, when the data processing system uses M target data fragments, the aforementioned multiple target data tasks (taking N as an example, where N is a positive integer and N is greater than M) can be grouped into groups of M according to the order arranged in step S102. For instance, the first M target data tasks among the N target data tasks (e.g., the M target data tasks with the longest execution time, which can be numbered sequentially from 1 to M) can be divided into the first group, and then the (M+1)th to 2Mth target data tasks can be divided into the second group, and so on. Specifically, the target data tasks in each group can be represented by their numbering as follows: , ... There are a total of K groups, among which , This represents the quotient (rounded up) of N divided by M.

[0022] Based on this, the aforementioned groups of target data tasks, after further allocation, can be used as row elements (one row for each group of target data tasks) to form a task matrix as shown in Formula 1 below, indicating that the M target data tasks of each group are sequentially allocated to the M target data fragments. This represents the i-th target data task assigned to the j-th target data shard. In some embodiments, it can also be specifically represented by the execution time corresponding to the target data task.

[0023] Formula 1:

[0024] in, Round up.

[0025] In some embodiments, if after sorting and grouping, the remaining last group of target data tasks has fewer than M tasks (taking P tasks as an example, where P is a positive integer and P is less than M), the group of target data tasks can be randomly assigned to P target data slices, or P slices can be selected from the M target data slices for allocation based on preset rules. For example, based on the execution time of each target data task, the P target data tasks can be assigned one by one to the P target data slices with the shortest total task duration among the M target data slices. The total task duration can be obtained by adding the execution times of the allocated target data tasks contained in each target data slice, thereby further balancing the processing time differences between different slices and reducing the total processing time of the data processing system for data tasks.

[0026] S106. Using X target threads, execute the target data tasks contained in each target data slice in sequence according to their respective execution time lengths, until all target data tasks contained in M ​​target data slices are completed; where X is a positive integer.

[0027] In this embodiment, the data processing system can call multiple threads to process the target data tasks contained in each target data slice sequentially. For example, the target data tasks contained in the first target data slice can be processed first. to Therefore, if all target data tasks contained in the first target data slice have been assigned to corresponding threads, threads that have completed their tasks can be called sequentially to process the target data tasks contained in the second target data slice. to This process continues until all target data tasks contained in the aforementioned M target data slices have been completed.

[0028] For example, the above process can be implemented using X target threads. For K target data tasks (K is a positive integer) contained in a certain target data slice, they can be sequentially assigned to the above X target threads for execution according to their corresponding execution times from longest to shortest.

[0029] In some embodiments, if K is less than or equal to X, the aforementioned target data tasks can be executed directly through the K idle threads among the X target threads. In other embodiments, if K is greater than X, the X target data tasks with the longest execution time can be assigned to the X target threads for execution. After any target thread completes its task execution, the remaining target data tasks can be assigned to the idle target threads in descending order of execution time.

[0030] For example, please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating a multi-threaded execution of a data sharding task, as disclosed in an embodiment of this application. Figure 2 As shown, taking X=3 as an example, the target threads can include T1, T2, and T3. Therefore, the three target data tasks with the longest execution time in the currently processed target data slice (based on...) , , For example, tasks can be assigned to the aforementioned target threads for execution; subsequently, after any target thread completes its task execution, the target data task with the longest remaining execution time in that target data slice can be reassigned to that target thread for further execution. Figure 2 T3 will complete the execution first. And then continue to execute For example); and so on, until all target data tasks in the target data slice. to Once all data slices have been assigned for execution, the above process can be repeated for the next target data slice.

[0031] It should be noted that, due to the data sharding performed in steps S102 to S104, each target data shard may contain tasks with longer execution times (hereinafter referred to as "long tasks") and tasks with shorter execution times (hereinafter referred to as "short tasks"). Therefore, even if the target data tasks are executed in step S106 in descending order of execution time, some short tasks can still be executed ahead of schedule, resulting in a more balanced task allocation among different threads and improving the flexibility of multi-threaded execution of sharded data tasks. Optionally, executing the target data tasks in ascending order of execution time in step S106 also allows different threads to execute long and short tasks in a balanced manner, similarly improving the flexibility of multi-threaded execution of sharded data tasks.

[0032] As can be seen, implementing the data sharding processing method described in the above embodiments can evenly allocate data tasks of different durations during data sharding and the actual execution of sharded data tasks. This makes the tasks contained in each data shard more diverse, avoids prioritizing the processing of time-consuming tasks, and minimizes the difference in the total processing time of all tasks contained in different data shards. This further balances the task allocation for different threads, reduces thread idle time, effectively improves the overall utilization of multithreading, optimizes the multithreaded distributed parallel processing flow, helps reduce the total processing time of data tasks, and improves the overall efficiency of the platform or system.

[0033] Please see Figure 3 , Figure 3 This is a flowchart illustrating another data fragmentation processing method disclosed in an embodiment of this application. Figure 3 As shown, the method may include the following steps: S302. For multiple target data tasks, sort them according to the execution time of each target data task.

[0034] Step S302 is similar to step S102 above, and will not be described again here.

[0035] S304. Group the sorted target data tasks into groups of M in sequence.

[0036] S306. Based on the execution time of each target data task in the target group, the M target data tasks of the target group are assigned to the M target data slices one by one until the traversal of each group of target data tasks is completed. The above target group is one of the groups of target data tasks.

[0037] Steps S304 and S306 are similar to some implementations of step S104 above. It should be noted that the K groups of target data tasks obtained after sorting and grouping each contain M target data tasks, except for the last group. The last group can contain M target data tasks (i.e., the total number of tasks N is divisible by M) or P target data tasks (P is less than M, i.e., the total number of tasks N is not divisible by M).

[0038] In some embodiments, taking a target group containing M target data tasks as an example, the target data task with the longest execution time that is not assigned in the target group can be assigned to the target data shard with the shortest total task duration that is not assigned in the M target data shards, until all M target data tasks in the target group are assigned. The total task duration can be obtained by adding the execution times of the assigned target data tasks in each target data shard. It can be understood that sorting the target data tasks in step S302 according to either the longest to shortest execution time or the shortest to longest execution time can achieve the assignment according to the above allocation rules. Furthermore, by assigning the target data task with the shortest execution time that is not assigned in the target group to the target data shard with the longest total task duration that is not assigned in the M target data shards, the adaptive allocation of the M target data tasks in the target group can also be completed accordingly.

[0039] For example, it can be done by To represent the number to be allocated x Each target data task and its corresponding execution time, for example This can be interpreted as the execution time for the first target data task being 7 seconds. Based on this, taking M=4 as an example, this applies to the four target data tasks within the first target group. , , , These tasks can be assigned to target data slices s1, s2, s3, and s4 respectively. It can be understood that since each of these target data slices does not yet contain any assigned tasks, their total task duration is equal, and therefore, the target data tasks can be randomly assigned. By arranging the target data tasks contained in the target data slices s1, s2, s3, and s4 into a matrix, we can obtain the following representation:

[0040] Furthermore, regarding the four target data tasks in the second target group... , , , The tasks can be allocated one by one according to the above allocation rules to minimize the difference in the total task duration corresponding to different data shards. It can be understood that when some target data shards have the same total task duration, a corresponding number of target data tasks can be randomly allocated to the aforementioned target data shards, resulting in the following representation:

[0041] Similarly, for the four target data tasks in the third target group , , , This can be further allocated to obtain the following representation:

[0042] By repeating the above allocation process until the traversal of each group of target data tasks is completed, the sorted target data tasks can be evenly distributed to each target data partition, and represented as a task matrix as shown in Formula 1 above. .

[0043] Optionally, if the last target group contains only P target data tasks (i.e. less than M), then the unassigned target data tasks with the longest execution time in the target group can be sequentially assigned to the P target data fragments that have not been assigned among the above M target data fragments and have the shortest total task duration, until the assignment is completed.

[0044] S308. For the target data tasks contained in the current data slice, the X target data tasks with the longest execution time are executed by X target threads respectively. Then, in order of execution time from longest to shortest, the remaining target data tasks contained in the current data slice are allocated and executed sequentially by the target thread that has completed the execution of the previous target data task. The current data slice is one of M target data slices.

[0045] S310. Traverse the target data slices in order from the 1st to the Mth slice until all target data tasks contained in the M target data slices have been executed.

[0046] In this embodiment, by invoking X target threads, the processing of target data tasks contained in each of the aforementioned target data slices can be implemented sequentially. For example, taking the current data slice (i.e., the target data slice currently being processed) as an example, the X target threads can execute the target data tasks in descending order of their execution times. It can be understood that at the beginning of the execution in the above order, i.e., when none of the X target threads have been assigned tasks, the X target data tasks with the longest execution times contained in the first target data slice can be randomly assigned to those X target threads for execution, and then the traversal execution process from the first to the Mth target data slices can begin.

[0047] For example, the first target data fragment s1 can be used as the current data fragment, which can contain the aforementioned target data task. , , ...Based on this, taking X=3 as an example, three target threads T1, T2, and T3 can be used to execute the three target data tasks with the longest execution time (i.e., the above-mentioned...). j 1. j 8. j 10).

[0048] Furthermore, whenever a target thread completes the execution of the previous target data task, a new target data task can be assigned to it. In some embodiments, if there are Y target threads that have not completed the execution of the previous target data task (Y is a positive integer and Y is less than X), then when allocating target data tasks contained in the current data slice, the XY target data tasks with the longest execution time can be executed by the XY idle target threads respectively.

[0049] For example, if there are still other tasks remaining in the first target data slice, the next target data task can be allocated and executed in descending order of execution time; if there are no remaining tasks in the first target data slice, the second target data slice s2 can be used as the current data slice, and the tasks contained in the second target data slice can continue to be allocated and executed. , , For example, target thread T3 completes the task of executing target data. j After step 10, if there are no remaining tasks in target data segment s1, then the target data task with the longest execution time in target data segment s2 can continue to be executed. j 2; Target thread T2 completes the task of executing the target data. j After step 8, the next target data task in the target data shard s2 can be executed.j 7; and so on, until all target data tasks contained in the M target data shards have been completed. It can be understood that when multiple target threads complete task execution simultaneously, they can each be assigned their corresponding next target data task at the same time. Optionally, similarly, the target data tasks can be executed sequentially by the X target threads according to their execution times from shortest to longest, until all target data tasks have been completed.

[0050] As can be seen, implementing the data sharding processing method described in the above embodiments can evenly allocate data tasks of different durations during data sharding and the actual execution of sharded data tasks. This makes the tasks contained in each data shard more diverse, avoids prioritizing the processing of time-consuming tasks, and minimizes the difference in the total processing time of all tasks contained in different data shards. This further balances the task allocation for different threads, reduces thread idle time, effectively improves the overall utilization of multithreading, and optimizes the multithreaded distributed parallel processing flow, which helps reduce the total processing time of data tasks and improves the overall efficiency of the platform or system. In addition, by allocating and executing each sharded data task according to its duration, the multithreaded processing flow can be further simplified, which helps ensure the efficient operation of the data processing system when facing a large number of data tasks.

[0051] Please see Figure 4 , Figure 4 This is a flowchart illustrating another data fragmentation processing method disclosed in an embodiment of this application. Figure 4 As shown, the method may include the following steps: S402. For multiple target data tasks, sort them according to the execution time of each target data task.

[0052] Step S402 is similar to step S102 above, and will not be described again here.

[0053] S404. Group the sorted target data tasks into groups of M in sequence.

[0054] S406. Assign the target data task that is not assigned in the target group and has the longest corresponding execution time to the target data shard that is not assigned in the M target data shards and has the shortest corresponding total task duration, until the assignment of the M target data tasks in the target group is completed. The total task duration is obtained by adding the execution times of the target data tasks contained in the corresponding target data shards.

[0055] Steps S404 and S406 are partially similar to steps S304 and S306 described above. It should be noted that the target group can be one of the target data tasks in each group, and the allocation process can be performed sequentially for each group of target data tasks until the traversal of each group of target data tasks is completed.

[0056] S408. For the target data tasks contained in the current data slice, the X target data tasks with the longest execution time are executed by X target threads respectively. Then, in order of execution time from longest to shortest, the remaining target data tasks contained in the current data slice are allocated and executed sequentially by the target thread that has completed the execution of the previous target data task. The current data slice is one of M target data slices.

[0057] S410. Traverse the target data slices in order from the 1st to the Mth slice until all target data tasks contained in the M target data slices have been executed.

[0058] Steps S408 and S410 are similar to steps S308 and S310 above, and will not be described again here.

[0059] S412. Calculate the total execution time of each target thread after completing all the corresponding target data tasks.

[0060] S414. Based on the total execution time of each target thread, determine the maximum value as the total task completion time.

[0061] In this embodiment, the total execution time for each target thread can be obtained by adding the actual execution times of all target data tasks it completes. It should be noted that, during the execution of a single target data task by a target thread, if the first... j If all tasks contained in a target data slice are assigned to other threads for execution, then that target thread will not execute the first task. j Any task on the target data shard, i.e., the target thread targeting the first... j The execution time for each target data shard can be 0.

[0062] Based on this, the total execution time corresponding to the target thread Tq Statistics can be performed using the following formula 2 ( ).

[0063] Formula 2:

[0064] in, This indicates that when the target thread Tq is idle, it will prioritize executing the first...j The target data task with the longest execution time among the target data slices; when the target thread Tq does not execute the first... j When any task is performed on a target data shard, its value can be 0. Furthermore, Rounding up indicates that the target thread Tq starts from the integer part of the first integer. j The maximum number of target data fragments contained Select the first task from the task. i There are several target data tasks. By summing the execution times for the first to Mth target data slices, the total execution time for the target thread Tq can be obtained. .

[0065] For example, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the method for calculating the total execution time of multiple threads, as disclosed in an embodiment of this application. Figure 5 As shown, the total execution time of each target thread T1, T2, and T3 can be calculated separately. , , This is to allow for the subsequent determination of the total task completion time. For example, target thread T1 is assigned to execute the target data task. j 1. j 6. j 5. Target thread T2 is assigned to execute the target data task. j 8. j 7. j 9. j 11. j 4. Target thread T3 is assigned to execute the target data task. j 10. j 2. j 3. j If the actual execution times of the corresponding tasks are added together, the total execution time of each target thread can be obtained.

[0066] Furthermore, after calculating the total execution time for each target thread, the maximum value can be taken as the total task completion time, i.e. For example, the total time to complete the task. By combining Formula 2 above, we can obtain the following expression as Formula 3.

[0067] Formula 3:

[0068] As can be seen, the data sharding processing method described in the above embodiments can evenly allocate data tasks of different durations during data sharding and the actual execution of sharded data tasks. This makes the tasks contained in each data shard more diverse, avoids prioritizing the processing of time-consuming tasks, and minimizes the difference in the total processing time of all tasks contained in different data shards. This further balances the task allocation for different threads, reduces thread idle time, effectively improves the overall utilization of multithreading, and optimizes the multithreaded distributed parallel processing flow, which helps reduce the total processing time of data tasks and improves the overall efficiency of the platform or system. In addition, by allocating the execution of each sharded data task according to the time sequence, the multithreaded processing flow can be further simplified, which helps ensure the efficient operation of the data processing system when facing a large number of data tasks. Furthermore, by statistically analyzing the total execution time of each thread, all data shards can be covered, and the actual execution information of multithreading, including the total task completion time, can be quickly determined. This helps to provide feedback and further optimize the multithreaded task processing flow of the data processing system based on data sharding.

[0069] Please see Figure 6 , Figure 6 This is a modular schematic diagram of a data fragmentation processing device disclosed in an embodiment of this application. Figure 6 As shown, the data fragmentation processing device may include a sorting unit 601, a data fragmentation unit 602, and an execution unit 603, wherein: The sorting unit 601 is used to sort multiple target data tasks according to the execution time of each target data task. The data sharding unit 602 is used to group multiple sorted target data tasks into groups of M in sequence, and then sequentially allocate the M target data tasks of each group to M target data shards; where M is a positive integer; The execution unit 603 is used to execute the target data tasks contained in each target data slice in sequence according to their respective execution time lengths through X target threads, until all target data tasks contained in M ​​target data slices are completed; where X is a positive integer.

[0070] As can be seen, the data sharding processing apparatus described in the above embodiments can evenly allocate data tasks of different durations during data sharding and the actual execution of sharded data tasks. This makes the tasks contained in each data shard more diverse, avoids prioritizing the processing of time-consuming tasks, and minimizes the difference in the total processing time of all tasks contained in different data shards. This further balances the task allocation for different threads, reduces thread idle time, effectively improves the overall utilization of multi-threading, optimizes the multi-threaded distributed parallel processing flow, helps reduce the total processing time of data tasks, and improves the overall efficiency of the platform or system.

[0071] In some embodiments, when the data sharding unit 602 sequentially allocates the M target data tasks of each group to the M target data shards, it may specifically include: Based on the execution time of each target data task in the target group, the M target data tasks in the target group are assigned to the M target data slices one by one until the traversal of each group of target data tasks is completed. The above target group is one of the groups of target data tasks.

[0072] As an optional implementation, for each of the above target groups, the data fragmentation unit 602 can specifically be used for: The target data task that is not assigned in the target group and has the longest corresponding execution time is assigned to the target data shard that is not assigned in the M target data shards and has the shortest corresponding total task duration, until the assignment of the M target data tasks in the target group is completed. The total task duration is obtained by adding the execution times of the target data tasks contained in the corresponding target data shards.

[0073] In some embodiments, when the data sharding unit 602 is used to sequentially allocate the M target data tasks of each group to the M target data shards, it may specifically include: If a target group contains P target data tasks, the P target data tasks in the target group are allocated one by one to the P target data slices with the shortest total task duration among the M target data slices, according to the execution time of each target data task; where P is a positive integer and P is less than M.

[0074] In some embodiments, the execution unit 603 described above can be specifically used for: For the target data tasks contained in the current data shard, the X target data tasks with the longest execution time are executed by X target threads respectively. Then, in order of execution time from longest to shortest, the remaining target data tasks contained in the current data shard are allocated and executed sequentially by the target thread that has completed the execution of the previous target data task. The current data shard is one of the M target data shards, and it is traversed in the order from the 1st to the Mth target data shards until all target data tasks contained in the M target data shards are completed.

[0075] In some embodiments, when the execution unit 603 is used to execute the X target data tasks with the longest execution times among the target data tasks contained in the current data slice through X target threads, it may specifically include: If Y target threads have not completed executing the previous target data task, for the target data tasks contained in the current data shard, the XY target data tasks with the longest execution time are executed by the XY idle target threads respectively; where Y is a positive integer and Y is less than X.

[0076] In some embodiments, the data fragmentation processing apparatus may further include a statistical unit (not shown), which may be used for: Calculate the total execution time of each target thread after completing all the corresponding target data tasks; Based on the total execution time of each target thread, the maximum value is determined to be the total task completion time.

[0077] As can be seen, the data sharding processing apparatus described in the above embodiments can further simplify the multi-threaded processing flow by allocating and executing each sharded data task in sequence according to its duration, which is beneficial to ensuring the efficient operation of the data processing system when faced with a large number of data tasks. In addition, by statistically analyzing the total execution time of each thread, all data shards can be covered, and the actual execution information of the multi-threaded processes, including the total task completion time, can be quickly determined. This helps to provide feedback and further optimize the multi-threaded task processing flow of the data processing system based on data sharding.

[0078] Please see Figure 7 , Figure 7 This is a modular schematic diagram of an electronic device disclosed in an embodiment of this application. For example... Figure 7 As shown, the electronic device may include: Memory 701 storing executable program code; Processor 702 coupled to memory 701; The processor 702 can call the executable program code stored in the memory 701 to execute all or part of the steps in any of the data fragmentation processing methods described in the above embodiments.

[0079] Furthermore, embodiments of this application disclose a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program enables a computer to execute all or part of the steps in any of the data fragmentation processing methods described in the above embodiments.

[0080] Furthermore, this application further discloses a computer program product that, when run on a computer, enables the computer to execute all or part of the steps in any of the data fragmentation processing methods described in the above embodiments.

[0081] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0082] The data fragmentation processing method, apparatus, and electronic device disclosed in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data fragmentation processing method, characterized in that, include: For multiple target data tasks, sort them according to the execution time of each target data task; The sorted target data tasks are grouped into groups of M in sequence, and the M target data tasks in each group are then assigned to M target data slices in turn; where M is a positive integer. Using X target threads, the target data tasks contained in each target data slice are executed sequentially according to their respective execution time lengths, until all target data tasks contained in M ​​target data slices are completed; where X is a positive integer; The step of sequentially allocating the M target data tasks of each group to M target data fragments includes: The target data task that is not assigned in the target group and has the longest corresponding execution time is assigned to the target data shard that is not assigned in the M target data shards and has the shortest corresponding total task duration, until the assignment of the M target data tasks in the target group is completed. The total task duration is obtained by adding the execution times of the assigned target data tasks contained in the corresponding target data shards. The traversal of each group of target data tasks is completed in sequence, and the target group is one of the groups of target data tasks. The process of sequentially executing the target data tasks contained in each of the target data slices through X target threads, according to their respective execution time lengths, until all target data tasks contained in M ​​target data slices have been completed includes: For the target data tasks contained in the current data shard, the X target data tasks with the longest execution time are executed by X target threads respectively. Then, in order of execution time from longest to shortest, the remaining target data tasks contained in the current data shard are sequentially allocated and executed by the target thread that has completed the previous target data task. The current data shard is one of the M target data shards, and is traversed in the order from the first to the Mth target data shards until all the target data tasks contained in the M target data shards are completed.

2. The method according to claim 1, characterized in that, The step of sequentially allocating the M target data tasks of each group to the M target data fragments includes: If a target group contains P target data tasks, the target data task with the longest execution time that is not assigned in the target group is assigned to the target data shard with the shortest total task duration that is not assigned in the P target data shards, until the P target data tasks in the target group are assigned; wherein, P is a positive integer and P is less than M, and the P target data shards are the P target data shards with the shortest total task duration among the M target data shards.

3. The method according to claim 1, characterized in that, The step of assigning the X longest-running target data tasks within the current data shard to X target threads for execution includes: If Y target threads have not completed the execution of the previous target data task, for the target data tasks contained in the current data shard, the XY target data tasks with the longest execution time will be executed by the XY idle target threads respectively; where Y is a positive integer and Y is less than X.

4. The method according to any one of claims 1 to 2, characterized in that, After the method involves sequentially executing the target data tasks contained in each of the target data slices using X target threads, according to their respective execution time lengths, until all target data tasks contained in M ​​target data slices have been completed, the method further includes: Calculate the total execution time of each target thread after completing all the corresponding target data tasks; Based on the total execution time of each target thread, the maximum value is determined to be the total task completion time.

5. A data fragmentation processing device, characterized in that, include: The sorting unit is used to sort multiple target data tasks according to the execution time of each target data task. The data sharding unit is used to group the sorted target data tasks into groups of M in sequence, and then sequentially allocate the M target data tasks of each group to M target data shards; where M is a positive integer; An execution unit is configured to use X target threads to sequentially execute the target data tasks contained in each of the target data slices according to their respective execution time lengths, until all the target data tasks contained in M ​​target data slices have been executed; where X is a positive integer; Specifically, when the data sharding unit sequentially allocates the M target data tasks of each group to the M target data shards, it includes: The target data task that is not assigned in the target group and has the longest corresponding execution time is assigned to the target data shard that is not assigned in the M target data shards and has the shortest corresponding total task duration, until the assignment of the M target data tasks in the target group is completed. The total task duration is obtained by adding the execution times of the assigned target data tasks contained in the corresponding target data shards. The traversal of each group of target data tasks is completed in sequence, and the target group is one of the groups of target data tasks. The execution unit, when used to sequentially execute the target data tasks contained in each of the target data slices according to their respective execution time lengths through X target threads, until all the target data tasks contained in M ​​target data slices have been completed, specifically includes: For the target data tasks contained in the current data shard, the X target data tasks with the longest execution time are executed by X target threads respectively. Then, in order of execution time from longest to shortest, the remaining target data tasks contained in the current data shard are sequentially allocated and executed by the target thread that has completed the previous target data task. The current data shard is one of the M target data shards, and is traversed in the order from the first to the Mth target data shards until all the target data tasks contained in the M target data shards are completed.

6. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Parallel Computer Task Distribution Method and Device

    CN106354552A

  • Task allocation method and device based on distributed system, electronic equipment and storage medium

    CN119557065A

  • Data processing method and device, storage medium and electronic equipment

    CN121807519A