Task allocation method and device based on distributed system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a task allocation method and device based on a distributed system, electronic equipment and a storage medium. The method can be used in the distributed system field. The method comprises the steps of determining a first process number according to a task number in a first task set, and determining a second process number according to computing power resources of a computing terminal; determining a target process number M corresponding to the first task set according to the first process number and the second process number; determining a first predicted processing duration corresponding to each task in the first task set, and sorting each task in the first task set according to the first predicted processing duration in a descending order to obtain a first sorting set; and adding execution marks to the first M tasks in the first sorting set to obtain M marked tasks, and determining to distribute the marked tasks to preset M target processes for processing.

Description

Technical Field

[0001] The present application relates to the field of distributed systems, and in particular to a method, device, electronic device, storage medium, and program product for allocating tasks based on a distributed system. Background Art

[0002] In modern computing and data processing systems, databases are a core component for storing and managing vast amounts of data. As data volumes and access requirements increase, optimizing database system performance becomes increasingly important. However, when multiple tasks or processes access the same database instance simultaneously, performance bottlenecks and resource contention can arise, impacting overall system efficiency.

[0003] However, in related fields, when multiple tasks or processes access the same database instance at the same time, the number of processes cannot be reasonably allocated according to the number of tasks and computing resources, resulting in process congestion and low task processing efficiency. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, electronic device, storage medium and program product for allocating tasks based on a distributed system, so as to reasonably allocate the number of processes and improve the efficiency of task processing.

[0005] In the first aspect, an embodiment of the present application provides a method for allocating tasks based on a distributed system, including: determining a first process number based on the number of tasks in a first task set, and determining a second process number based on the computing power resources of a computing terminal; determining a target process number M corresponding to the first task set based on the first process number and the second process number; determining a first estimated processing time corresponding to each task in the first task set, and sorting each task in the first task set in descending order according to the first estimated processing time to obtain a first sorted set; adding execution marks to the first M tasks in the first sorted set to obtain M marked tasks, and determining to allocate the marked tasks to the preset M target processes for processing.

[0006] In a second aspect, an embodiment of the present application provides a task allocation device based on a distributed system, comprising: a first determination module for determining a first process number based on the number of tasks in a first task set, and determining a second process number based on the computing power resources of a computing terminal; a second determination module for determining a target process number M corresponding to the first task set based on the first process number and the second process number; a sorting module for determining a first estimated processing time corresponding to each task in the first task set, and sorting each task in the first task set in descending order according to the first estimated processing time to obtain a first sorted set; and an allocation module for adding execution tags to the first M tasks in the first sorted set to obtain M marked tasks, and determining to allocate the marked tasks to preset processes for processing. This is to achieve the effect of reasonably allocating the number of processes and improving task processing efficiency.

[0007] In a third aspect, an embodiment of the present application provides a distributed system-based task allocation device, comprising: a memory, a processor;

[0008] The memory stores computer-executable instructions;

[0009] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0010] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.

[0011] The embodiments of the present application provide a method, apparatus, electronic device, storage medium, and program product for allocating tasks based on a distributed system. The method determines the number of first processes that need to be started based on the number of tasks in a first task set. This ensures that there are enough processes to handle all tasks. Next, the number of second processes that can run simultaneously is determined based on the computing resources of the computing terminal (i.e., the computing device). The final target number of processes, M, is determined by combining the first and second numbers of processes. This M value represents the number of processes that can actually be used to process tasks, and is subject to dual constraints of the number of tasks and computing resources. The estimated processing time of each task in the first task set is evaluated and sorted in descending order to form a first sorted set. This is done to prioritize tasks that are expected to take longer to process, thereby reducing their impact on the overall processing time. Execution marks are added to the first M tasks in the sorted first sorted set, and these marked tasks are assigned to the preset M target processes for processing. In this way, each process is ensured to have tasks to execute, and the workload of each process is balanced as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0013] Figure 1 Schematic diagram of the task allocation scenario in the related fields provided for this application;

[0014] Figure 2 Schematic diagram of the distributed system task allocation process provided by this application Figure 1 ;

[0015] Figure 3 Schematic diagram of the distributed system task allocation process provided by this application Figure 2 ;

[0016] Figure 4 A schematic diagram of the structure of a distributed system-based task allocation device provided by this application;

[0017] Figure 5 This is a schematic diagram of the structure of the distributed system-based task allocation device provided in this application.

[0018] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0019] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0020] Figure 1 A schematic diagram of the task allocation scenario in the related fields provided for this application, such as Figure 1As shown, the specific application scenario of the present application is in a distributed architecture, where the database adopts a sharding approach. If the database tasks within the same time period are concentrated on the same database instance, all database processes will compete for the resources (IO and CPU) of the same database instance. When multiple tasks are stuck on one database instance, other database processes executed on this database instance will also be affected. Using only the Peterson algorithm will result in the same database instance only getting one thread resource, which may affect the task execution efficiency. It is necessary to improve the single flag mark of the Peterson algorithm based on the optimal number of processes used for a single instance.

[0021] Based on the above scenarios, it can be seen that in the existing technology, when multiple tasks or processes access the same database instance simultaneously, the number of processes is often not reasonably allocated according to the number of tasks and computing resources, resulting in process congestion and low task processing efficiency.

[0022] The distributed system-based task allocation provided by this application determines the number of first processes that need to be started based on the number of tasks in the first task set. Ensure that there are enough processes to handle all tasks. Then, the number of second processes that can run simultaneously is determined based on the computing power resources of the operation terminal (i.e., the computing device). The final target number of processes M is determined by combining the first and second numbers of processes. This M value is the number of processes that can actually be used to process tasks, which is subject to the dual constraints of the number of tasks and computing power resources. The estimated processing time of each task in the first task set is evaluated and sorted in order from long to short to form a first sorted set. The purpose of this is to give priority to tasks that are expected to take a long time to reduce their impact on the overall processing time. Execution marks are added to the first M tasks in the sorted first sorted set, and these marked tasks will be assigned to the preset M target processes for processing. In this way, it is ensured that each process has tasks to execute and the workload of each process is balanced as much as possible.

[0023] It should be noted that the method and device for allocating tasks based on a distributed system in the present application can be used in the field of distributed systems, and can also be used in any field other than the field of distributed systems. The application field of the method and device for allocating tasks based on a distributed system in the present application is not limited.

[0024] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0025] Figure 2Schematic diagram of the process of allocating tasks based on distributed systems provided by this application Figure 1 ,like Figure 2 As shown, the method includes:

[0026] S201, determining the number of first processes according to the number of tasks in the first task set, and determining the number of second processes according to the computing power resources of the computing terminal;

[0027] First number of processes: This is determined based on the number of tasks in the first task set. Assuming there are T tasks in the task set, the first number of processes N1 can simply be set to T (i.e., one process is assigned to each task).

[0028] Second process number: This is determined based on the computing power resources of the computing terminal. Assuming that the computing power resources of the computing terminal can support R parallel processes, the second process number N2 is R.

[0029] S202: Determine a target number of processes M corresponding to the first task set based on the first number of processes and the second number of processes.

[0030] The target number of processes, M, is the smaller of the first number of processes, N1, and the second number of processes, N2. This is because it cannot exceed the number of tasks or the computing resources of the computing terminal. Therefore, M = min(N1, N2).

[0031] S203: Determine a first estimated processing time corresponding to each task in the first task set, and sort each task in the first task set in descending order according to the first estimated processing time to obtain a first sorted set;

[0032] For each task in the first task set, we determine its estimated processing time P. Then, based on these estimated processing times, we sort the tasks from largest to smallest, resulting in the first sorted set. This prioritizes tasks with the longest estimated processing time, optimizing resource usage and reducing waiting time.

[0033] S204: Add execution marks to the first M tasks in the first sorted set to obtain M marked tasks, and determine to assign the marked tasks to preset M target processes for processing.

[0034] From the sorted task set, the first M tasks (i.e., the tasks with the longest estimated processing time) are selected and marked for execution. These marked tasks are then assigned to the preset M target processes for processing.

[0035] This task scheduling strategy ensures that the most time-consuming tasks are prioritized without exceeding available resources, thereby improving overall efficiency.

[0036] The method for allocating tasks based on a distributed system provided in an embodiment of the present application determines the number of first processes that need to be started based on the number of tasks in the first task set. Ensure that there are enough processes to handle all tasks. Then, determine the number of second processes that can run simultaneously based on the computing power resources of the operation terminal (i.e., the computing device). Combined with the first number of processes and the second number of processes, the final target number of processes M is determined. This M value is the number of processes that can actually be used to process tasks, which is subject to the dual constraints of the number of tasks and computing power resources. The estimated processing time of each task in the first task set is evaluated and sorted in order from long to short to form a first sorted set. The purpose of this is to give priority to tasks that are expected to take a long time to reduce their impact on the overall processing time. Add execution marks to the first M tasks in the sorted first sorted set, and these marked tasks will be assigned to the preset M target processes for processing. In this way, it is ensured that each process has tasks to execute and the workload of each process is balanced as much as possible.

[0037] In an optional embodiment, if the number of tasks is less than a preset threshold, it is determined to directly add an execution mark for each task in the first task set; if the number of tasks is greater than the preset threshold, the number of access objects corresponding to each task in the first task set is determined, wherein the access object is used to indicate the object to be called for processing the task in the computing terminal; and the number of access objects is determined as the first process number.

[0038] The number of tasks is compared against a preset threshold. This threshold is a pre-set value used to determine the size of the task set. This threshold can be set based on system performance and historical data analysis to ensure consistent performance when handling varying numbers of tasks. If the number of tasks is less than the preset threshold, indicating a small number of tasks, each task in the first task set can be marked for execution. In this case, each task can be assigned a separate process, allowing for rapid completion of all tasks. If the number of tasks is greater than the preset threshold, indicating a large number of tasks requiring a more complex processing strategy, the system then determines the number of access objects corresponding to each task in the first task set. Access objects are objects in the computing terminal that are to be called to process the task, such as database connections and file handles. The number of access objects reflects the complexity of the task and the amount of resources required. By defining the number of access objects as the first number of processes, the system can dynamically adjust the number of processes based on the resource requirements of each task. This approach ensures more efficient resource utilization, especially when processing resource-intensive tasks.

[0039] In an optional embodiment, if an idle reminder of any one of the M target processes is received, it is detected whether there is a task update in the first task set, and an updated second task set is obtained, where the idle reminder is used to indicate that there is no pending task on the current process; determine the second estimated processing duration corresponding to each task in the second task set, and sort each task in the second task set in descending order according to the second estimated processing duration to obtain a second sorted set; add an execution mark to the first M tasks in the second sorted set, and determine to assign the marked tasks in the second sorted set to M preset target processes for processing.

[0040] When any target process becomes idle, the system receives an idle reminder. This means that there are no pending tasks on this process and it can receive new tasks. At this time, the system checks whether there is a task update in the first task set. Task updates may include the addition of new tasks, the modification or deletion of existing tasks. An updated second task set is obtained. For each task in the second task set, the system determines its second estimated processing duration. This duration reflects the processing time of the task in the current state. According to the second estimated processing duration, the tasks in the second task set are sorted in descending order to obtain a second sorted set. The purpose of this is to prioritize the tasks with the longest estimated processing time to optimize resource utilization and reduce waiting time. Among the sorted task set, the first M tasks (i.e., the tasks with the longest estimated processing duration) are selected and an execution mark is added to them. These marked tasks will be assigned to M preset target processes for processing.

[0041] In an optional embodiment, the first number of processes is compared with the second number of processes to obtain a comparison result; in the case where the comparison result indicates that the first number of processes is less than the second number of processes, it is determined that the target number of processes M is equal to the first number of processes; in the case where the comparison result indicates that the second number of processes is less than the first number of processes, it is determined that the target number of processes M is equal to the second number of processes; in the case where the comparison result indicates that the first number of processes is equal to the second number of processes, it is determined that the target number of processes M is equal to the first number of processes or the second number of processes.

[0042] The comparison result indicates that the first number of processes is less than the second number of processes: If N1 < N2, then the target number of processes M is equal to the first number of processes N1. This means that the system will limit the number of parallel processes according to the number of tasks. [[ID=X]] [[ID=Y]]

[0043] The comparison result indicates that the second number of processes is less than the first number of processes: If N2 < N1, then the target number of processes M is equal to the second number of processes N2. This means that the system will limit the number of parallel processes according to the computing power resources of the computing terminal. [[ID=z]] [[ID=w]]

[0044] The comparison result indicates that the first process number is equal to the second process number: if N1=N2, then the target process number M can be the first process number N1 or the second process number N2. In this case, the system can select any value as the target process number because they are equal.

[0045] This task scheduling strategy ensures that the most time-consuming tasks are prioritized without exceeding available resources, thereby improving overall efficiency.

[0046] In an optional embodiment, the data volume of each task in the first task set is identified; the number of accesses to the access objects corresponding to each task in the first task set is determined; the data volume is multiplied by a preset first weight to obtain a first index value, and the number of accesses is multiplied by a second weight to obtain a second index value; and a first estimated processing time is determined based on the first index value and the second index value.

[0047] Data size: For each task in the first task set, the system first identifies its data size, which may include the size of the input data, output data, and other related data required for task processing.

[0048] Access objects: In a computing terminal, a task may need to call or access multiple objects (such as databases, files, network resources, etc.). The system counts the number of objects that each task needs to access.

[0049] First weight and second weight: These weights are pre-set based on experience or system configuration and are used to adjust the impact of data size and number of accessed objects on the expected processing time.

[0050] First index value: multiply the data amount by the preset first weight to obtain the first index value.

[0051] Second index value: The second index value is obtained by multiplying the number of visits to the access object by the second weight.

[0052] First estimated processing time: Based on the first and second indicator values, the system can calculate the estimated processing time for each task. This usually involves a formula or algorithm that combines the effects of data volume and access times to provide a comprehensive estimated processing time.

[0053] This strategy provides a more accurate prediction of task processing time by comprehensively considering the data volume and access frequency of the task, thereby helping the system allocate resources and schedule tasks more efficiently.

[0054] In an optional embodiment, the first indicator value and the second indicator value are added to obtain a third indicator value; the third indicator value is matched with the historical indicator value pre-stored in the historical database to obtain a matching result, wherein the historical database at least includes: multiple historical indicator values, multiple historical processing times, and a mapping relationship between the multiple historical indicator values ​​and the multiple historical processing times; the historical time indicated by the matching result is determined as the first estimated processing time.

[0055] The third index value is obtained by adding the first and second index values. This index value is a metric that combines the task data size and the number of accessed objects, and is used to assess the complexity and resource requirements of the task.

[0056] Historical Database: The system maintains a historical database that stores multiple historical metric values ​​and their corresponding historical processing times. This data is collected from past task executions and reflects the relationship between different metric values ​​and actual processing times.

[0057] Matching results: The calculated third indicator value is matched with the pre-stored historical indicator values ​​in the historical database. The matching process may involve finding the closest indicator value or using some interpolation method to determine the most appropriate historical period.

[0058] Historical duration: Based on the matching results, the system can determine a historical duration, which is the actual processing time of the task under similar conditions in the past. This historical duration is used as the first estimated processing time for the current task.

[0059] This strategy can improve scheduling accuracy and efficiency by leveraging historical data to predict task processing times. It is particularly suitable for environments with repetitive or similar tasks because historical data provides reference information.

[0060] In an optional embodiment, the marking status of each task in the first task set is identified, wherein the marking status includes: marked, unmarked; if there is a task in the first task set whose corresponding marking status is unmarked, the unmarked task is added to a waiting queue, wherein none of the tasks in the waiting queue are assigned a process.

[0061] Marking Status: Each task has a marking status, which can be "marked" or "unmarked". This status indicates whether the task has been selected by the system for further processing or assignment.

[0062] Unmarked tasks: The system checks the tasks in the first task set to determine which tasks have a marking status of "unmarked". These tasks have not yet been assigned a process and therefore require special attention.

[0063] Waiting queue: All unmarked tasks are added to a waiting queue. This queue is a special data structure used to store tasks that have not yet been assigned a process.

[0064] This strategy ensures that only unmarked tasks are considered for further processing or assignment, thus avoiding duplicate processing or mistakenly skipping certain tasks. By maintaining a waiting queue, the system can process all pending tasks in an orderly manner, improving the efficiency and accuracy of scheduling.

[0065] Figure 3 Schematic diagram of the distributed system task allocation process provided by this application Figure 2 ,like Figure 3 As shown, this embodiment Figure 2 Based on the embodiment, the allocation of tasks based on the distributed system is described in detail. The method includes:

[0066] First, the system identifies the task and determines whether it belongs to the same database instance. If there is overlap, the complexity is increased by one.

[0067] Routing: Determines which database instance the task is assigned to based on the task's routing information.

[0068] Calculating the optimal number of processes to use: For each database instance, the system calculates the optimal number of processes to use.

[0069] Calculate task complexity (non-labeled): In the same database instance, if there are other tasks that overlap with the current task, the complexity is increased by one.

[0070] Task sorting (non-labeling): Sort tasks from smallest to largest according to their complexity.

[0071] Calculate the optimal number of task processes: Start selecting based on the sorted tasks. For every task that does not overlap with the selected task, the number of task processes increases by one.

[0072] Best Line Program: Calculates the best line program and then executes the marked tasks.

[0073] Actual number of processes: Mark the sorted processes according to the actual number of processes.

[0074] Calculate minimum value: Finally, the system calculates the minimum value based on the actual number of processes to optimize resource usage.

[0075] The number of first processes that need to be started is determined based on the number of tasks in the first task set. Ensure that there are enough processes to handle all tasks. Then, the number of second processes that can run simultaneously is determined based on the computing power resources of the operation terminal (i.e., the computing device). The final target number of processes M is determined by combining the first and second numbers of processes. This M value is the number of processes that can actually be used to process tasks, which is subject to the dual constraints of the number of tasks and computing power resources. The estimated processing time of each task in the first task set is evaluated and sorted in order from long to short to form a first sorted set. The purpose of this is to give priority to tasks that are expected to take a long time to reduce their impact on the overall processing time. Execution marks are added to the first M tasks in the sorted first sorted set, and these marked tasks will be assigned to the preset M target processes for processing. In this way, it is ensured that each process has tasks to execute and the workload of each process is balanced as much as possible.

[0076] Figure 4 This is a schematic diagram of the structure of the distributed system-based task allocation device provided by this application, such as Figure 4 As shown, the distributed system-based task allocation 40 provided in this embodiment includes:

[0077] A first determining module 401 is configured to determine the first number of processes according to the number of tasks in the first task set, and determine the second number of processes according to the computing power resources of the computing terminal;

[0078] A second determining module 402 is configured to determine a target number M of processes corresponding to the first task set based on the first number of processes and the second number of processes;

[0079] a sorting module 403 configured to determine a first estimated processing time corresponding to each task in the first task set, and sort each task in the first task set in descending order according to the first estimated processing time to obtain a first sorted set;

[0080] The allocation module 404 is configured to add execution marks to the first M tasks in the first sorted set to obtain M marked tasks, and determine to allocate the marked tasks to a preset process for processing.

[0081] In one possible implementation, the first determination module 401 is further used to determine whether to directly add an execution mark for each task in the first task set if the number of tasks is less than a preset threshold; if the number of tasks is greater than the preset threshold, determine the number of access objects corresponding to each task in the first task set, wherein the access objects are used to indicate the objects to be called for processing tasks in the computing terminal; and determine the number of access objects as the first process number.

[0082] In one possible implementation, the allocation module 404 is also used to detect whether there is a task update in the first task set if an idle reminder of any process among the M target processes is received, and obtain an updated second task set, wherein the idle reminder is used to indicate that there are no pending tasks on the current process; determine the second estimated processing time corresponding to each task in the second task set, and sort each task in the second task set in descending order according to the second estimated processing time to obtain a second sorted set; add execution marks to the first M tasks in the second sorted set, and determine to allocate the marked tasks in the second sorted set to the preset M target processes for processing.

[0083] In one possible implementation, the second determination module 402 is further used to compare the first process number with the second process number to obtain a comparison result; when the comparison result indicates that the first process number is less than the second process number, determine that the target process number M is equal to the first process number; when the comparison result indicates that the second process number is less than the first process number, determine that the target process number M is equal to the second process number; when the comparison result indicates that the first process number is equal to the second process number, determine that the target process number M is equal to the first process number or the second process number.

[0084] In one possible implementation, the sorting module 403 is also used to identify the data size of each task in the first task set; determine the number of accesses to the access object corresponding to each task in the first task set; multiply the data size by a preset first weight to obtain a first index value, and multiply the number of accesses by a second weight to obtain a second index value; and determine a first estimated processing time based on the first index value and the second index value.

[0085] In one possible implementation, the sorting module 403 is further used to add the first indicator value and the second indicator value to obtain a third indicator value; match the third indicator value with the historical indicator value pre-stored in the historical database to obtain a matching result, wherein the historical database at least includes: multiple historical indicator values, multiple historical processing times, and a mapping relationship between the multiple historical indicator values ​​and the multiple historical processing times; and determine the historical time indicated by the matching result as the first estimated processing time.

[0086] In one possible implementation, the allocation module 404 is also used to identify the marking status of each task in the first task set, wherein the marking status includes: marked and unmarked; if there is a task in the first task set whose corresponding marking status is unmarked, the unmarked task is added to a waiting queue, wherein no process is assigned to the tasks in the waiting queue.

[0087] The distributed system-based task allocation device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.

[0088] Figure 5 This is a schematic diagram of the structure of the distributed system-based task distribution device provided by this application. Figure 5 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0089] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that the at least one processor 501 performs the above method.

[0090] The specific implementation process of the processor 501 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0091] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0092] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0093] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0094] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0095] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0096] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0097] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0098] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0099] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0100] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0101] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0102] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0103] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A method for allocating tasks based on a distributed system, characterized in that: include: Determine the first number of processes according to the number of tasks in the first task set, and determine the second number of processes according to the computing power resources of the computing terminal; Determine a target number of processes M corresponding to the first task set according to the first number of processes and the second number of processes; Determine a first estimated processing time corresponding to each task in the first task set, and sort each task in the first task set in descending order according to the first estimated processing time to obtain a first sorted set; Add execution marks to the first M tasks in the first sorted set to obtain M marked tasks, and determine to allocate the marked tasks to preset M target processes for processing.

2. The method according to claim 1, characterized in that Determining the first number of processes according to the number of tasks in the first task set includes: If the number of tasks is less than a preset threshold, determining to directly add an execution mark to each task in the first task set; If the number of tasks is greater than a preset threshold, determining the number of access objects corresponding to each task in the first task set, wherein the access objects are used to indicate objects to be called for processing tasks in the computing terminal; The number of access objects is determined as the first process number.

3. The method according to claim 1, characterized in that Before adding execution marks to the first M tasks in the first sorted set, the method further includes: If an idle reminder of any process among the M target processes is received, detecting whether there is a task update in the first task set, and obtaining an updated second task set, wherein the idle reminder is used to indicate that there is no pending task on the current process; Determine a second estimated processing time corresponding to each task in the second task set, and sort each task in the second task set in descending order according to the second estimated processing time to obtain a second sorted set; Add execution marks to the first M tasks in the second sorted set, and determine to allocate the marked tasks in the second sorted set to preset M target processes for processing.

4. The method according to claim 1, characterized in that: Determining a target number of processes M corresponding to the first task set according to the first number of processes and the second number of processes includes: Comparing the first process number with the second process number to obtain a comparison result; When the comparison result indicates that the first process number is less than the second process number, determining that the target process number M is equal to the first process number; When the comparison result indicates that the second process number is less than the first process number, determining that the target process number M is equal to the second process number; In the case where the comparison result indicates that the first process number is equal to the second process number, it is determined that the target process number M is equal to the first process number or the second process number.

5. The method according to claim 1, characterized in that The determining a first estimated processing time corresponding to each task in the first task set includes: Identify the data size of each task in the first task set; Determine the number of accesses to the access object corresponding to each task in the first task set; The data volume is multiplied by a preset first weight to obtain a first index value, and the number of accesses is multiplied by a second weight to obtain a second index value; A first estimated processing time is determined according to the first indicator value and the second indicator value.

6. The method according to claim 5, characterized in that The determining a first estimated processing time according to the first indicator value and the second indicator value includes: Adding the first index value and the second index value to obtain a third index value; Matching the third indicator value with the historical indicator value pre-stored in the historical database to obtain a matching result, wherein the historical database at least includes: a plurality of historical indicator values, a plurality of historical processing durations, and a mapping relationship between the plurality of historical indicator values ​​and the plurality of historical processing durations; The historical duration indicated by the matching result is determined as the first estimated processing duration.

7. The method according to claim 1, characterized in that After adding execution marks to the first M tasks in the first sorted set, the method further includes: Identify a marking state of each task in the first task set, wherein the marking state includes: marked, unmarked; If there is a task in the first task set whose corresponding mark state is unmarked, the unmarked task is added to a waiting queue, wherein no process is assigned to the tasks in the waiting queue.

8. A task allocation device, characterized in that: include: A first determination module, configured to determine the first number of processes according to the number of tasks in the first task set, and determine the second number of processes according to the computing power resources of the computing terminal; A second determining module, configured to determine a target number of processes M corresponding to the first task set according to the first number of processes and the second number of processes; a sorting module, configured to determine a first estimated processing time corresponding to each task in the first task set, and sort each task in the first task set in descending order according to the first estimated processing time to obtain a first sorted set; The allocation module is used to add execution marks to the first M tasks in the first sorted set to obtain M marked tasks, and determine to allocate the marked tasks to a preset process for processing.

9. A task distribution device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images