Task allocation method, electronic equipment and computer readable storage medium
By splitting the tasks in the distributed system into periodic and interactive tasks, and being processed by the timed task service module and the back-end business service module, the problem of unreasonable task allocation is solved and more efficient task allocation and resource utilization is achieved.
Patent Information
- Application Number
- CN202510264857.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-06
- Publication Date
- 2025-08-01
AI Technical Summary
In the prior art, task allocation in distributed system clusters relies on designated service nodes, which makes it difficult to allocate tasks in a timely and reasonable manner, affecting system efficiency.
The pending tasks are split into periodic tasks and interactive tasks, and are processed by the timed task service module and the back-end business service module respectively, and cache middleware is used to optimize information acquisition and resource allocation.
It improves the rationality and efficiency of task allocation, reduces the overhead of information acquisition, optimizes system resource utilization, and enhances the scalability and flexibility of distributed systems.
Smart Images
Figure CN120407147A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of data processing, and in particular, to a task allocation method, an electronic device, and a computer-readable storage medium. Background Art
[0002] A distributed system cluster includes multiple service nodes. In the prior art, it is usually to use a specified service node in the distributed system cluster to allocate tasks of a fixed type to other service nodes, resulting in the task allocation being completely dependent on the specified service node, and the specified service node itself also bears a large number of tasks, so that the tasks are difficult to be allocated in a timely and reasonable manner. In view of this, how to improve the rationality of task allocation has become an urgent problem to be solved. Summary of the Invention
[0003] The main technical problem to be solved by this application is to provide a task allocation method, an electronic device, and a computer-readable storage medium, which can improve the rationality of task allocation.
[0004] To solve the above technical problem, a first aspect of this application provides a task allocation method, including: obtaining a to-be-processed task, and determining a periodic task and an interactive task that match the to-be-processed task; using the timed task service module to process the periodic task, and using the backend service module to process the interactive task.
[0005] To solve the above technical problem, a second aspect of this application provides an electronic device, including a memory and a processor coupled to each other, wherein program instructions are stored in the memory, and the processor is configured to execute the program instructions to implement the method described in the first aspect above.
[0006] To solve the above technical problem, a third aspect of this application provides a computer-readable storage medium, storing program instructions that can be run by a processor, and the program instructions are used to implement the method described in the first aspect above.
[0007] In the above solution, after obtaining the to-be-processed task, the to-be-processed task is split into a periodic task and an interactive task, and is allocated to the timed task service module and the backend service module. The timed task service module is used to process the periodic task, and the backend service module is used to process the interactive task. In the distributed system environment, by dividing the to-be-processed task into a periodic task and an interactive task and allocating them to the corresponding modules for processing, the system realizes the reasonable allocation of the to-be-processed task, thereby improving the rationality of task allocation. Brief Description of the Drawings
[0008] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings. Among them:
[0009] Figure 1 It is a schematic flowchart of an implementation manner of the task allocation method of the present application;
[0010] Figure 2 It is a schematic structural diagram of a service node in a distributed system cluster of the present application corresponding to an implementation manner;
[0011] Figure 3 It is a schematic flowchart of another implementation manner of the task allocation method of the present application;
[0012] Figure 4 It is a schematic structural diagram of an electronic device of the present application corresponding to an implementation manner;
[0013] Figure 5 It is a schematic structural diagram of a computer-readable storage medium of the present application corresponding to an implementation manner. Specific Embodiments
[0014] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments, and adaptive combinations can be made between different implementation manners. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.
[0015] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article only describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after. In addition, "multiple" in this article means two or more than two.
[0016] Please refer to Figure 1 , Figure 1 It is a schematic flowchart of an implementation manner of the task allocation method of the present application. The task allocation method is applied to a distributed system cluster, and the distributed system cluster includes a corresponding number of service nodes. The service node corresponds to a timed task service module and a backend business service module. Among them, the task allocation method includes:
[0017] S101: Obtain the task to be processed and determine the periodic tasks and interactive tasks that the task to be processed matches.
[0018] Specifically, after obtaining the task to be processed, split the task to be processed into periodic tasks and interactive tasks.
[0019] It should be noted that based on the complex business scenarios in the distributed system, after the system cluster obtains the task to be processed, it will split some periodically executed tasks included in the task to be processed, such as cleaning records regularly every day, updating status records regularly, backing up information regularly, etc., into corresponding periodic tasks, and split some backend business service tasks included in the task to be processed, such as external request functions, information interaction functions related to operations with the using object, etc., into corresponding interactive tasks. Among them, the periodic tasks correspond to periodically executed data processing tasks, and the interactive tasks correspond to backend business service tasks with information interaction functions, and the execution priority of the periodic tasks is lower than that of the interactive tasks.
[0020] In one application method, after obtaining the task to be processed, split the task to be processed based on the trigger mechanism to determine the corresponding periodic tasks and interactive tasks.
[0021] In one application scenario, the trigger mechanism corresponding to the periodic tasks is triggered at fixed time intervals, such as timing, polling, etc., while the trigger mechanism corresponding to the interactive tasks is triggered in real time by the using object or external events, such as clicking, API calls, etc.
[0022] In another application method, after obtaining the task to be processed, split the task to be processed based on the execution priority to determine the corresponding periodic tasks and interactive tasks.
[0023] In one application scenario, the periodic tasks correspond to low-priority tasks, such as tasks that need to be executed during the idle period of the system, such as data archiving, historical record cleaning, etc., while the interactive tasks correspond to high-priority tasks, such as operation tasks that need to be executed immediately, such as real-time search requests by the using object, instant message push, etc.
[0024] S102: Use the scheduled task service module to process the periodic tasks and use the backend business service module to process the interactive tasks.
[0025] Specifically, assign the periodic tasks to the scheduled task service module and use the scheduled task service module to process the periodic tasks, and assign the interactive tasks to the backend business service module and use the backend business service module to process the interactive tasks.
[0026] In an application mode, obtain the task types corresponding to the periodic tasks and interactive tasks respectively, use the timed task service module to process the periodic tasks based on the task types, and use the back-end business service module to process the interactive tasks based on the task types.
[0027] In some application scenarios, the task types corresponding to the periodic tasks include timed record cleaning, timed status record updating, timed information backup, etc., and the task types corresponding to the interactive tasks include external request functions, information interaction functions related to the operations of the using objects, etc.
[0028] In another application mode, obtain the resource usage information required for the periodic tasks and interactive tasks respectively, use the timed task service module to process the periodic tasks based on the resource usage information, and use the back-end business service module to process the interactive tasks based on the resource usage information.
[0029] In a specific application scenario, please refer to Figure 2 , Figure 2 is a schematic structural diagram of a service node corresponding to an embodiment in the distributed system cluster of the present application. The service node is correspondingly split into a timed task service module and a back-end business service module, and these two types of service modules operate independently in each service node of the cluster, and each supports horizontal expansion of multiple replicas. Each replica can correspondingly process the corresponding tasks, so as to be able to well handle the scenario of a sharp increase in subsequent tasks to be processed.
[0030] For the above solution, after obtaining the tasks to be processed, split the tasks to be processed into periodic tasks and interactive tasks, allocate them to the timed task service module and the back-end business service module, use the timed task service module to process the periodic tasks, and use the back-end business service module to process the interactive tasks. In the distributed system environment, by dividing the tasks to be processed into periodic tasks and interactive tasks and allocating them to the corresponding modules for processing, the system realizes the reasonable allocation of the tasks to be processed, thereby improving the rationality of task allocation.
[0031] Please refer to Figure 3 , Figure 3 is a schematic flowchart of another embodiment of the task allocation method of the present application. Among them, the task allocation method includes:
[0032] S301: Obtain the tasks to be processed, and determine the periodic tasks and interactive tasks that match the tasks to be processed.
[0033] Specifically, after obtaining the tasks to be processed, split the tasks to be processed into periodic tasks and interactive tasks.
[0034] S302: Obtain the registration information uploaded by the scheduled task service module at the first preset time interval by using the cache middleware module; wherein, the registration information at least includes the identification number and the resource usage information of the service node.
[0035] Specifically, the distributed system cluster further includes a cache middleware module, which is used to obtain the registration information uploaded by the scheduled service module at the first preset time interval. The registration information includes the identification number of the service node, the resource usage information of the service node, and the IP address of the service node.
[0036] S303: Obtain and store the task information corresponding to the periodic tasks in the distributed system cluster at the second preset time interval by using the scheduled task service module; wherein, the task information includes the number of databases included in the periodic task, the names of the databases, and the names of the tables to be processed included in the databases.
[0037] Specifically, the scheduled task service module obtains the task information corresponding to the periodic tasks in the distributed system cluster at the second preset time interval and stores the obtained task information. The task information includes the number of databases included in the periodic task, the names of the databases, and the names of the tables to be processed included in the databases. By setting up a cache middleware in the distributed system cluster, the distributed system can directly obtain the registration information from the cache middleware without querying the service node every time, which greatly reduces the time overhead of information acquisition. And by pre-storing the task information corresponding to the periodic tasks through the scheduled task service module, the distributed system can timely allocate the periodic tasks to the corresponding scheduled task service modules according to the task information, thereby improving the efficiency of task allocation.
[0038] In an implementation scenario, the cache middleware module is a redis cache component. In other implementation scenarios, other types of cache components such as Memcached and Ehcache can also be used as the cache middleware module. This application does not make specific restrictions on this.
[0039] Optionally, the first preset time interval and the second preset time interval can be determined according to the actual situation. For example, the first preset time interval is 3 minutes and the second preset time interval is 2 minutes. This application does not make specific restrictions on this.
[0040] In a specific implementation scenario, after the distributed system is started, the timed task service module first obtains the IP addresses of the service nodes, generates a unique identification number for each service node through hashing based on the IP address values, and obtains the resource usage information of the corresponding service nodes through the detection component. The resource usage information includes the CPU resource utilization rate, memory resource utilization rate, disk read / write utilization rate, etc. of the service nodes. After obtaining the above information, it will be organized into a string of type string, and after generating the corresponding registration information, it will be uploaded to the redis cache component at a time interval of 3 minutes. Each timed task service module will obtain the number of databases (a total of four), the names of the databases (such as DB1, DB4), and the names of the tables to be processed in the databases (such as base_file_1, base_file_127) included in the periodic tasks at a time interval of 2 minutes, and store the above task information in the memory of the timed task service module itself in the form of a list, such as {DB1.base_file_1}.
[0041] S304: Use the timed task service module to process periodic tasks and use the backend business service module to process interactive tasks.
[0042] Specifically, use the timed task service module to process periodic tasks and use the backend business service module to process interactive tasks.
[0043] In one implementation, using the timed task service module to process periodic tasks in step S304 specifically includes: obtaining all timed task service modules that match the registration information; obtaining the tables to be processed in all databases that match the task information; allocating the tables to be processed in the databases to the timed task service module, and using the timed task service module to process the tables to be processed.
[0044] Specifically, obtain all corresponding timed task service modules according to the registration information stored in the cache middleware module, and obtain the tables to be processed in all matching databases according to the task information stored in the timed task service module. Then, use the timed task service module to process the allocated tables to be processed. Before processing the tables to be processed, first perform information screening and matching through the cache middleware module and the timed task service module, reducing invalid queries and accesses to the database and optimizing the utilization of system resources.
[0045] In one implementation scenario, after obtaining all timed task service modules that match the registration information, it further includes: in response to at least one registration information in the cache middleware module expiring within the first preset time interval, determining that the timed task service module matching the expired registration information has failed; updating all timed task service modules.
[0046] Specifically, when at least one piece of registration information stored in the cache middleware module becomes invalid within the first preset time interval, it is determined that the timed task service module matching the invalid registration information has failed. At this time, the failed timed task service module is in an unavailable state. Update all matching timed task service modules according to the remaining valid registration information. The failed registration information can quickly determine the corresponding faulty timed task service module, which helps to quickly locate the problem points in the distributed system, reduce the system crashes caused by faulty modules, and update all matching timed task service modules according to the remaining valid registration information, which can enable the distributed system to dynamically adjust the task allocation strategy, thereby helping to optimize resource utilization, ensure that subsequent periodic tasks can be assigned to available timed task service modules, and improve the processing efficiency and throughput of the distributed system.
[0047] In another implementation scenario, after obtaining all timed task service modules that match the registration information, it further includes: in response to new registration information being added to the cache middleware module within the first preset time interval, determining that a new timed task service module has been added to the distributed system cluster; where the newly added timed task service module matches the newly added registration information; updating all timed task service modules.
[0048] Specifically, when new registration information is added to the cache middleware module within the first preset time interval, it is determined that a corresponding new timed task service module has been added to the distributed system cluster. At this time, the newly added timed task service module is in an available state. Update all matching timed task service modules according to all valid registration information. The processing mechanism for the newly added registration information enables the distributed system to adaptively respond to changes in the cluster. When a new timed task service module joins, the distributed system can automatically identify and include it in the scheduling scope, thereby enhancing the scalability and flexibility of the distributed system.
[0049] In one implementation manner, weights are correspondingly set for the resource usage information of the service nodes. The steps of allocating the table to be processed in the database to the timed task service module and using the timed task service module to process the table to be processed specifically include: allocating the table to be processed in the database to all timed task service modules based on the resource usage information and its corresponding weights; using the timed task service module to process the allocated table to be processed.
[0050] Specifically, each service node corresponds to its own resource usage information, and weights are set for each resource usage information. After calculating the resource operation statistical value of each service node based on each resource usage information and its corresponding weight, the table to be processed is allocated to all scheduled task modules according to the resource operation statistical value, and the scheduled task service module is used to process the allocated table to be processed. The distributed system can dynamically adjust the task allocation according to the resource usage of the service nodes to achieve load balancing. When the load of a certain node approaches or reaches the upper limit, the system will reduce the tasks allocated to it; while when the load of a certain node is low, the system will increase the tasks allocated to it, thereby further improving the rationality of task allocation.
[0051] It can be understood that the larger the resource operation statistical value, the worse the performance of the corresponding service node, that is, the closer the load is to the upper limit.
[0052] In an implementation scenario, the table to be processed corresponds to a task type. The steps of using the scheduled task service module to process the allocated table to be processed specifically include: grouping and sorting the allocated table to be processed based on the task type; using the scheduled task service module to process the table to be processed in sequence based on the grouping and sorting.
[0053] Specifically, the allocated table to be processed is grouped and sorted according to the task type corresponding to the table to be processed, and the scheduled task service module is used to process the table to be processed in sequence according to the grouping and sorting. Through the grouping and sorting, tasks of the same type can be processed together, which is convenient for batch processing, improves the task processing efficiency, and the grouping and sorting also helps to achieve dynamic resource allocation and flexibly adjust the resource allocation strategy, thereby improving the overall resource utilization efficiency of the system.
[0054] Optionally, the task types corresponding to the table to be processed include the following categories: regularly updating the status of unfinished tasks, regularly synchronizing the database records of resource usage amounts, regularly backing up files and updating database records, regularly cleaning up database log record information, etc., and a running cycle is also defined and declared for each task type, which can be achieved by defining the execution cron expression.
[0055] In a specific implementation scenario, there are 3 service nodes in the distributed system cluster, and 4 databases, namely DB1, DB2, DB3, and DB4. Each database contains 128 tables to be processed. Moreover, a timed task service module and a backend business service module are deployed corresponding to each service node. Among them, the backend business service module is used to process interaction tasks. Each service node has its own resource usage information, such as the CPU resource usage rate, the memory resource usage rate, and the disk read / write usage rate. And a weight ratio is introduced. For example, the weight ratio of the CPU is 40%, the memory resource usage rate is 30%, and the disk read / write usage rate is 30%. According to the above information, the resource operation statistical value of each service node is calculated. When the resource operation statistical values of each service node calculated are the same, all timed task service modules follow the principle of uniform distribution. The first timed task service module gets 43 tables to be processed from each database, totaling 172 tables. The second timed task service module gets 43 tables to be processed from each database, totaling 172 tables. The third timed task service module gets 42 tables to be processed from each database, totaling 168 tables. After the timed task service module gets the tables to be processed, the tables to be processed are grouped and sorted according to the task type and the tables to be processed are processed in sequence according to the grouping and sorting. Specifically, it can be manifested as operations such as adding, updating, and deleting database records, thereby completing the execution of periodic tasks.
[0056] Furthermore, when a service node in the distributed system cluster fails, for example, the second service node fails and becomes unavailable. Through the detection mechanism of the redis cache component, it is confirmed that the registration information of the corresponding second timed task service module becomes invalid, thereby confirming that only the first timed task service module and the third timed task service module are in an available state in this distributed system cluster. At this time, the first timed task service module gets 64 tables to be processed from each database, totaling 256 tables. The third timed task service module gets 64 tables to be processed from each database, totaling 256 tables. When a new service node is added to the distributed system cluster, for example, a fourth service node is added. Through the detection mechanism of the redis cache component, it is confirmed that the registration information of the corresponding fourth timed task service module is valid, thereby confirming that there are a total of four timed task service modules in an available state in this distributed system. At this time, the first timed task service module gets 32 tables to be processed from each database, totaling 128 tables. The second timed task service module gets 32 tables to be processed from each database, totaling 128 tables. The third timed task service module gets 32 tables to be processed from each database, totaling 128 tables. The fourth timed task service module gets 32 tables to be processed from each database, totaling 128 tables.
[0057] In an implementation scenario, after step S102, it further includes: recording the key information generated during the process of the timing task service module processing periodic tasks into the log.
[0058] Specifically, during the process of the timing task service module processing periodic tasks, some key information will be generated, and these key information will be recorded into the log for the using object to view.
[0059] Optionally, the generated key information may be the execution duration used for processing periodic tasks, the amount of data processed in this task, etc., and the present application does not make specific limitations on this.
[0060] Please refer to Figure 4 , Figure 4 FIG. is a schematic structural diagram of an implementation manner of an electronic device according to the present application. The electronic device 40 includes a memory 400 and a processor 402 that are coupled to each other. Among them, the memory 400 stores program data (not shown in the figure), and the processor 402 calls the program data to implement the method in any of the above embodiments. For the description of related content, please refer to the detailed description of the above method embodiments, and details will not be repeated here. Specifically, the electronic device 40 includes but is not limited to: desktop computers, laptop computers, tablet computers, servers, etc., and no limitation is made here. In addition, the processor 402 may also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 402 may be an integrated circuit chip with signal processing capabilities. The processor 402 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Additionally, the processor 402 may be implemented jointly by integrated circuit chips.
[0061] Please refer to Figure 5 , Figure 5 FIG. is a schematic structural diagram of an implementation manner of a computer-readable storage medium according to the present application. The computer-readable storage medium 50 stores program data 500, and when the program data 500 is executed by the processor, it implements the method in any of the above embodiments. For the description of related content, please refer to the detailed description of the above method embodiments, and details will not be repeated here.
[0062] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0063] In addition, each functional unit in various embodiments of the present application may be integrated into a processing unit, may exist separately as individual physical units, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
[0064] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs and other various media that can store program codes.
[0065] The above is only the embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.
Claims
1. A task allocation method, characterized in that, Applied to a distributed system cluster, the distributed system cluster includes a corresponding number of service nodes, and each service node corresponds to a timing task service module and a backend business service module. The method includes: Obtain a task to be processed, and determine the periodic task and interactive task that the task to be processed matches; Use the timing task service module to process the periodic task, and use the backend business service module to process the interactive task.
2. The method according to claim 1, wherein The distributed system cluster further includes a cache middleware module. Before using the timing task service module to process the periodic task and using the backend business service module to process the interactive task, it includes: Use the cache middleware module to obtain the registration information uploaded by the timing task service module at a first preset time interval; wherein, the registration information at least includes the identification number of the service node and the resource usage information of the service node; Use the timing task service module to obtain and store the task information corresponding to the periodic task in the distributed system cluster at a second preset time interval; wherein, the task information includes the number of databases included in the periodic task, the names of the databases, and the names of the tables to be processed included in the databases.
3. The method according to claim 2, characterized in that The using the timing task service module to process the periodic task includes: Obtain all the timing task service modules that match the registration information; Obtain all the tables to be processed in the databases that match the task information; Allocate the tables to be processed in the databases to all the timing task service modules, and use the timing task service modules to process the tables to be processed.
4. The method according to claim 3, wherein Weights are correspondingly set for the resource usage information of the service nodes. The allocating the tables to be processed in the databases to all the timing task service modules and using the timing task service modules to process the tables to be processed includes: Allocate the tables to be processed in the databases to all the timing task service modules based on the resource usage information and its corresponding weights; Use the timing task service modules to process the allocated tables to be processed.
5. The method according to claim 4, characterized in that, The tables to be processed correspond to task types. The using the timing task service modules to process the allocated tables to be processed includes: Group and sort the allocated tables to be processed based on the task types; Use the timing task service modules to process the tables to be processed in sequence based on the grouping and sorting.
6. The method according to claim 3, wherein After obtaining all the timing task service modules that match the registration information, it further includes: In response to at least one of the registration information in the cache middleware module expiring within the first preset time interval, determine that the timing task service module that matches the expired registration information has a fault; Update all the timing task service modules.
7. The method according to claim 3, wherein After obtaining all the timing task service modules that match the registration information, it further includes: In response to the addition of the registration information in the cache middleware module within the first preset time interval, it is determined that a new timing task service module is added to the distributed system cluster; wherein, the newly added timing task service module matches the newly added registration information; Update all the timing task service modules.
8. The method according to claim 1, wherein After using the timing task service module to process the periodic task and using the backend business service module to process the interaction task, it further includes: Record the key information generated during the process of the timing task service module processing the periodic task into the log.
9. An electronic device, characterized in that, It includes a mutually coupled memory and a processor, and program instructions are stored in the memory, and the processor is used to execute the program instructions to implement the method according to any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, Stored with program instructions that can be run by a processor, and the program instructions are used to implement the method according to any one of claims 1-8.