Task processing method and device and electronic equipment

Abstract

The invention relates to a task processing method and device and electronic equipment, and relates to the technical field of data processing.The method comprises the steps that firstly, the task processing state of each computing node in a distributed system is monitored, and the computing nodes in the distributed system are different computer equipment; different computing nodes can read the business data in the preset storage position; determining a target computing node of which the task processing state is an idle state; and then, according to the task parameters of the current to-be-executed task, sending a task execution instruction to the target computing node, so that the target computing node executes the current to-be-executed task based on business data corresponding to the task parameters in the preset storage position. By applying the technical scheme of the invention, a distributed system for realizing free scheduling of asynchronous tasks is provided, the problem of shortage of single-machine computing resources can be effectively improved, and the asynchronous task processing efficiency is improved.

Description

Task processing method, device and electronic equipment Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a task processing method, device and electronic device. Background Art

[0002] With the advent of the era of mobile Internet, cloud computing, and big data, the amount of information tasks that need to be processed is increasing. Among them, asynchronous task processing has always been a concern for developers.

[0003] Currently, various asynchronous task processing frameworks can only meet the needs of asynchronous task processing on a single machine, such as an asynchronous task processing system that runs multiple computing nodes on a single machine by calling multiple threads.

[0004] However, stacking a large number of asynchronous tasks on a single machine can strain computing resources. Furthermore, due to the performance limitations of a single machine, the speed of task processing is limited, which in turn affects the efficiency of asynchronous task processing.

[0005] Summary of the Invention

[0006] In view of this, the present application provides a task processing method, device and electronic device, the main purpose of which is to improve the technical problem in the current existing technology that a large number of asynchronous tasks are uniformly stacked on a single machine to run, which not only makes computing resources tight, but also affects the efficiency of asynchronous task processing.

[0007] In a first aspect, the present application provides a task processing method, comprising:

[0008] Monitoring the task processing status of each computing node in the distributed system, wherein the computing nodes in the distributed system are different computer devices, and different computing nodes can read business data in a preset storage location;

[0009] Determine a target computing node whose task processing state is an idle state;

[0010] Get the task parameters of the current task to be executed;

[0011] According to the task parameters, a task execution instruction is sent to the target computing node, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.

[0012] In a second aspect, the present application provides a task processing method, applied to a first computer device, the method comprising:

[0013] Sending the current task processing status of the first computer device to the second computer device;

[0014] receiving a task execution instruction sent by the second computer device, wherein the task execution instruction is sent by the second computer device when the second computer device determines that the task processing state is an idle state, and the task execution instruction carries task parameters of the current task to be executed;

[0015] Execute the current task to be executed according to the task parameters.

[0016] In a third aspect, the present application provides a task processing method, applied to a second computer device, the method comprising:

[0017] Requesting the first computer device to obtain the current task processing status of the first computer device;

[0018] If the task processing state is an idle state, then obtaining the task parameters of the task to be executed currently;

[0019] A task execution instruction is sent to the first computer device according to the task parameters, where the task execution instruction is used to instruct the first computer device to execute the current task to be executed according to the task parameters.

[0020] In a fourth aspect, the present application provides a task processing device, comprising:

[0021] a monitoring module configured to monitor a task processing status of each computing node in a distributed system, wherein the computing nodes in the distributed system are different computer devices, and each computing node can read business data in a preset storage location;

[0022] A determination module is configured to determine a target computing node whose task processing state is an idle state;

[0023] An acquisition module is configured to obtain task parameters of a task to be executed;

[0024] The sending module is configured to send a task execution instruction to the target computing node according to the task parameters, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.

[0025] In a fifth aspect, the present application provides a task processing apparatus, applied to a first computer device, the apparatus comprising:

[0026] a sending module, configured to send a current task processing status of the first computer device to a second computer device;

[0027] a receiving module configured to receive a task execution instruction sent by the second computer device, wherein the task execution instruction is sent by the second computer device when the second computer device determines that the task processing state is an idle state, and the task execution instruction carries task parameters of the current task to be executed;

[0028] The execution module is configured to execute the current task to be executed according to the task parameters.

[0029] In a sixth aspect, the present application provides a task processing apparatus, applied to a second computer device, the apparatus comprising:

[0030] A receiving module is configured to request the first computer device to obtain a current task processing status of the first computer device;

[0031] an acquisition module configured to acquire task parameters of a currently to-be-executed task if the task processing state is an idle state;

[0032] The sending module is configured to send a task execution instruction to the first computer device according to the task parameters, wherein the task execution instruction is used to instruct the first computer device to execute the current task to be executed according to the task parameters.

[0033] In a seventh aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the task processing method described in the first aspect, the second aspect, or the third aspect.

[0034] In an eighth aspect, the present application provides an electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein when the processor executes the computer program, the task processing method described in the first aspect, the second aspect, or the third aspect is implemented.

[0035] By means of the above technical solution, the present application provides a task processing method, device and electronic device. Compared with the current existing technology, the present application can effectively improve the degree of freedom of asynchronous task scheduling, and can mobilize computing nodes deployed on multiple machines to perform distributed processing of asynchronous tasks. Specifically, the task processing status of each computing node in the distributed system is first monitored. The computing nodes in the distributed system are different computer devices, and different computing nodes can read the business data in the preset storage location; then determine the target computing node whose task processing status is idle; then send a task execution instruction to the target computing node based on the task parameters of the current task to be executed, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location. By applying the technical solution of the present application, a distributed system for realizing free scheduling of asynchronous tasks is provided, which can distribute a large number of asynchronous tasks through computing nodes deployed on multiple machines, effectively improving the problem of tight computing resources of a single machine and improving the efficiency of asynchronous task processing.

[0036] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] FIG1 is a schematic diagram showing a flow chart of a task processing method provided in an embodiment of the present application;

[0040] FIG2 is a schematic diagram showing a flow chart of another task processing method provided in an embodiment of the present application;

[0041] FIG3 shows a schematic structural diagram of a distributed system architecture provided in an embodiment of the present application;

[0042] FIG4 is a schematic diagram showing an example of a front-end display effect provided by an embodiment of the present application;

[0043] FIG5 is a schematic diagram showing an exemplary architecture provided in an embodiment of the present application;

[0044] FIG6 shows a flowchart of another task processing method provided in an embodiment of the present application;

[0045] FIG7 shows a flowchart of another task processing method provided in an embodiment of the present application;

[0046] FIG8 shows a schematic structural diagram of a task processing device provided in an embodiment of the present application;

[0047] FIG9 shows a schematic structural diagram of another task processing device provided in an embodiment of the present application;

[0048] FIG10 shows a schematic structural diagram of another task processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to more clearly understand the above-mentioned objectives, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0050] In order to efficiently schedule asynchronous tasks running on different servers to solve the performance bottleneck of a single server and make the asynchronous task process transparent and controllable, this embodiment provides a task processing method, as shown in Figure 1, the method includes:

[0051] Step 101: Monitor the task processing status of each computing node in the distributed system.

[0052] The computing nodes in the distributed system can be different computer devices (such as network computers, servers, etc.), and different computing nodes can read business data in a preset storage location.

[0053] This embodiment adopts the decentralized concept of distributed systems, distributing all computing nodes across various computer devices to facilitate development and maintenance. Furthermore, for the use of this distributed system, a unified scheduling platform (Monitor) acts as a manager to distribute asynchronous tasks. The scheduling platform can be a program with a front-end and back-end, and it is unique within the distributed system. It can be deployed on any computer device on the same network segment. It is responsible for task scheduling. Users submit tasks in the front-end, and the back-end records the submitted task information in a database, monitoring and distributing these tasks.

[0054] The preset storage location can be a database, folder, or data table, and can be used to store business-related data, such as all sequencing data. Sequencing data generated by sequencing instruments will also be transferred to this preset storage location for storage. The path to the preset storage location is mounted on each computer device, allowing different computing nodes to access the business data (data required to perform tasks) in the preset storage location.

[0055] Step 102: Determine a target computing node whose task processing state is idle.

[0056] For example, there are five computing nodes in a distributed system, namely computing node A, computing node B, computing node C, computing node D, and computing node E. Among them, only computing node A is in the idle state, while the other four computing nodes are in the task processing state. In this case, computing node A can be determined as the target computing node.

[0057] Step 103: Obtain the task parameters of the task to be executed currently.

[0058] Task parameters can be parameters referenced by task execution (such as the task execution subject, task execution action, task execution purpose, etc.). When subsequent tasks are executed, the target computing node will execute according to the task parameters.

[0059] Step 104: Send a task execution instruction to the target computing node according to the task parameters of the current task to be executed.

[0060] Furthermore, the target computing node is enabled to execute the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.

[0061] For example, computing node A, whose task processing status is idle, parses the received task execution instruction to obtain the task information (including task parameters) that needs to be executed, and then reads the business data corresponding to the task parameters in the preset storage location according to the path of the preset storage location, and then executes the current task to be executed assigned to computing node A.

[0062] Compared with the current existing technology, this embodiment can effectively improve the degree of freedom of asynchronous task scheduling, and can mobilize computing nodes deployed on multiple machines to perform distributed processing of asynchronous tasks. Specifically, the task processing status of each computing node in the distributed system is first monitored. The computing nodes in the distributed system are different computer devices, and different computing nodes can read the business data in the preset storage location; then determine the target computing node whose task processing status is idle; then send a task execution instruction to the target computing node based on the task parameters of the current task to be executed, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location. By applying the technical solution of this embodiment, a distributed system for realizing free scheduling of asynchronous tasks is provided, which can distribute a large number of asynchronous tasks through computing nodes deployed on multiple machines, effectively improving the problem of tight computing resources of a single machine and improving the efficiency of asynchronous task processing.

[0063] Furthermore, as a refinement and extension of the above embodiment, in order to fully illustrate the specific implementation process of the method of this embodiment, this embodiment provides a specific method as shown in FIG2 , which includes:

[0064] Step 201: Monitor the task processing status of each computing node in the distributed system according to a preset time interval.

[0065] The computing nodes in the distributed system are different computer devices, and different computing nodes can read business data in a preset storage location.

[0066] To improve fluency, asynchronous tasks can be run in the background. The Celery framework is a common and widely used framework in Python, combined with a third-party messaging service (Broker) to implement task scheduling. However, when it comes to in-depth business operations, the Celery framework's limitations are discovered. It does not support Python's native process pool, and even using a compatible billboard process pool can still cause memory leaks. Furthermore, task code maintenance is complex and difficult, requiring flexible addition, deletion, modification, and querying of tasks. Most importantly, it's impossible to schedule task nodes distributed across multiple computers. Therefore, how to use a unified scheduling platform to schedule task nodes distributed across multiple computers and devices and manage tasks with complete freedom is a technical issue worth considering and addressing.

[0067] In order to solve the above technical problems, this embodiment can use the redis database (for illustrative purposes only, other persistent storage databases can also be used) and a timed polling script to implement the role of the message middleware in the asynchronous framework, freeing the scheduling of asynchronous tasks from the celery framework, so that developers can process the redis database and modify the script to adapt to various data analysis scenarios.

[0068] Specifically, this embodiment provides a distributed system that implements free scheduling of asynchronous tasks. This distributed system consists of hardware or software systems distributed across different computer devices, which can communicate and coordinate with each other through message passing. In this distributed system, a group of independent computer devices appears to the user as a unified entity. The system has multiple common physical and logical resources, which can dynamically allocate tasks. These distributed physical and logical resources exchange information via a computer network.

[0069] As shown in FIG3 , the distributed system may include: a data center (preset storage location), a task message middleware, a scheduling platform (Monitor), computing nodes, a scheduling layer front-end page, a scheduling layer backend, and the like.

[0070] Among them, the data center (preset storage location, such as NFS server) will store all business data (such as sequencing data. The sequencing data generated by the sequencing instrument will also be transferred to the data center for storage). At the same time, as shown in Figure 3, the path where the business data is stored will be mounted to each computing node. In this way, even if the service is deployed on other computer devices, the business data stored in the data center can be read.

[0071] Task Message Middleware: The database can be a Redis database. It should be noted that other persistent storage databases can also be used based on actual needs, and this embodiment does not limit this. Each task information created by the user after selecting the corresponding parameters and data name on the front-end page will be stored in the Redis database. The Redis database also stores the parameter configuration of each asynchronous task. Specifically, it may include the following:

[0072] Configuration Information Database (ConfigDB): A Redis database that stores the parameter configurations of each task, including parameter options, parameter default values, the maximum number of workers for each task, and so on.

[0073] Pending Queue Database (first preset database, PendingQueueDB): Redis database. Any created task information is first stored in this pending queue database waiting for processing.

[0074] Running status database (second preset database, RunningBufferDB): Redis database, records the running tasks. If a task enters the running status database, it means that it is already in the running state.

[0075] Scheduling platform (scheduler, Monitor): a role that manages the task queue. It can be implemented as a scheduled script. Its main function is to push the tasks to be executed that are queued in the first preset database to the idle computing nodes for execution.

[0076] The worker corresponding to the compute node is actually the actual executor of the asynchronous task. Specifically, it can be a package deployed on a computer device (such as a server), which can complete the task logic through remote Secure Shell (SSH) command line calls. The worker is equivalent to a software virtual device, that is, a package for a data analysis function deployed on a single computer device (a single compute node). It can be thought of as a background program that responds to these SSH command lines and ultimately executes the asynchronous task.

[0077] The front-end page of the scheduling layer: For example, users can select sequencing data and corresponding parameters on the page and click Submit Task. The front-end sends the parameters to the back-end. The front-end also displays information about all currently queued / running tasks and provides a user interface for adding, deleting, modifying, and querying the task queue.

[0078] Scheduling layer backend: The backend for scheduling tasks can use the Python Django backend framework, which provides many interfaces to the front-end page to complete the modification of task information and the replacement of task queues. Specifically, it can perform various operations such as reading, writing, and modifying the redis database.

[0079] Based on the above-mentioned distributed system, the task processing method provided in this embodiment is specifically executed, wherein the execution subject of the method of this embodiment can be the scheduling platform, which can monitor the task processing status of each computing node in the distributed system at a preset time interval (monitoring is performed every 1 second), and then timely discover those target computing nodes whose task processing status is idle, and assign them the tasks to be executed queued in the first preset database.

[0080] Step 202: Determine a target computing node whose task processing state is idle.

[0081] Step 203: Extract the task parameters of the tasks to be executed that are ranked high from the first preset database according to the order in which the tasks to be executed are arranged.

[0082] The order of arrangement of the to-be-executed tasks in the first preset database may be determined according to the creation time of the to-be-executed tasks and / or the corresponding set priorities. After extracting the task parameters of the to-be-executed tasks from the first preset database, the to-be-executed tasks and the task parameters of the to-be-executed tasks are deleted from the first preset database to conserve storage space in the first preset database.

[0083] For example, the earlier the creation time of the task to be executed, the higher its arrangement order in the first preset database (the higher its ranking), and the later the creation time of the task to be executed, the lower its arrangement order in the first preset database (the lower its ranking). For another example, the user can set the priority of the task to be executed according to actual needs. The higher the priority of the task to be executed, the higher its arrangement order in the first preset database (the higher its ranking), and the lower the priority of the task to be executed, the lower its arrangement order in the first preset database (the lower its ranking).

[0084] Step 204: The extracted task parameters are combined to form command line information to be executed corresponding to the task.

[0085] Step 205: Send command line information to the target computing node through the secure shell protocol.

[0086] A software tool is installed on the target computing node, and the target computing node executes the current task to be executed in response to the command line information through the software tool.

[0087] Specifically, the target computing node responds to the ssh command line through a software tool and executes an asynchronous task in a background program, wherein the software tool may be a software virtual device with a data analysis function, namely, a worker.

[0088] For example, on the premise that computing node A is idle, the task parameters of the top-ranked pending tasks in PendingQueueDB (the first preset database) are extracted, the command line to be executed corresponding to the task is spliced ​​out, and then sent to computing node A via SSH, so that computing node A executes the current pending task based on the business data corresponding to the task parameters in the preset storage location.

[0089] In order to ensure the security of data transmission, step 205 may optionally include: obtaining a key based on the current time information and the identification of the local computer device (such as the MAC address, IP address, motherboard ID number, etc. of the terminal) (the current time information and the identification of the local computer device can be combined to calculate a hash value or MD5 value, etc. to obtain the key); after encrypting the command line information with the key, sending it to the target computing node through the secure shell protocol, so that the target computing node can infer the key based on the time information of the received command line information and the identification of each computer device that has established trust with the target computing node, and use the key to decrypt the command line information.

[0090] For example, the target computing node obtains the time within 3 seconds (such as 9:59:59, 9:59:58, 9:59:57) based on the time information of the received command line information (such as 10:00:00), and then combines the time information within 3 seconds with the identification of each computer device that has established trust with the target computing node to calculate a hash value or MD5 value, and then uses the hash value or MD5 value one by one to try to decrypt the encrypted command line information until the decryption is successful, thereby obtaining the decrypted command line information.

[0091] Further optionally, the method of this embodiment may also include: creating a running record of the current task to be executed in a second preset database (RunningBufferDB), wherein the running records of each task created in the second preset database can be provided to the client for display, so that the user can view the progress of task processing in real time, thereby improving the user experience.

[0092] In order to meet more needs, further optionally, the method of this embodiment may also include: responding to the update setting instruction of the distributed system, obtaining the number of computing nodes to be updated; and then determining the computing nodes to be started or shut down according to the number of computing nodes based on the computing performance information and real-time task processing status of each computing node in the distributed system.

[0093] For example, users can set the number of compute nodes in a distributed system, either before or during the distributed system's operation. Before the distributed system is running, the number of compute nodes set by the user can prioritize the startup of those with better computing performance. During the distributed system's operation, the number of compute nodes set by the user can be used to prioritize the startup or shutdown of appropriate compute nodes based on computing performance information and real-time task processing status, thereby achieving optimal overall computing performance for the distributed system.

[0094] To illustrate the specific implementation process of the above embodiment, the following application scenario is given, but not limited to this:

[0095] Taking the basecall task as an example, this task converts identified read signals into bases. As shown in Figure 4, on the front-end page, select the file to be analyzed and the task parameters, then click the Add button to create the task. The task record will appear in the PendingQueueDB (the first preset database). Due to the limited number of compute nodes, submitted tasks are queued and processed in the order of submission. You can see that the recently submitted analysis task is at position 169 in the queue. Therefore, in the Redis database of the stack structure, it is the first task. If the task has a high priority, the front-end page can call the back-end interface to process the task queue and increase the priority of certain tasks. If a task is moved from the end of the queue to the first, while the task is still queued, the task position and parameters can still be changed by modifying the Redis database. When a compute node becomes idle, the task in the stack structure is extracted, its parameters are concatenated into the corresponding SSH remote command line, and sent to the idle compute node for execution. Simultaneously, a new task record is created in the RunningBuffer (the second preset database) to track progress. The overall example diagram can be shown in Figure 5.

[0096] This embodiment transforms the single-machine task scheduling mode into a distributed system scheduling mode, which can fully utilize the computing nodes of multiple computing devices and effectively solve problems that are difficult to solve in a single-machine performance bottleneck scenario.

[0097] Furthermore, taking the processing of a single computing node as an example, a method as shown in FIG6 is provided, which can be applied to a first computer device (a single computing node). The method includes:

[0098] Step 301: A first computer device sends the current task processing status of the first computer device to a second computer device.

[0099] The first computer device may be any computing node in a distributed system, and the second computer device is a computer device deployed with a scheduling platform.

[0100] Step 302: The first computer device receives a task execution instruction sent by the second computer device.

[0101] The task execution instruction is sent by the second computer device when it is determined that the task processing state is an idle state. The task execution instruction carries the task parameters of the current task to be executed.

[0102] To improve data transmission security, optionally, step 302 may specifically include: the first computer device receiving command line information sent by the second computer device via a secure shell protocol, where the command line information is obtained by splicing based on task parameters;

[0103] To further ensure the security of data transmission, in some examples, a first computer device receives command line information sent by a second computer device via a secure shell protocol, which may specifically include: the first computer device receives encrypted command line information sent by the second computer device via the secure shell protocol; then, based on the time information of receiving the command line information and the identification of each computer device that has established trust with the first computer device, determines the key required to decrypt the command line information; and then uses the key to decrypt the command line information.

[0104] The specific decryption process can be found in the corresponding description in step 205 and will not be repeated here.

[0105] Step 303: The first computer device executes the current task to be executed according to the task parameters.

[0106] Based on the optional manner of step 302, correspondingly, step 303 may specifically include: executing the current task to be executed by executing command line information.

[0107] Exemplarily, step 303 may specifically include: executing the current task to be executed based on the business data corresponding to the task parameters in a preset storage location, wherein the preset storage location stores business data corresponding to different task parameters.

[0108] This embodiment transforms the single-machine task scheduling mode into a distributed system scheduling mode, which can fully utilize the computing nodes of multiple computing devices and effectively solve problems that are difficult to solve in a single-machine performance bottleneck scenario.

[0109] Furthermore, taking the processing of the scheduling platform as an example, a method as shown in FIG7 is provided, which can be applied to a second computer device (which can be a computer device deployed with the scheduling platform). The method includes:

[0110] Step 401: The second computer device requests the first computer device to obtain the current task processing status of the first computer device.

[0111] The first computer device may be any computing node in a distributed system, and the second computer device is a computer device deployed with a scheduling platform.

[0112] Step 402: If the current task processing state of the first computer device is an idle state, the second computer device obtains the task parameters of the current task to be executed.

[0113] Optionally, obtaining the task parameters of the current task to be executed in step 402 may specifically include: extracting the task parameters of the top-ranked tasks to be executed from the first preset database according to the arrangement order of the tasks to be executed, wherein the arrangement order between the tasks to be executed in the first preset database is determined according to the creation time of the tasks to be executed and / or the corresponding set priority.

[0114] For example, the earlier the creation time of the task to be executed, the higher its arrangement order in the first preset database (the higher its ranking), and the later the creation time of the task to be executed, the lower its arrangement order in the first preset database (the lower its ranking). For another example, the user can set the priority of the task to be executed according to actual needs. The higher the priority of the task to be executed, the higher its arrangement order in the first preset database (the higher its ranking), and the lower the priority of the task to be executed, the lower its arrangement order in the first preset database (the lower its ranking).

[0115] Step 403: The second computer device sends a task execution instruction to the first computer device according to the task parameters.

[0116] The task execution instruction is used to instruct the first computer device to execute the current task to be executed according to the task parameters.

[0117] To improve data transmission security, step 403 may optionally include: combining task parameters to generate command line information corresponding to the task to be executed; and sending the command line information to the first computer device via a secure shell protocol.

[0118] To further ensure data transmission security, in some examples, sending the command line information to the first computer device via the secure shell protocol may specifically include: obtaining a key based on the current time information and the local computer device's identifier; encrypting the command line information using the key; and sending the command line information to the first computer device via the secure shell protocol. The specific decryption process can be found in the corresponding descriptions of steps 205 and 302 and will not be repeated here.

[0119] Further, optionally, the method of this embodiment may further include: creating a running record of the currently pending task in a second preset database; receiving an instruction to view the task running record; and determining a display result corresponding to the viewing instruction based on the running records of each task created in the second preset database. This allows the user to view the progress of task processing in real time, thereby improving the user experience.

[0120] This embodiment transforms the single-machine task scheduling mode into a distributed system scheduling mode, which can fully utilize the computing nodes of multiple computing devices and effectively solve problems that are difficult to solve in a single-machine performance bottleneck scenario.

[0121] Furthermore, as a specific implementation of the method shown in FIG. 1 and FIG. 2 , this embodiment provides a task processing device, as shown in FIG. 8 , which includes: a monitoring module 51 , a determination module 52 , an acquisition module 53 , and a sending module 54 .

[0122] A monitoring module 51 is configured to monitor the task processing status of each computing node in the distributed system, wherein the computing nodes in the distributed system are different computer devices, and different computing nodes can read business data in a preset storage location;

[0123] A determination module 52 is configured to determine a target computing node whose task processing state is an idle state;

[0124] The acquisition module 53 is configured to obtain the task parameters of the task to be executed;

[0125] The sending module 54 is configured to send a task execution instruction to the target computing node according to the task parameters, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.

[0126] In a specific application scenario, the acquisition module 53 is specifically configured to extract the task parameters of the top-ranked tasks to be executed from the first preset database according to the arrangement order of the tasks to be executed, wherein the arrangement order between the tasks to be executed in the first preset database is determined according to the creation time of the tasks to be executed and / or the corresponding set priority.

[0127] In a specific application scenario, the determination module 52 is further configured to create a running record of the currently executed task in a second preset database, wherein the running record of each task created in the second preset database can be provided to the client for display.

[0128] In a specific application scenario, the sending module 54 is specifically configured to combine the task parameters to generate command line information corresponding to the task to be executed; and send the command line information to the target computing node via a secure shell protocol.

[0129] In a specific application scenario, the sending module 54 is further configured to obtain a key based on the current time information and the identification of the local computer device; after encrypting the command line information using the key, it is sent to the target computing node through the secure shell protocol, so that the target computing node can infer the key based on the time information of receiving the command line information and the identification of each computer device that has established trust with the target computing node, and use the key to decrypt the command line information.

[0130] In a specific application scenario, the acquisition module 53 is further configured to obtain the number of computing nodes to be updated in response to an update setting instruction of the distributed system;

[0131] The determination module 52 is further configured to determine the computing nodes to be started or shut down according to the number of computing nodes based on the computing performance information and real-time task processing status of each computing node in the distributed system.

[0132] In a specific application scenario, the monitoring module 51 is specifically configured to monitor the task processing status of each computing node in the distributed system according to a preset time interval.

[0133] It should be noted that for other corresponding descriptions of the functional units involved in the task processing device provided in this embodiment, reference can be made to the corresponding descriptions in FIG. 1 and FIG. 2 , which will not be repeated here.

[0134] Furthermore, as a specific implementation of the method shown in FIG6 , this embodiment provides a task processing apparatus, which is applied to a first computer device. As shown in FIG9 , the apparatus includes: a sending module 61 , a receiving module 62 , and an executing module 63 .

[0135] A sending module 61 is configured to send the current task processing status of the first computer device to the second computer device;

[0136] a receiving module 62 configured to receive a task execution instruction sent by the second computer device, wherein the task execution instruction is sent by the second computer device when the second computer device determines that the task processing state is an idle state, and the task execution instruction carries task parameters of the current task to be executed;

[0137] The execution module 63 is configured to execute the current task to be executed according to the task parameters.

[0138] In a specific application scenario, the execution module 63 is specifically configured to execute the current task to be executed based on the business data corresponding to the task parameters in a preset storage location, where the preset storage location stores business data corresponding to different task parameters.

[0139] In a specific application scenario, the receiving module 62 is specifically configured to receive command line information sent by the second computer device through the secure shell protocol, where the command line information is obtained by splicing according to the task parameters;

[0140] The execution module 63 is specifically configured to execute the current task to be executed by executing the command line information.

[0141] In a specific application scenario, the receiving module 62 is further configured to receive the encrypted command line information sent by the second computer device through the secure shell protocol; determine the key required to decrypt the command line information based on the time information of receiving the command line information and the identification of each computer device that has established trust with the first computer device; and use the key to decrypt and obtain the command line information.

[0142] It should be noted that for other corresponding descriptions of the functional units involved in the task processing device provided in this embodiment, reference can be made to the corresponding description in FIG6 , which will not be repeated here.

[0143] Furthermore, as a specific implementation of the method shown in FIG7 , this embodiment provides a task processing apparatus, which is applied to a second computer device. As shown in FIG10 , the apparatus includes: a receiving module 71 , an acquiring module 72 , and a sending module 73 .

[0144] The receiving module 71 is configured to request the first computer device to obtain the current task processing status of the first computer device;

[0145] The acquisition module 72 is configured to acquire the task parameters of the currently executed task if the task processing state is an idle state;

[0146] The sending module 73 is configured to send a task execution instruction to the first computer device according to the task parameters, wherein the task execution instruction is used to instruct the first computer device to execute the current task to be executed according to the task parameters.

[0147] In a specific application scenario, the acquisition module 72 is specifically configured to extract the task parameters of the top-ranked tasks to be executed from the first preset database according to the arrangement order of the tasks to be executed, wherein the arrangement order between the tasks to be executed in the first preset database is determined according to the creation time of the tasks to be executed and / or the corresponding set priority.

[0148] In a specific application scenario, the acquisition module 72 is also configured to create an operation record of the current task to be executed in a second preset database; receive a viewing instruction for the task operation record; and determine the display result corresponding to the viewing instruction based on the operation records of each task created in the second preset database.

[0149] In a specific application scenario, the sending module 73 is specifically configured to combine the task parameters to generate command line information corresponding to the task to be executed; and send the command line information to the first computer device via the secure shell protocol.

[0150] In a specific application scenario, the sending module 73 is further configured to obtain a key based on the current time information and the identification of the local computer device; after encrypting the command line information using the key, it is sent to the first computer device through the secure shell protocol.

[0151] It should be noted that for other corresponding descriptions of the functional units involved in the task processing device provided in this embodiment, reference can be made to the corresponding description in FIG7 , which will not be repeated here.

[0152] Based on the above-mentioned methods shown in Figures 1 and 2, this embodiment also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method shown in Figures 1 and 2 is implemented. Based on the above-mentioned method shown in Figure 6, this embodiment also provides another computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method shown in Figure 6 is implemented. Based on the above-mentioned method shown in Figure 7, this embodiment also provides another computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method shown in Figure 7 is implemented.

[0153] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0154] Based on the above-mentioned method shown in Figures 1 and 2, and the virtual device embodiment shown in Figure 8, in order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, which may specifically include a personal computer, a server, a network computer and other devices, and the device includes a storage medium and a processor; the storage medium is used to store computer programs; the processor is used to execute the computer program to implement the above-mentioned method shown in Figures 1 and 2.

[0155] Based on the above-mentioned method shown in Figure 6 and the virtual device embodiment shown in Figure 9, in order to achieve the above-mentioned purpose, the embodiment of the present application also provides another electronic device, which may specifically include a personal computer, a server, a network computer and other devices, and the device includes a storage medium and a processor; the storage medium is used to store computer programs; the processor is used to execute the computer program to implement the above-mentioned method shown in Figure 6.

[0156] Based on the above-mentioned method shown in FIG7 and the virtual device embodiment shown in FIG10 , in order to achieve the above-mentioned purpose, the embodiment of the present application also provides another electronic device, which may specifically include a personal computer, a server, a network computer and other devices, and the device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the above-mentioned method shown in FIG7 .

[0157] Optionally, the physical device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a Wi-Fi module, and the like. The user interface may include a display, an input unit such as a keyboard, and the like. Optional user interfaces may also include a USB interface, a card reader interface, and the like. The network interface may optionally include a standard wired interface, a wireless interface (such as a Wi-Fi interface), and the like.

[0158] Those skilled in the art will understand that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0159] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device, supporting the execution of information processing programs and other software and / or programs. The network communication module is used to enable communication between components within the storage medium, as well as with other hardware and software within the physical information processing device.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware. By applying the solution of this embodiment, the degree of freedom of asynchronous task scheduling can be effectively improved, and computing nodes deployed on multiple machines can be mobilized to perform distributed processing of asynchronous tasks. A distributed system for realizing free scheduling of asynchronous tasks is provided, which can distribute a large number of asynchronous tasks by deploying computing nodes on multiple machines, effectively improving the problem of tight computing resources on a single machine and improving the efficiency of asynchronous task processing.

[0161] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0162] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand and implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments described herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A task processing method, characterized in that: include: Monitoring the task processing status of each computing node in the distributed system, wherein the computing nodes in the distributed system are different computer devices, and different computing nodes can read business data in a preset storage location; Determine a target computing node whose task processing state is an idle state; Get the task parameters of the current task to be executed; According to the task parameters, a task execution instruction is sent to the target computing node, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.

2. The method according to claim 1, characterized in that The step of obtaining the task parameters of the currently executed task includes: From the first preset database, task parameters of the top-ranked tasks to be executed are extracted according to the arrangement order among the tasks to be executed, wherein the arrangement order among the tasks to be executed in the first preset database is determined according to the creation time of the tasks to be executed and / or the corresponding set priority.

3. The method according to claim 1, characterized in that The method further comprises: A running record of the currently to-be-executed task is created in a second preset database, wherein the running record of each task created in the second preset database can be provided to a client for display.

4. The method according to claim 1, wherein The sending of the task execution instruction to the target computing node according to the task parameters includes: The task parameters are combined to form the command line information to be executed corresponding to the task; The command line information is sent to the target computing node through a secure shell protocol.

5. The method according to claim 4, characterized in that The sending the command line information to the target computing node through the secure shell protocol includes: Obtaining a key based on current time information and an identifier of a local computer device; After encrypting the command line information using the key, the command line information is sent to the target computing node via a secure shell protocol, so that the target computing node can calculate the key based on the time information of receiving the command line information and the identification of each computer device that has established trust with the target computing node, and use the key to decrypt the command line information.

6. The method according to claim 4, characterized in that A software tool is installed on the target computing node, and the target computing node executes the current task to be executed in response to the command line information through the software tool.

7. The method according to claim 1, characterized in that The method further comprises: In response to an update setting instruction of the distributed system, obtaining the number of computing nodes to be updated; According to the computing performance information and real-time task processing status of each computing node in the distributed system, the computing nodes to be started or shut down are determined according to the number of computing nodes.

8. The method according to claim 1, characterized in that Monitoring the task processing status of each computing node in the distributed system includes: The task processing status of each computing node in the distributed system is monitored at preset time intervals.

9. A task processing method, characterized in that: Applied to a first computer device, the method includes: Sending the current task processing status of the first computer device to the second computer device; receiving a task execution instruction sent by the second computer device, wherein the task execution instruction carries task parameters of a current task to be executed; The current task to be executed is executed based on the business data corresponding to the task parameters in a preset storage location, where the business data corresponding to different task parameters are stored.

10. The method according to claim 9, characterized in that The receiving the task execution instruction sent by the second computer device includes: receiving command line information sent by the second computer device through a secure shell protocol, wherein the command line information is obtained by splicing according to the task parameters; The executing the current task to be executed according to the task parameters includes: The current task to be executed is executed by executing the command line information.

11. A task processing method, characterized in that: Applied to a second computer device, the method includes: Requesting the first computer device to obtain the current task processing status of the first computer device; If the task processing state is an idle state, then obtaining the task parameters of the task to be executed currently; A task execution instruction is sent to the first computer device according to the task parameters, where the task execution instruction is used to instruct the first computer device to execute the current task to be executed according to the task parameters.

12. The method according to claim 11, characterized in that The step of obtaining the task parameters of the currently executed task includes: From the first preset database, task parameters of the top-ranked tasks to be executed are extracted according to the arrangement order among the tasks to be executed, wherein the arrangement order among the tasks to be executed in the first preset database is determined according to the creation time of the tasks to be executed and / or the corresponding set priority.

13. The method according to claim 11, characterized in that The method further comprises: Creating a running record of the currently to-be-executed task in a second preset database; Receive instructions for viewing task running records; A display result corresponding to the viewing instruction is determined according to the running records of each task created in the second preset database.

14. The method according to claim 11, characterized in that The sending a task execution instruction to the first computer device according to the task parameters includes: The task parameters are combined to form the command line information to be executed corresponding to the task; The command line information is sent to the first computer device through a secure shell protocol.

15. A task processing device, characterized in that: include: a monitoring module configured to monitor a task processing status of each computing node in a distributed system, wherein the computing nodes in the distributed system are different computer devices, and each computing node can read business data in a preset storage location; A determination module is configured to determine a target computing node whose task processing state is an idle state; An acquisition module is configured to obtain task parameters of a task to be executed; The sending module is configured to send a task execution instruction to the target computing node according to the task parameters, so that the target computing node executes the current task to be executed based on the business data corresponding to the task parameters in the preset storage location.

16. A task processing device, characterized in that: Applied to a first computer device, the apparatus includes: a sending module, configured to send a current task processing status of the first computer device to a second computer device; a receiving module configured to receive a task execution instruction sent by the second computer device, wherein the task execution instruction is sent by the second computer device when the second computer device determines that the task processing state is an idle state, and the task execution instruction carries task parameters of the current task to be executed; The execution module is configured to execute the current task to be executed according to the task parameters.

17. A task processing device, characterized in that: Applied to a second computer device, the apparatus includes: A receiving module is configured to request the first computer device to obtain a current task processing status of the first computer device; an acquisition module configured to acquire task parameters of a currently to-be-executed task if the task processing state is an idle state; The sending module is configured to send a task execution instruction to the first computer device according to the task parameters, wherein the task execution instruction is used to instruct the first computer device to execute the current task to be executed according to the task parameters.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

19. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 14 is implemented.

Images