Long task asynchronous scheduling system and method

Abstract

The long task scheduling system comprises a long task center and at least two different types of service servers, the long task center comprises an interface for communicating with a terminal, used for receiving different types of long task requests submitted by the terminal; a scheduling module, used for sending a calling notification to the service server; and a monitoring module, used for listening to a processing result of the long task by the service server; an interface for communicating with the service server, used for receiving the processing result of the different types of service servers. The application also provides a long task asynchronous processing method, comprising the following steps: receiving a long task processing request; adding the long task processing request to a priority queue; sending a calling notification to a service server; receiving a query request of the service server and providing an input parameter to the service server according to the query request; and receiving a long task processing result and feeding back to a terminal.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to an asynchronous scheduling system for long tasks in a data center. Background Technology

[0002] A common scenario is that users submit tasks, the business system processes them in the background, and then informs the user of the result. In the context of multimedia and big data, the background processing of user-submitted tasks often takes a long time (referred to as long tasks). To improve user experience, progress bars or percentages are typically used to display the processing progress to the user on the client side when processing long tasks in the background. However, in existing methods, the client is usually suspended during the waiting period, preventing the user from performing other operations and negatively impacting the user experience.

[0003] One solution is to use one process to respond to user input. When a user submits a long task request, a new process is started to handle the long task asynchronously, avoiding client suspension. However, this approach has a drawback: asynchronous processing of long tasks involves a series of common functions such as task submission, scheduling, execution, and monitoring. Many types of long tasks require asynchronous processing. For example, pasting large attachments in emails, uploading extremely large attachments, encrypting them after upload, and performing security scans are all long tasks. In video editing, background rendering and enhancement of the video are also long tasks. If all these different long tasks require asynchronous processing, each asynchronous system needs to implement a series of common functions such as task submission, scheduling, execution, and monitoring, resulting in wasted development costs. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a unified long task scheduling system that can provide common functions such as task submission, scheduling, termination, and monitoring for different types of long tasks, so that various long task processing systems only need to focus on the long task processing itself, thereby simplifying the development of various long task processing systems and saving costs.

[0005] The long task scheduling system of the present invention includes: a long task center and at least two different types of business servers. The long task center includes: an interface for communicating with a terminal, used to receive different types of long task requests submitted by the terminal; a scheduling module for sending call notifications to the business servers; a monitoring module for listening to the processing results of the long tasks by the business servers; and an interface for communicating with the business servers for receiving the processing results of the different types of business servers.

[0006] Preferably, in the above-mentioned long task scheduling system, the long task request and the processing result of the long task are JSON strings as parameters.

[0007] Preferably, in the above-mentioned long task scheduling system, the JSON string includes key-value pairs, which are used to represent the name and value of the parameter, respectively.

[0008] Preferably, in the above-mentioned long task scheduling system, the scheduling module maintains a priority queue for the long task based on the identity of the requester and the waiting time of the long task.

[0009] Preferably, in the above-mentioned long task scheduling system, the long task center assigns a number to the long task; the long task number is included in the call notification and the processing result.

[0010] Preferably, in the above-mentioned long task scheduling system, the monitoring module is also responsible for actively initiating query requests to the business server according to a preset polling cycle.

[0011] The present invention also provides a method for asynchronous processing of long tasks, comprising: receiving a long task processing request; adding the long task processing request to a priority queue; sending a call notification to a business server; receiving a query request from the business server and providing input parameters to the business server according to the query request; and receiving the long task processing result and feeding it back to the terminal.

[0012] Preferably, in the above-described asynchronous processing method for long tasks, the long task request and the processing result of the long task are presented as JSON strings as parameters.

[0013] Preferably, in the above-described asynchronous processing method for long tasks, the long task processing request is added to a priority queue based on the requester of the long task and / or the waiting time of the long task.

[0014] Prior to the above-mentioned long-task asynchronous processing method, before the step of receiving the query request from the business server and providing input parameters to the business server according to the request, the method further includes the step of parsing the JSON string into "key-value" pairs.

[0015] Preferably, in the above-mentioned long-task asynchronous processing method, after the step of providing input parameters to the business server according to the query request, it further includes the step of actively initiating a query request to the business server according to a preset polling period. Attached Figure Description

[0016] Figure 1 The diagram shown is a logical structure diagram of an application scenario for the long-task asynchronous processing system of the present invention.

[0017] Figure 2 The diagram shows a flowchart of the asynchronous processing method for long tasks according to the present invention. Detailed Implementation

[0018] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0019] Figure 1 The diagram shows the logical structure of the long-task asynchronous processing system of this invention in an application scenario. Clients 1, 2...n represent logical terminals that may request long tasks; these could be an app or a computer application. The long-task center is the core of this invention, acting as middleware between the terminals and the business servers, responsible for the unified reception, scheduling, and monitoring of long tasks. Business servers 1, 2...n represent various servers responsible for processing long tasks, such as email servers handling long tasks like uploading large email attachments, encrypting them after uploading, and performing security scans, or video editing servers handling long tasks like rendering and enhancing videos.

[0020] Different long-task requests from different clients are first submitted to the long-task center, which uniformly receives, schedules (including process scheduling and task priority scheduling), and monitors them. One challenge lies in the interface implementation between the client and the long-task center. Since the parameter types and contents of different types of long tasks are completely different, traditional interface design methods are unlikely to provide the flexibility or scalability to adapt to different long-task types. Therefore, this invention utilizes the flexibility of JSON strings to shield the personalized differences in access between various long-task systems. Specifically, it uses key-value pairs to define the parameter type and content, and then transmits the key-value pairs as JSON strings. For example, for a long video editing task, it can include key-value pairs such as: "url:. / temp / test.avi", "submitter:benjamin", "src:xxx" to represent the video URL, submitter, required resources, etc., respectively. These key-value pairs are then transmitted as a JSON string (hereinafter referred to as "input parameters"). Because JSON has a standard syntax structure, parsing the JSON string according to JSON syntax rules yields the key-value pair information, thus obtaining the parameter name and value.

[0021] In addition to basic services such as task number assignment and process management, the core functions of priority management and task scheduling in the long task center are handled by its scheduling module. The scheduling module establishes priority queues for long tasks based on factors such as the requester's identity and the task's waiting time, and notifies the corresponding business servers according to the task type. Taking email and video editing long tasks as examples, after receiving email and video editing task requests, the scheduling center adds the tasks to priority queues according to a priority scheduling algorithm and notifies the email and video editing servers, including the task number. The priority scheduling algorithm can consider factors such as the submitter and waiting time of the long task; for example, tasks submitted by VIP users are given higher priority, as are long tasks with longer waiting times. Multiple priority queues can also be used, such as creating separate priority queues for each different type of long task.

[0022] Upon receiving the notification, if the mail server still has resources to process the new task, it sends a query request to the long task center to retrieve the input parameters of the long task. The request includes the task number. Upon receiving this query request, the long task center locates the terminal's long task processing request based on the task number and sends the input parameters of that long task request to the mail server. Preferably, the long task center may also include a parsing module to parse the input parameters into key-value pairs before sending them to the mail server, thereby further reducing the difficulty of mail server access. The processing procedure for the video editing server is similar and will not be described further.

[0023] The long-task center also includes a monitoring module responsible for listening to the processing results of long-tasks. Continuing the example: after the mail server processes a long-task, it sends the processing result and the corresponding task number back to the long-task center. The monitoring module listens for the feedback result, and the long-task center then sends the result back to the terminal that initiated the long-task request, completing the service. The feedback results for different types of long-tasks can vary significantly. Therefore, the interface between the long-task center and the business server can also use JSON strings (hereinafter referred to as "output parameters") for communication, thereby masking the individual differences between business servers. The principle is the same. Add ginseng This will not be elaborated further. Preferably, the long task center sets different polling cycles for different types of long tasks, so that when no feedback is received from the business server for a long time, the monitoring module actively sends a task processing query request to the business server, and promptly detects and reports problems such as business server downtime and blockage to the requester.

[0024] Considering that between the time the business server receives the notification (T1) and the time T2, after having resources to process the task, initiates a query request to the long task center, the long task center may receive new tasks or adjust the task priority queue. Therefore, in the second embodiment of this invention, when the long task center sends a notification to the business server, it does not need to include the task number; it only needs to inform the business server that there is a new task request. Thus, this notification can be of any predefined form, or even just a flag. Upon receiving the query request from the business server, the long task center then sends the highest priority task number in the priority queue along with the task's input parameters to the business server, thereby ensuring that the highest priority task is processed first.

[0025] This invention also provides a method for asynchronous processing of long tasks, the process of which is as follows: Figure 2 As shown.

[0026] Step S1 indicates receiving a long task processing request. The entity responsible for receiving long task processing requests could be a long task center, which acts as middleware between the terminal and the business server. In practice, this can include unified terminal and business server. The SDK is designed to facilitate access from different business servers and terminals. The parameter types and values ​​of long task processing requests are defined as key-value pairs and encapsulated into JSON strings (hereinafter referred to as "input parameters").

[0027] Step S2 indicates that the received long task processing request is added to a priority queue. A scheduling module is set up in the long task center to create priority queues for long tasks based on factors such as the requester's identity and the task's waiting time. For example, tasks submitted by VIP users have higher priority, as do long tasks with longer waiting times. Multiple priority queues can be used, such as separate priority queues for each different type of long task.

[0028] Step S3 involves sending a call notification to the business server. The notification includes a task number, which is distributed by the long task center and serves as a unique identifier for the task. This step S2 can be performed in parallel.

[0029] Step S4 involves receiving a query request from the business server and providing input parameters to the business server based on the request. Upon receiving the notification, if the business server still has resources to process the new task, it sends a query request to the long task center to retrieve the input parameters of the long task. The request includes the task number. Upon receiving the query request, the long task center retrieves the terminal's long task processing request based on the task number and sends the input parameters of the long task request to the business server. Preferably, the long task center may also include a parsing module to parse the input parameters into key-value pairs before sending them to the mail server, thereby further reducing the difficulty of mail server access.

[0030] Step S5 indicates receiving the long task processing result and feeding it back to the terminal. After processing the long task, the business server sends the processing result and the corresponding task number back to the long task center. The long task center listens for the feedback result and then sends it back to the terminal that initiated the long task request, completing the service. The feedback results for different types of long tasks vary significantly. Therefore, the interface between the long task center and the business server can also use JSON strings (hereinafter referred to as "output parameters") for communication, thereby masking the individual differences between business servers. The principle is the same as for business trips and will not be elaborated further. Preferably, different polling cycles can be set for different types of long tasks so that when no feedback is received from the business server for an extended period, a task processing query request is proactively sent to the business server to promptly detect and report issues such as business server downtime or congestion to the requester.

[0031] Considering that between the time the business server receives the notification (T1) and the time T2, after having resources to process the task, initiates a query request to the long task center, the long task center may receive new tasks or adjust the task priority queue. Therefore, in the second embodiment of this invention, when the long task center sends a notification to the business server, it does not need to include the task number; it only needs to inform the business server that there is a new task request. Thus, this notification can be of any predefined form, or even just a flag. Upon receiving the query request from the business server, the long task center then sends the highest priority task number in the priority queue along with the task's input parameters to the business server, thereby ensuring that the highest priority task is processed first.

[0032] The long-task asynchronous scheduling system and method of the present invention extracts a series of common functions such as process scheduling, task submission, scheduling, execution, and monitoring involved in the asynchronous processing of different types of long tasks. By utilizing the flexibility of JSON strings, it shields the individuality of different long tasks and business servers, avoids the duplication of investment in the development of different business servers, and thus has high industrial value.

[0033] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Those skilled in the art will understand that the modules, devices, and interfaces in the above embodiments are logical divisions for illustrative purposes, and the specific implementation can be software, hardware, or a combination of both. Anyone skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A long-task asynchronous scheduling system, characterized in that, include: A long-duration task center and at least two different types of business servers, wherein the long-duration task center includes: An interface for communicating with a terminal is used to receive different types of long task requests submitted by the terminal; any type of long task processing request takes a JSON string as a parameter, the JSON string includes key-value pairs, which are used to represent the name and value of the parameter, respectively; The scheduling module is used to send call notifications to the business server; And a monitoring module, used to monitor the processing results of the long task by the business server; An interface for communicating with the business server is used to receive processing results from the different types of business servers; the processing results of any type of long task are communicated using a JSON string as a parameter, and the JSON string corresponding to the processing result includes key-value pairs, which are used to represent the name and value of the parameter corresponding to the processing result, respectively.

2. The long-task asynchronous scheduling system according to claim 1, characterized in that, The scheduling module maintains a priority queue for the long task based on the identity of the requester and the waiting time of the long task.

3. The long-task asynchronous scheduling system according to claim 1, characterized in that, The long task center assigns a number to the long task; the long task number is included in the call notification and the processing result.

4. The long-task asynchronous scheduling system according to claim 1, characterized in that, The monitoring module is also responsible for proactively initiating query requests to the business server according to a preset polling cycle.

5. A method for asynchronous processing of long tasks, characterized in that, include: Steps for receiving a long task processing request; The long task processing request uses a JSON string as a parameter, and the JSON string includes key-value pairs, which are used to represent the name and value of the parameter, respectively. The step of adding the long task processing request to the priority queue; The steps to send a call notification to the business server; The steps of receiving a query request from the business server and providing input parameters to the business server according to the query request; The steps for receiving the processing results of long tasks and feeding them back to the terminal; The processing results of any type of long task are communicated using a JSON string as a parameter. The JSON string corresponding to the processing result includes key-value pairs, which are used to represent the name and value of the parameter corresponding to the processing result, respectively.

6. The long-task asynchronous processing method according to claim 5, characterized in that, The long task processing request is added to a priority queue based on the requester of the long task and / or the waiting time of the long task.

7. The long-task asynchronous processing method according to claim 5, characterized in that, Before the steps of receiving a query request from the business server and providing input parameters to the business server according to the request, the method further includes the step of parsing the JSON string into "key-value" pairs.

8. The long-task asynchronous processing method according to claim 5, characterized in that, After the step of providing input parameters to the business server according to the query request, the method further includes the step of actively initiating a query request to the business server according to a preset polling cycle.

Images