Message processing task scheduling method and device

Abstract

The invention discloses a message processing task scheduling method and device which are applied to a distributed system, and the method comprises the steps: extracting the features of a to-be-processed message; analyzing the features of the message by using a convolutional neural network to obtain the complexity of a processing task of the message, the processing task being that a computing node in a distributed system asynchronously processes the message through a processing resource; based on the complexity of the processing tasks, priorities are allocated to the processing tasks, and the priorities of the processing tasks are in negative correlation with the complexity of the processing tasks; determining a scheduling strategy corresponding to the processing task based on the priority of the processing task; and executing the processing task according to the scheduling strategy.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for scheduling message processing tasks. Background Art

[0002] In distributed systems, the scheduling of message processing tasks is crucial to system performance. In particular, when processing large messages, traditional synchronous processing methods may lead to reduced system throughput and responsiveness. Therefore, a method that can effectively schedule large message asynchronous processing tasks is needed. Summary of the invention

[0003] The embodiments of the present application provide a method and device for scheduling message processing tasks, which are helpful to improve the scheduling efficiency of large message asynchronous processing tasks in a distributed system.

[0004] In a first aspect, a method for scheduling a message processing task is provided, which is applied to a distributed system, and the method includes:

[0005] Extracting features of the message to be processed;

[0006] Using a convolutional neural network to analyze the features of the message to obtain the complexity of a processing task of the message, wherein the processing task is that a computing node in the distributed system asynchronously processes the message through processing resources;

[0007] Based on the complexity of the processing task, assigning a priority to the processing task, wherein the priority of the processing task is negatively correlated with the complexity of the processing task;

[0008] Determining a scheduling strategy corresponding to the processing task based on the priority of the processing task;

[0009] The processing task is executed according to the scheduling strategy.

[0010] In some implementations, the convolutional neural network is trained based on features of sample messages and the complexity of the sample messages, and the complexity of the sample messages is obtained based on historical processing time or resource consumption of the sample messages.

[0011] In some implementations, assigning a priority to the processing task based on the complexity of the processing task includes:

[0012] If the level of complexity of the processing task belongs to the lowest complexity, determining the level of priority of the processing task to be the highest priority;

[0013] If the complexity level of the processing task belongs to medium complexity, determining the priority level of the processing task to be medium priority;

[0014] If the level of complexity of the processing task belongs to the highest complexity, the level of priority of the processing task is determined to be the lowest priority.

[0015] In some implementations, extracting features of the message to be processed includes:

[0016] The size of the message, the transmission time of the message, the type of the message, the content complexity of the message, and the timeliness requirement of the message are extracted from the message as features of the message.

[0017] In some implementations, assigning a priority to the processing task based on the complexity of the processing task includes:

[0018] Priorities are assigned to the processing tasks based on the complexity, urgency, and processing time requirements of the processing tasks.

[0019] In some implementations, determining the scheduling strategy corresponding to the processing task based on the priority of the processing task includes:

[0020] If the priority level of the processing task is the highest priority, the processing task is preferentially allocated to available processing resources.

[0021] In some embodiments, the method further comprises:

[0022] If the priority level of the processing task is the lowest priority and the processing resources of the processing task are insufficient, the scheduling frequency of the processing task is reduced.

[0023] In a second aspect, a scheduling device for message processing tasks is provided, which is applied to a distributed system, and the device includes:

[0024] An extraction unit, used for extracting features of a message to be processed;

[0025] An analysis unit, configured to analyze the features of the message using a convolutional neural network to obtain the complexity of a processing task of the message, wherein the processing task is that a computing node in the distributed system asynchronously processes the message through processing resources;

[0026] an allocating unit, configured to allocate a priority to the processing task based on the complexity of the processing task, wherein the priority of the processing task is negatively correlated with the complexity of the processing task;

[0027] A determination unit, configured to determine a scheduling strategy corresponding to the processing task based on the priority of the processing task;

[0028] An execution unit is used to execute the processing task according to the scheduling strategy.

[0029] In some implementations, the convolutional neural network is trained based on features of sample messages and the complexity of the sample messages, and the complexity of the sample messages is obtained based on historical processing time or resource consumption of the sample messages.

[0030] In some embodiments, the allocation unit is used to determine the priority level of the processing task as the highest priority if the complexity level of the processing task belongs to the lowest complexity; determine the priority level of the processing task as the medium priority if the complexity level of the processing task belongs to the medium complexity; and determine the priority level of the processing task as the lowest priority if the complexity level of the processing task belongs to the highest complexity.

[0031] In some embodiments, the extraction unit is used to extract the size of the message, the transmission time of the message, the type of the message, the content complexity of the message, and the timeliness requirement of the message from the message as characteristics of the message.

[0032] In some implementations, the allocation unit is configured to allocate priorities to the processing tasks based on the complexity, urgency, and processing time requirements of the processing tasks.

[0033] In some implementations, the determining unit is configured to preferentially allocate the processing task to available processing resources if the priority level of the processing task is the highest priority.

[0034] In some implementations, the determining unit is further configured to reduce the scheduling frequency of the processing task if the priority level of the processing task is the lowest priority and the processing resources of the processing task are insufficient.

[0035] In a third aspect, a computer device is provided, the computer device comprising a processor, the processor being coupled to a memory, the memory storing at least one computer program instruction, the at least one computer program instruction being loaded and executed by the processor, so that the computer device implements the method provided in the first aspect or any optional manner of the first aspect. The specific details of the computer device provided in the third aspect can be found in the first aspect or any optional manner of the first aspect, and will not be repeated here.

[0036] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium. When the instruction is executed on a computer, the computer executes the method provided by the first aspect or any optional manner of the first aspect.

[0037] In a fifth aspect, a computer program product is provided, which includes one or more computer program instructions. When the computer program instructions are loaded and executed by a computer, the computer executes the method provided by the first aspect or any optional manner of the first aspect.

[0038] It can be seen that the embodiments of the present application have the following beneficial effects:

[0039] The embodiment of the present application combines a convolutional neural network with a priority-based scheduling strategy. The complexity of the message processing task is predicted by the convolutional neural network, and the priority is assigned to the processing task according to the predicted complexity. The higher the complexity of the processing task, the lower the priority of the processing task, and the lower the complexity of the processing task, the higher the priority of the processing task. Therefore, the simpler the message, the more priority it can be processed, thereby reducing the probability of system congestion caused by processing complex large messages, thereby helping to improve the scheduling efficiency of asynchronous processing tasks of large messages in distributed systems, and improving system throughput and responsiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of a method for scheduling message processing tasks provided in an embodiment of the present application;

[0041] Figure 2 It is a structural diagram of a scheduling device for message processing tasks provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0043] The following is an explanation of some terminology concepts involved in the embodiments of the present application.

[0044] Distributed system: A computing system consisting of multiple independent computing nodes connected through a network and working together.

[0045] Convolutional Neural Network (CNN): A deep learning model that has the ability to automatically extract features and is widely used in image recognition, speech recognition and other fields.

[0046] Priority scheduling: A task scheduling strategy that sorts tasks according to their priority, and high-priority tasks are assigned to available processing resources first.

[0047] The existing technology uses static priority rules to schedule message processing tasks. However, static priority rules cannot fully consider the dynamic characteristics of processing tasks, such as the actual processing complexity of processing tasks. Therefore, when facing large message asynchronous processing tasks, it may lead to unbalanced allocation of processing resources, reducing the system's throughput, success rate and responsiveness.

[0048] Please refer to the attached Figure 1 , attached Figure 1 is a flowchart of a method for scheduling message processing tasks provided by an embodiment of the present application, Figure 1 The method shown is applied to distributed systems. Figure 1 The method shown includes the following steps.

[0049] Step S110, extracting features of the message to be processed.

[0050] The message to be processed is, for example, a large message. For example, if the size of a message input to a computing node in a distributed system exceeds a cache size threshold or a processing capacity threshold of the computing node, the message is regarded as a large message, and the method provided by the embodiment of the present application is executed for the message. For example, when the size of a message exceeds 1KB, 1MB, or other specific number of bytes, it is defined as a large message.

[0051] The extracted features are key information for complexity analysis. For example, the size of the message, the transmission time of the message, the type of the message, the content complexity of the message, and the timeliness requirement of the message are extracted from the message as the features of the message.

[0052] The size of the message is positively correlated with the complexity of the processing task. Larger messages require more memory and computing resources to process, so the complexity of the processing task for larger messages will be higher.

[0053] The transmission time of a message reflects the delay of the message transmission on the network. A longer transmission time usually means network congestion or processing delay, which also increases the complexity of the message processing task.

[0054] The complexity of the message content, such as whether it contains encrypted or compressed data or how many times a third-party system is called.

[0055] The timeliness requirements of the message, for example, the message needs to return the processing result within 1 hour, the message needs to return the processing result before the end of the day, and the message needs to return the processing result within 5 days.

[0056] In some implementations, the message is further fragmented. Message fragmentation is to divide a large message into multiple smaller sub-messages for subsequent processing.

[0057] Step S120: Use a convolutional neural network to analyze the features of the message to obtain the complexity of the message processing task.

[0058] The processing task is to asynchronously process messages by computing nodes in a distributed system through processing resources. Convolutional neural networks include multiple convolutional layers, fully connected layers, and activation functions. Convolutional neural networks can learn the relationship between message features to predict the processing time and resource requirements of processing tasks.

[0059] In some embodiments, the convolutional neural network is trained based on the features of the sample message and the complexity of the sample message. The complexity of the sample message is used as a label for the sample message. The complexity of the sample message is obtained based on the historical processing time or resource consumption of the sample message. The features of the sample message include the size of the sample message, the transmission time of the sample message, the type of the sample message, the content complexity of the sample message, and the timeliness requirement of the sample message.

[0060] For example, the input message features are first converted into a format suitable for CNN input (such as an image or vector). Multiple convolutional layers are used to extract features, and the complexity is predicted based on the extracted features through a fully connected layer. Activation functions (such as ReLU) are used to introduce nonlinear characteristics.

[0061] Step S130: assigning priorities to the processing tasks based on the complexity of the processing tasks.

[0062] The priority of a processing task is negatively correlated with the complexity of the processing task. The higher the complexity of the processing task, the lower the priority of the processing task; the lower the complexity of the processing task, the higher the priority of the processing task.

[0063] In some embodiments, if the complexity level of the processing task belongs to the lowest complexity, the priority level of the processing task is determined to be the highest priority; if the complexity level of the processing task belongs to the medium complexity, the priority level of the processing task is determined to be the medium priority; if the complexity level of the processing task belongs to the highest complexity, the priority level of the processing task is determined to be the lowest priority.

[0064] Exemplarily, for the three processing tasks, Task 1, Task 2 and Task 3, the complexity of Task 1 is 10, and the level of Task 1 is the highest priority; the complexity of Task 2 is 20, and the level of Task 2 is medium priority; the complexity of Task 3 is 30, and the level of Task 3 is the lowest priority, so that Task 1 will be processed with the highest priority and Task 3 will be processed last.

[0065] In some implementations, when determining the priority, not only the complexity of the processing task is considered, but also the preset priority rules. The preset priority rules can be adjusted according to actual needs. For example, the preset priority rules can include factors such as the urgency of the message and the processing time requirement. For example, the priority is assigned to the processing task based on the complexity of the processing task, the urgency of the processing task, and the processing time requirement of the processing task.

[0066] According to the urgency of the task to be processed, an urgency level can be set. If the urgency level of the task to be processed is high urgency (need to be processed immediately), the priority level of the task to be processed is determined to be the highest priority; if the urgency level of the task to be processed is medium urgency (need to be processed in a short time), the priority level of the task to be processed is determined to be medium priority; if the urgency level of the task to be processed is low urgency (can be processed later), the priority level of the task to be processed is determined to be the lowest priority.

[0067] Regarding the processing time requirements of processing tasks, if a task is required to be completed within a specific time, a higher priority can be configured for the processing task, compared with the time requirements of other tasks.

[0068] Step S140: determining a scheduling strategy corresponding to the processing task based on the priority of the processing task.

[0069] In some implementations, a priority-based scheduling algorithm is used to sort the processing tasks. For example, if the priority level of a processing task is the highest priority, the processing task is preferentially allocated to available processing resources so that urgent or important processing tasks are processed preferentially, thereby improving the responsiveness of the system.

[0070] The scheduling strategy includes the scheduling frequency of processing tasks, and may also include the distribution method of processing tasks, the processing order between tasks, the allocation of resources, and the dynamic adjustment strategy to adapt to the changing processing environment.

[0071] In some implementations, a time slice rotation method is adopted to allocate execution time slices to the processing task based on the priority level of the processing task, and the number of execution time slices is positively correlated with the priority level of the processing task. In other words, if the priority level of the processing task is higher, more execution time slices are allocated to the processing task.

[0072] In some implementations, when a high-priority task is being processed, the execution frequency of the corresponding low-priority task is automatically increased.

[0073] In some implementations, the allocation of processing resources is also dynamically adjusted according to the priority of the processing task. For example, in the case of high load and high concurrency, more processing resources are allocated to high-priority tasks.

[0074] Step S150, executing the processing task according to the scheduling strategy.

[0075] In some implementations, the scheduling strategy for processing tasks is dynamically adjusted within a specified time period based on the processing task priority and processing resource conditions. For example, if the priority level of a processing task is the lowest priority and the processing resources for the processing task are insufficient, the scheduling frequency of the processing task is reduced so that high-priority processing tasks are processed in a timely manner, which helps to improve the throughput and responsiveness of the system.

[0076] In some implementations, the results of the completed processing tasks are merged and outputted, the processed sub-messages are merged into a complete message, and the processing results are outputted to a designated target.

[0077] Please refer to the attached Figure 2 , attached Figure 2 200 is a schematic diagram of a scheduling device for a message processing task provided in an embodiment of the present application. The device 200 includes:

[0078] An extraction unit 210, used to extract features of a message to be processed;

[0079] An analysis unit 220 is used to analyze the characteristics of the message using a convolutional neural network to obtain the complexity of the message processing task, where the processing task is that the computing node in the distributed system asynchronously processes the message through processing resources;

[0080] An allocating unit 230, configured to allocate a priority to the processing task based on the complexity of the processing task, wherein the priority of the processing task is negatively correlated with the complexity of the processing task;

[0081] A determination unit 240, configured to determine a scheduling strategy corresponding to the processing task based on the priority of the processing task;

[0082] The execution unit 250 is used to execute the processing task according to the scheduling strategy.

[0083] In some implementations, the convolutional neural network is trained based on features of sample messages and the complexity of the sample messages, and the complexity of the sample messages is obtained based on historical processing time or resource consumption of the sample messages.

[0084] In some embodiments, the allocation unit 230 is used to determine the priority level of the processing task as the highest priority if the complexity level of the processing task belongs to the lowest complexity; determine the priority level of the processing task as the medium priority if the complexity level of the processing task belongs to the medium complexity; and determine the priority level of the processing task as the lowest priority if the complexity level of the processing task belongs to the highest complexity.

[0085] In some implementations, the extraction unit 210 is used to extract the size of the message, the transmission time of the message, the type of the message, the content complexity of the message, and the timeliness requirement of the message from the message as the characteristics of the message.

[0086] In some implementations, the allocating unit 230 is configured to allocate priorities to the processing tasks based on the complexity, urgency, and processing time requirements of the processing tasks.

[0087] In some implementations, the determining unit 240 is configured to preferentially allocate the processing task to available processing resources if the priority level of the processing task is the highest priority.

[0088] In some implementations, the determination unit 240 is further configured to reduce the scheduling frequency of the processing task if the priority level of the processing task is the lowest priority and the processing resources of the processing task are insufficient.

[0089] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0090] A refers to B, which means that A is the same as B or A is a simple variant of B.

[0091] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in accordance with the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0092] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for scheduling message processing tasks, characterized in that: Applied to a distributed system, the method comprises: Extracting features of the message to be processed; Using a convolutional neural network to analyze the features of the message to obtain the complexity of a processing task of the message, wherein the processing task is that a computing node in the distributed system asynchronously processes the message through processing resources; Based on the complexity of the processing task, assigning a priority to the processing task, wherein the priority of the processing task is negatively correlated with the complexity of the processing task; Determining a scheduling strategy corresponding to the processing task based on the priority of the processing task; The processing task is executed according to the scheduling strategy.

2. The method according to claim 1, characterized in that The convolutional neural network is trained based on the characteristics of sample messages and the complexity of the sample messages, and the complexity of the sample messages is obtained based on the historical processing time or resource consumption of the sample messages.

3. The method according to claim 1, characterized in that The assigning a priority to the processing task based on the complexity of the processing task includes: If the level of complexity of the processing task belongs to the lowest complexity, determining the level of priority of the processing task to be the highest priority; If the complexity level of the processing task belongs to medium complexity, determining the priority level of the processing task to be medium priority; If the level of complexity of the processing task belongs to the highest complexity, the level of priority of the processing task is determined to be the lowest priority.

4. The method according to claim 1, characterized in that The extracting the features of the message to be processed includes: The size of the message, the transmission time of the message, the type of the message, the content complexity of the message, and the timeliness requirement of the message are extracted from the message as features of the message.

5. The method according to claim 1, characterized in that The assigning a priority to the processing task based on the complexity of the processing task includes: Priorities are assigned to the processing tasks based on the complexity, urgency, and processing time requirements of the processing tasks.

6. The method according to claim 1, characterized in that The determining, based on the priority of the processing task, a scheduling strategy corresponding to the processing task includes: If the priority level of the processing task is the highest priority, the processing task is preferentially allocated to available processing resources.

7. The method according to claim 1, characterized in that The method further comprises: If the priority level of the processing task is the lowest priority and the processing resources of the processing task are insufficient, the scheduling frequency of the processing task is reduced.

8. A scheduling device for message processing tasks, characterized in that: Applied to a distributed system, the device comprises: An extraction unit, used for extracting features of a message to be processed; An analysis unit, configured to analyze the features of the message using a convolutional neural network to obtain the complexity of a processing task of the message, wherein the processing task is that a computing node in the distributed system asynchronously processes the message through processing resources; an allocating unit, configured to allocate a priority to the processing task based on the complexity of the processing task, wherein the priority of the processing task is negatively correlated with the complexity of the processing task; A determination unit, configured to determine a scheduling strategy corresponding to the processing task based on the priority of the processing task; An execution unit is used to execute the processing task according to the scheduling strategy.

9. The device according to claim 8, characterized in that The convolutional neural network is trained based on the characteristics of sample messages and the complexity of the sample messages, and the complexity of the sample messages is obtained based on the historical processing time or resource consumption of the sample messages.

10. The device according to claim 8, characterized in that The allocation unit is used to determine the priority level of the processing task as the highest priority if the complexity level of the processing task belongs to the lowest complexity; determine the priority level of the processing task as the medium priority if the complexity level of the processing task belongs to the medium complexity; and determine the priority level of the processing task as the lowest priority if the complexity level of the processing task belongs to the highest complexity.

Images