Asynchronous complex multi-model cooperative scheduling method and system

Abstract

The invention relates to the technical field of artificial intelligence, discloses an asynchronous complex multi-model cooperative scheduling method, system and device, a storage medium and a program product, and aims at solving the problem that in the prior art, when complex distributed asynchronous tasks which need to be completed through complex multi-model cooperation and are high in dependency are completed, the scheduling efficiency is high. And task scheduling cannot be performed simply, conveniently and efficiently. The method comprises the following steps: acquiring task dependency relationship information of a to-be-processed task, and splitting the to-be-processed task into a plurality of asynchronous subtasks according to the task dependency relationship information; receiving an asynchronous subtask receiving request, obtaining a source task processor corresponding to the asynchronous subtask receiving request, and determining a subtask type which can be processed by the source task processor based on complex multi-model information configured in the source task processor; and monitoring the task state of each asynchronous sub-task, and allocating each asynchronous sub-task to the corresponding task processor for processing based on the sub-task type which can be processed by the source task processor and the task state of each asynchronous sub-task.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an asynchronous complex multi-model collaborative scheduling method, system, device, storage medium and computer program product. Background Art

[0002] Technological development in the field of artificial intelligence has led to the gradual application of various large-scale models in different usage scenarios, and the types of tasks that various large-scale models can handle are increasing, and the content of the tasks that can be handled is becoming more and more complex. Therefore, the training and use requirements of various large-scale models are gradually increasing. As the processing tasks become more and more complex, when large models are working, it is often the case that an overall task requires multiple sub-models to complete the processing together; for example, when a generative artificial intelligence model is constructed by combining independent machine learning or deep learning models on specific computing resources and storage resources of multiple sub-models, an overall task requires multiple sub-models to cooperate, and multiple steps of sub-tasks are processed in a certain order, and the service resources of each sub-model need to be scheduled and coordinated. Therefore, when executing specific tasks, it is necessary not only to consider the dependencies between the sub-tasks, but also to consider how to better schedule and coordinate the service resources of each sub-model.

[0003] In the prior art, when the application scenario is an asynchronous task flow and complex multi-model task scheduling is required, most task scheduling methods are to schedule subtasks through a centralized task scheduler, which relies on manual configuration of task processing steps in the early stage. In addition, when there are many task flow links, strong dependencies, and complex system collaboration relationships, the scheduling occupies a large amount of processing resources and the scheduling effect is not good. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem in the prior art that it is impossible to simply and efficiently schedule complex tasks with strong dependencies that require complex multi-model collaboration to complete.

[0005] A first aspect of the present invention provides an asynchronous complex multi-model collaborative scheduling method, comprising:

[0006] Obtaining task dependency information of the task to be processed, wherein the task to be processed is a task that requires complex multi-model collaboration to complete, and the task dependency information is obtained after parsing and identifying the received task to be processed;

[0007] Split the pending task into multiple asynchronous subtasks based on task dependency information;

[0008] Receiving an asynchronous subtask access request and obtaining a source task processor corresponding to the asynchronous subtask access request, and determining a subtask type that can be processed by the source task processor based on complex multi-model information configured in the source task processor, wherein the asynchronous subtask access request is issued by an idle task processor at a preset interval;

[0009] The task status of each asynchronous subtask is monitored, and each asynchronous subtask is assigned to a corresponding task processor for processing based on the subtask type that can be processed by the source task processor and the task status of each asynchronous subtask.

[0010] Optionally, in a first implementation of the first aspect of the present invention, the task dependency information is a directed acyclic graph generated based on a processing flow of the task to be processed;

[0011] Before obtaining the task dependency information of the task to be processed, the method further includes:

[0012] Receive and parse the tasks to be processed to obtain the task material data, task interface address and interface request parameters;

[0013] Identify the task type of the task to be processed according to the task interface address and the interface request parameter;

[0014] Based on the task type of the task to be processed, a corresponding task flow template is searched, and a directed acyclic graph of the task to be processed is generated according to the task material data and the task flow template.

[0015] Optionally, in a second implementation of the first aspect of the present invention, splitting the to-be-processed task into a plurality of asynchronous subtasks according to the task dependency information includes:

[0016] Splitting the task to be processed based on the nodes in the directed acyclic graph of the task to be processed and the node connection order to obtain a plurality of asynchronous subtasks;

[0017] The monitoring of the task status of each asynchronous subtask comprises:

[0018] Obtain the number of predecessor tasks of each asynchronous subtask, and monitor the number of predecessor tasks that are successfully executed for each asynchronous subtask in real time;

[0019] The task status of each asynchronous subtask is obtained according to the number of predecessor tasks of each asynchronous subtask and the number of predecessor tasks of each asynchronous subtask that are successfully executed.

[0020] Optionally, in a third implementation of the first aspect of the present invention, the task state includes a waiting state and a ready state;

[0021] The step of obtaining the task status of each asynchronous subtask according to the number of predecessor tasks of each asynchronous subtask and the number of predecessor tasks of each asynchronous subtask that are successfully executed includes:

[0022] When the number of predecessor tasks of an asynchronous subtask is greater than the number of successfully executed predecessor tasks, the task state of the asynchronous subtask is a waiting state;

[0023] When the number of predecessor tasks of an asynchronous subtask is equal to the number of successfully executed predecessor tasks, the task state of the asynchronous subtask is a ready state;

[0024] The step of allocating each asynchronous subtask to a corresponding task processor for processing based on the subtask type processable by the source task processor and the task status of each asynchronous subtask comprises:

[0025] According to the target subtask type corresponding to the asynchronous subtask in the ready state, assigning the asynchronous subtask in the ready state to a task processor capable of processing the target subtask type;

[0026] The model required for executing the asynchronous subtask in the task processor is called to process the allocated asynchronous subtask.

[0027] Optionally, in a fourth implementation of the first aspect of the present invention, the task state further includes a running state;

[0028] The step of allocating the asynchronous subtask in the ready state to a task processor capable of processing the target subtask type according to the target subtask type corresponding to the asynchronous subtask in the ready state comprises:

[0029] Determine whether the source task processor corresponding to the asynchronous subtask access request is capable of processing the target subtask type corresponding to the asynchronous subtask in the ready state;

[0030] If so, the task state of the asynchronous subtask in the ready state is first changed to the running state, and then the asynchronous subtask whose task state is changed to the running state is allocated to the source task processor.

[0031] Optionally, in a fifth implementation of the first aspect of the present invention, the task status further includes an execution failure status;

[0032] After changing the task state of the asynchronous subtask in the ready state to the running state, and then assigning the asynchronous subtask whose task state is changed to the running state to the source task processor, the method further includes:

[0033] Receive heartbeat signals from the task processor that is processing the task;

[0034] Determine whether a heartbeat timeout occurs in the task processor that is processing the task according to the heartbeat signal, and if a heartbeat timeout occurs, change the task state of the asynchronous subtask being executed by the task processor from a running state to an execution failure state;

[0035] The task scheduling method further includes:

[0036] When the task status of the asynchronous subtask shows an execution failure state, the allocation and scheduling of the asynchronous subtask is stopped, and the pending task is terminated.

[0037] Optionally, in a sixth implementation of the first aspect of the present invention, the task status further includes a service failure status;

[0038] After changing the task state of the asynchronous subtask in the ready state to the running state, and then assigning the asynchronous subtask whose task state is changed to the running state to the source task processor, the method further includes:

[0039] Determine whether a business logic failure occurs when the task processor executes the asynchronous subtask. If so, change the task state of the asynchronous subtask from the running state to the business failure state;

[0040] The task scheduling method further includes:

[0041] When the task status of the asynchronous subtask reaches a business failure state, the allocation and scheduling of the asynchronous subtask is stopped, and the pending task is terminated.

[0042] A second aspect of the present invention provides an asynchronous complex multi-model collaborative scheduling system, comprising:

[0043] The business server is used to receive the tasks to be processed, parse and identify them to obtain task dependency information of the tasks to be processed, and send the task dependency information to the task scheduler, wherein the tasks to be processed are tasks that require complex multi-model collaboration to complete;

[0044] A task scheduler is used to obtain task dependency information of tasks to be processed; split the tasks to be processed into multiple asynchronous subtasks according to the task dependency information; receive an asynchronous subtask access request and obtain a source task processor corresponding to the asynchronous subtask access request, determine the subtask type that can be processed by the source task processor based on the complex multi-model information configured in the source task processor, monitor the task status of each asynchronous subtask, and assign each asynchronous subtask to a corresponding task processor for processing based on the subtask type that can be processed by the source task processor and the task status of each asynchronous subtask;

[0045] The task processor is used to send subtask access requests to the task scheduler at preset intervals when idle; and to process asynchronous subtasks assigned by the task scheduler.

[0046] The third aspect of the present invention provides an asynchronous complex multi-model collaborative scheduling device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the asynchronous complex multi-model collaborative scheduling device executes the steps of the above-mentioned asynchronous complex multi-model collaborative scheduling method.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions that, when executed on a computer, enable the computer to execute the steps of the above-mentioned asynchronous complex multi-model collaborative scheduling method.

[0048] The fifth aspect of the present invention provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the asynchronous complex multi-model collaborative scheduling method as described above are implemented.

[0049] In the technical solution provided by the present invention, the task dependency information of the task to be processed is obtained; the task to be processed is split into multiple asynchronous subtasks according to the task dependency information, an asynchronous subtask access request is received and the source task processor corresponding to the asynchronous subtask access request is obtained, and the subtask type that can be processed by the source task processor is determined based on the complex multi-model information configured in the source task processor, wherein the subtask access request is issued by an idle task processor at preset intervals; the task status of each asynchronous subtask is monitored, and each asynchronous subtask is assigned to the corresponding task processor for processing based on the subtask type that can be processed by the source task processor and the task status of each asynchronous subtask. This method can simply and efficiently schedule distributed asynchronous complex multi-model tasks when executing complex tasks with strong dependencies that require complex multi-model collaboration to complete. An asynchronous complex multi-model collaborative scheduling system, electronic device, readable storage medium and computer program product provided by the present invention also solve corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0051] Figure 1 It is a flowchart of a first embodiment of an asynchronous complex multi-model collaborative scheduling method according to an embodiment of the present invention;

[0052] Figure 2It is a flowchart of a second embodiment of the asynchronous complex multi-model collaborative scheduling method in an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of component deployment of a second embodiment of an asynchronous complex multi-model collaborative scheduling method according to an embodiment of the present invention;

[0054] Figure 4 It is a schematic diagram of an embodiment of an asynchronous complex multi-model collaborative scheduling system in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] Exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in a variety of forms, and should not be construed as limiting the present invention to the embodiments set forth herein. On the contrary, providing these exemplary embodiments enables the present invention to be more comprehensive and complete, and is more convenient for fully conveying the inventive concept to those skilled in the art. The same reference numerals in the figures represent the same or similar elements, components or parts, and thus their repeated description will be omitted.

[0056] Under the premise of being consistent with the technical concept of the present invention, the features, structures, characteristics or other details described in a specific embodiment do not exclude that they can be combined in one or more other embodiments in a suitable manner.

[0057] In the description of specific embodiments, the features, structures, characteristics or other details described in the present invention are intended to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can practice the technical solutions of the present invention without one or more of the specific features, structures, characteristics or other details.

[0058] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0059] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0060] The term "and / or" or "and / or" includes all combinations of any one or more of the associated listed items.

[0061] See also Figure 1The first embodiment of the asynchronous complex multi-model collaborative scheduling method in the embodiment of the present invention includes:

[0062] S101, obtaining task dependency information of tasks to be processed;

[0063] It is understandable that the execution subject of the present invention may be a task scheduling system, or a terminal or a server, such as a task scheduling server, which is not specifically limited here. The embodiment of the present invention is described by taking the task scheduling server as the execution subject as an example.

[0064] In this embodiment, the pending task refers to a complex task that requires multiple models to be completed collaboratively. After receiving the asynchronous complex multi-model collaborative scheduling request, the task scheduling server must first obtain the task dependency information corresponding to the pending task, wherein the task dependency information is obtained by the business server in response to parsing and identifying the received pending task. The task dependency information refers to the relationship between the processing order and processing level of multiple asynchronous subtasks contained in the pending task, and the task dependency information can be saved in the form of a pre-configured task structure process chain.

[0065] In a specific implementation, the business server may respond to and receive a task request issued by a user, obtain a specific task type corresponding to the task request issued by the user, and obtain a task dependency corresponding to the task to be processed according to the specific task type.

[0066] S102, splitting the task to be processed into multiple asynchronous subtasks according to the task dependency information;

[0067] After obtaining the task dependency information corresponding to the task to be scheduled, the task scheduler splits the received overall task to be processed into multiple asynchronous subtasks that can be processed independently according to the task dependency information, generates an asynchronous subtask table, and saves the relevant information of each asynchronous subtask in the asynchronous subtask table.

[0068] In a specific implementation, the task dependency information can be saved through a directed acyclic graph. When the task to be processed is split into multiple asynchronous subtasks, the task to be processed can be split based on the nodes in the directed acyclic graph of the task to be processed and the node connection order to obtain multiple asynchronous subtasks.

[0069] S103, receiving an asynchronous subtask access request and obtaining a source task processor corresponding to the asynchronous subtask access request, and determining a subtask type that can be processed by the source task processor based on complex multi-model information configured in the source task processor;

[0070] In this embodiment, the task scheduler does not actively search for a matching task processor based on the asynchronous subtask to be processed, but passively receives a task access request sent from an idle task processor. The task scheduler receives an asynchronous subtask access request sent by an idle task processor, and obtains the information of the corresponding source task processor based on the asynchronous subtask access request; specifically, the task processor information can be the code of the task processor, which can be sent to the task scheduler together with the asynchronous subtask access request, or can be included in the asynchronous subtask access request. After the task scheduler parses the received asynchronous subtask access request, it obtains the code of the task processor, thereby obtaining the source task processor corresponding to the asynchronous subtask access request.

[0071] In addition, when parsing the asynchronous subtask acceptance request, the complex multi-model information configured in the task processor that issued the request and contained in the asynchronous subtask acceptance request can also be obtained. The complex multi-model information refers to the model that can be called in the corresponding task processor; based on the complex multi-model information, the type of subtask that the task processor can handle can be determined.

[0072] In a specific implementation, a plurality of different models may be deployed in a task processor, and collaboration of multiple models in the same processor may be achieved, thereby achieving efficient use of resources; that is, a task processor may contain a plurality of subtask types that can be processed.

[0073] In this embodiment, an asynchronous subtask acceptance request is sent by an idle task processor to the task scheduler at preset time intervals; the preset time is determined based on information such as the processing time of each asynchronous subtask and the acceptable asynchronous subtask waiting time. The preset time can satisfy the requirement of not frequently sending requests to the task scheduler when there are idle task processors but no executable asynchronous subtasks in a ready state, and not delaying the allocation of tasks for too long when there are executable asynchronous subtasks in a ready state.

[0074] S104: Monitor the task status of each asynchronous subtask, and assign each asynchronous subtask to a corresponding task processor for processing based on the subtask type that can be processed by the source task processor and the task status of each asynchronous subtask.

[0075] In this embodiment, the asynchronous subtask table includes a task status field for the asynchronous subtask, and the task status in the asynchronous subtask table will be updated according to the execution status of each asynchronous subtask; therefore, the task scheduler in this embodiment will also monitor the task status of the asynchronous subtask.

[0076] After the task scheduler receives a task acceptance request from a task processor in an idle state, it first parses the task acceptance request to obtain the asynchronous subtask type that the task processor corresponding to the task acceptance request can handle; after obtaining the subtask type that the idle task processor can handle, it can allocate the asynchronous subtask in the ready state.

[0077] In a specific implementation, the task status includes a waiting state, a ready state, a running state, a successful state, and a failed state. Among them, a certain asynchronous subtask is in a waiting state, which means that based on the task dependency and the specific asynchronous subtask execution status, the current asynchronous subtask still has unfinished predecessor subtasks, and it is necessary to wait for the predecessor subtask to be processed before it can be allocated and processed; a certain asynchronous subtask is in a ready state, which means that based on the task dependency and the specific asynchronous subtask execution status, the current asynchronous subtask has no unfinished predecessor subtasks, and can be allocated and processed; a certain asynchronous subtask is in a running state, which means that the current asynchronous subtask is being executed by the assigned processor; a certain asynchronous subtask is in a successful state, which means that the asynchronous subtask has been successfully processed; a certain asynchronous subtask is in a failed state, which means that the asynchronous subtask fails to execute or the execution time exceeds the preset time.

[0078] Based on the subtask types that can be processed by the source task processor obtained in the previous steps and the task status of each asynchronous subtask, the asynchronous subtask in the ready state is assigned to the task processor of the target asynchronous subtask type corresponding to the asynchronous subtask in the current ready state for processing; when the asynchronous subtask is assigned to the task processor, the task status in the asynchronous subtask table needs to be updated to running.

[0079] In this embodiment, the asynchronous subtask table also includes the task code of each asynchronous subtask. After the asynchronous subtask is assigned to the task processor for processing, the code of the task processor will be recorded in the asynchronous subtask table accordingly to correspond the asynchronous subtask code and the task processor so as to record the specific execution status.

[0080] Based on the solution in this embodiment, all split asynchronous subtasks are assigned to corresponding models in the task processor for processing according to the task dependency information, until all asynchronous subtasks are executed and the tasks to be processed are completed.

[0081] The asynchronous complex multi-model collaborative scheduling method in this embodiment can provide a simple and efficient distributed asynchronous task scheduling method when executing complex tasks with strong dependencies that require multi-model collaboration. It can realize the combination of multiple model services and can quickly and stably carry out project deployment.

[0082] Please see Figure 2 as well as Figure 3 The second embodiment of the asynchronous complex multi-model collaborative scheduling method in the embodiment of the present invention includes:

[0083] This embodiment is explained by taking the AIGC (Artificial Intelligence Generated Content) system as an example. For example, it can be used in application scenarios such as AI (Artificial Intelligence) generation of AIGC advertising materials in the Internet field and automatic labeling of advertising materials. Specifically, it can be an asynchronous task flow that requires very complex multiple models to collaborate. Users can submit pending tasks and task materials required for pending tasks to the business server of AIGC through the interactive interface of the front-end page through the client or web page, and save the task materials required for the pending tasks to the file storage server through the business server; wherein, the file storage server can provide object storage services, such as S3 (Simple Storage Service) storage services and distributed file storage services for upload / download through https.

[0084] In a specific implementation manner, the task material required for the task to be processed can be submitted to the business server at the same time as the task to be processed, or it can be submitted in advance before the task to be processed is submitted; for example, it can be the same task material as the task executed last time. At this time, the business server and task scheduler can directly download the corresponding task material from the file storage server.

[0085] The business server receives the pending tasks from the user and parses them. According to the specific information contained in the pending tasks, it obtains the task interface address and interface request parameters, and simultaneously obtains the submitted task material data or obtains the corresponding task material data in the S3 file storage service according to the specific information of the pending tasks. The task type of the pending tasks is identified according to the task interface address and interface request parameters. According to the task type described in this step, it can be known what specific task the current pending task is, so as to subsequently obtain the task process relationship information corresponding to the specific type of task; in a specific implementation, the task process relationship information can be recorded through a directed acyclic graph.

[0086] Search for the corresponding task flow template based on the task type to be processed, wherein each task flow template contains information such as a fixed set of subtasks and the interdependence between asynchronous subtasks; fill in the variables in the task flow template according to the task material data to be processed, and finally generate a directed acyclic graph of the tasks to be processed.

[0087] S201, obtaining a directed acyclic graph generated based on the processing flow of the task to be processed;

[0088] S202, splitting the task to be processed based on the nodes in the directed acyclic graph of the task to be processed and the node connection order to obtain multiple asynchronous subtasks;

[0089] In this embodiment, after the business server generates a directed acyclic graph of the tasks to be processed, it will send the directed acyclic graph to the task scheduler. The task scheduler receives the directed acyclic graph, splits the tasks to be processed into multiple asynchronous sub-tasks according to the nodes in the directed acyclic graph and the order of node connection, and determines the connection information according to the connection relationship between each asynchronous sub-task and other asynchronous sub-tasks in the directed acyclic graph; wherein the connection information also includes the number of successor tasks of each asynchronous sub-task and the execution results of each successor task.

[0090] In a specific implementation, after the task to be processed is split, the coding and connection information of each asynchronous subtask is saved in an asynchronous subtask table.

[0091] S203, receiving an asynchronous subtask access request and obtaining a source task processor corresponding to the asynchronous subtask access request, and determining a subtask type that can be processed by the source task processor based on complex multi-model information configured in the source task processor;

[0092] An idle task processor sends an asynchronous subtask acceptance request to the task scheduler at preset intervals; the task scheduler receives the asynchronous subtask acceptance request and parses it to obtain the request parameters, wherein the request parameters include the subtask types supported by the task processor that issued the asynchronous subtask acceptance request. A task processor may include multiple different executable models, which may use different models according to the specific asynchronous subtasks. After execution, the memory may be released, which can improve the flexibility of task execution and save resources. When the task processor executes an asynchronous subtask, the multiple models contained in the task processor may be used to collaboratively execute the asynchronous subtask, thereby reducing data transmission between multiple model services.

[0093] The preset time is determined based on information such as the processing time of each asynchronous subtask and the acceptable waiting time of the asynchronous subtask. The preset time can meet the requirements of not frequently sending requests to the task scheduler when there are idle task processors but no executable asynchronous subtasks in a ready state, and not delaying too long in allocating tasks when there are executable asynchronous subtasks in a ready state.

[0094] In a specific implementation, in order to reduce the delay of the specific processing of asynchronous subtasks when the task processor is in an idle state, the preset time is preferably a shorter duration, and the shortest preset time can be calculated based on the number of task processors and the performance of asynchronous subtask table query. Take a specific example to illustrate, when the asynchronous subtask table query uses the MySQL query method, the performance of the asynchronous subtask table query can support SQL (Structured Query Language) queries n times per second, then the preset time is the duration obtained by dividing the number of task processors by n, and in a preferred embodiment, the preset time is 5 seconds.

[0095] In a preferred embodiment, after receiving a task acceptance request, the task scheduler will also assign it to a suitable task processor based on the specific information of the asynchronous subtask. For example, the task processor may have a variety of different characteristics, and different asynchronous subtasks can be assigned to a suitable task processor based on the characteristics of the task processor.

[0096] S204, obtaining the number of predecessor tasks of each asynchronous subtask, and monitoring in real time the number of predecessor tasks of each asynchronous subtask that are successfully executed;

[0097] S205, obtaining the task status of each asynchronous subtask according to the number of predecessor tasks of each asynchronous subtask and the number of predecessor tasks of each asynchronous subtask that are successfully executed;

[0098] In this embodiment, the constructed asynchronous subtask table includes a task status field for the asynchronous subtask, and the task status in the asynchronous subtask table will be updated according to the execution status of each asynchronous subtask; therefore, the task scheduler in this embodiment will also monitor the task status of the asynchronous subtask.

[0099] In a specific implementation, the task state of an asynchronous subtask includes a waiting state and a ready state; the task state of an asynchronous subtask can be monitored to determine whether it is a waiting state or a ready state by the total number of predecessor tasks of each asynchronous subtask and the number of predecessor tasks that are successfully executed. When the number of predecessor tasks of an asynchronous subtask is greater than the number of predecessor tasks that are successfully executed, that is, there are predecessor subtasks that are not successfully completed, the task state of the asynchronous subtask is a waiting state, and asynchronous subtasks in these states will not be assigned to the task processor for processing, but will only execute specific task scheduling when the task state is updated to the ready state; when the number of predecessor tasks of an asynchronous subtask is equal to the number of predecessor tasks that are successfully executed, that is, there are no predecessor asynchronous subtasks or all predecessor asynchronous subtasks are successfully executed, the task state of the asynchronous subtask is a ready state.

[0100] S206, according to the target subtask type corresponding to the asynchronous subtask in the ready state, assigning the asynchronous subtask in the ready state to a task processor capable of processing the target subtask type;

[0101] S207: Call the model required to execute the asynchronous subtask in the task processor, and process the allocated asynchronous subtask.

[0102] After receiving the asynchronous subtask access request from the idle task processor, the task processor assigns the subtask in the ready state to the task processor that can process it for processing according to the task type that the task processor can process, and marks the asynchronous subtask that has started to be processed as running. In this embodiment, the idle task processor actively sends the asynchronous subtask access request to the task scheduler instead of the task scheduler actively assigning tasks to the task processor, which simplifies the communication between the task scheduler and the task processor.

[0103] In a specific implementation, in response to an asynchronous subtask access request from an idle task processor, the task scheduler first obtains the target task type of the asynchronous subtask in the ready state to be assigned, and then determines whether there is an idle task processor that can process the corresponding target task type; if there is, the task state of the asynchronous subtask in the ready state is first changed to the running state, and then the asynchronous subtask whose task state is changed to running is assigned to the idle task processor for processing. This is to prevent the task scheduler from assigning the same asynchronous subtask to multiple different task processors for processing. Specifically, the status of the task can be updated through the update SQL statement with the where condition of the relational database. Taking a practical example for explanation, the program statement can be: "update task_table set status='running',processor_id='172.16.12.34'where task_id=123and status='ready'", wherein task represents an asynchronous subtask, task_id is the asynchronous subtask code, task_table refers to the asynchronous subtask table, status represents the subtask status, processor represents the task processor, and processor_id represents the task processor code. In this way, it can be ensured that only one task processor receives the asynchronous subtask with the asynchronous subtask code of 123, because after the update (status update) statement is executed successfully, the subtask status in the asynchronous subtask table will be updated to running, and for another SQL executed at the same time, this asynchronous subtask is no longer an asynchronous subtask in the ready state, and it will not be assigned.

[0104] In this embodiment, when performing task allocation, the subtask types that the source task processor can process are mainly determined based on the complex multi-model information configured in the source task processor. Therefore, it is actually based on the model required for executing the asynchronous subtask in the task processor to determine which corresponding task processor the asynchronous subtask should be assigned to. In this embodiment, the task processor can be provided by a variety of different hardware types, which can be a CPU server and a GPU server. Please continue to refer to Figure 3 Different task processing models can be configured in each different task processor, and multiple task processing models can be configured in each task processor; for example, a model that can meet the processing requirements using the CPU can be configured in a high-configuration CPU server so that it can share the work of the GPU server; a model that can call external multi-business shared model service functions for processing can be configured in a low-configuration CPU server; a variety of task processing models that can handle large model tasks can be configured in some GPU servers, such as models that generate videos based on artificial intelligence; models that can call other task execution tools based on generated data can be configured in other GPU servers, such as models that synthesize and export generated videos, etc. The task scheduler allocates different asynchronous subtasks based on the information of these specific task processors and executes the tasks to be processed.

[0105] After the asynchronous subtask in the ready state is assigned to the task processor that can handle the target subtask type, the model required to execute the asynchronous subtask in the task processor is called to process the assigned asynchronous subtask.

[0106] In this embodiment, the task status also includes successful execution. When the asynchronous subtask is assigned to the task processor for execution without any exception, and the logic code of the task processor determines that the business result of the subtask is correct, the task processor will feedback the successful execution information to the task scheduler. After receiving the information, the task scheduler will update the successful execution status to the asynchronous subtask table, and update the information corresponding to the successor asynchronous subtask of the successfully executed asynchronous subtask, for example, the number of successfully executed predecessor nodes corresponding to the successor asynchronous subtask of the successfully executed asynchronous subtask will be increased by 1.

[0107] In a specific implementation, the task status also includes an execution failure status; the asynchronous complex multi-model collaborative scheduling method described in this embodiment also includes: receiving a heartbeat signal from a task processor that is executing an asynchronous subtask; and judging whether a heartbeat timeout occurs in the task processor that is executing the asynchronous subtask based on the heartbeat signal, if a heartbeat timeout occurs, the task status of the asynchronous subtask being executed by the task processor is changed from a running state to an execution failure state; in addition, execution failure also includes that the task processor captures an exception during the execution of the asynchronous subtask. When the task status appears to be in an execution failure state, the allocation and scheduling of the asynchronous subtask is stopped, even if the current asynchronous subtask has a subsequent asynchronous subtask in a waiting state, it will no longer be executed, and the pending task will be directly terminated, and the specific information of the timeout failure will be fed back to the user.

[0108] In a specific implementation, the task status also includes a business failure status; the asynchronous complex multi-model collaborative scheduling method described in this embodiment also includes: determining whether a business logic failure occurs when the task processor executes an asynchronous subtask, and if so, changing the task status of the asynchronous subtask from a running status to a business failure status; the task scheduling method also includes: when a business failure status occurs in the task status, stopping the allocation and scheduling of the asynchronous subtask and ending the pending task. Specifically, the business failure status is different from the aforementioned execution failure status, and refers to a failure in business logic, such as problems with tasks submitted by users, input data, etc., which result in the asynchronous subtask being unable to obtain corresponding results. After the execution fails, the specific information of the execution failure is fed back to the system administrator for troubleshooting.

[0109] The embodiments of the present invention can simply and efficiently schedule distributed asynchronous tasks when executing complex artificial intelligence-related tasks with strong dependencies that require complex multi-model collaboration to complete. It can also conveniently combine and match multiple model services, fully utilize service resources, and quickly and stably deploy projects.

[0110] The above describes the asynchronous complex multi-model collaborative scheduling method in the embodiment of the present invention. The following describes the asynchronous complex multi-model collaborative scheduling system in the embodiment of the present invention. Figure 4 , an embodiment of the asynchronous complex multi-model collaborative scheduling system in an embodiment of the present invention includes:

[0111] The business server 401 is used to receive the tasks to be processed, parse and identify them to obtain task dependency information of the tasks to be processed, and send the task dependency information to the task scheduler, wherein the tasks to be processed are tasks that require complex multi-model collaboration to complete;

[0112] The task scheduler 402 is used to obtain task dependency information of the task to be processed; split the task to be processed into multiple asynchronous subtasks according to the task dependency information; split the task to be processed into multiple asynchronous subtasks according to the task dependency information; receive an asynchronous subtask access request and obtain a source task processor corresponding to the asynchronous subtask access request, determine the subtask type that can be processed by the source task processor based on the complex multi-model information configured in the source task processor, monitor the task status of each asynchronous subtask, and assign each asynchronous subtask to a corresponding task processor for processing based on the subtask type that can be processed by the source task processor and the task status of each asynchronous subtask;

[0113] The task processor 403 is used to send a subtask access request to the task scheduler at preset intervals when idle; and process the asynchronous subtasks assigned by the task scheduler.

[0114] The embodiments of the present invention can simply and efficiently schedule distributed asynchronous complex multi-model tasks when executing complex tasks with strong dependencies that require complex multi-model collaboration.

[0115] In another embodiment of the present application, the task dependency information is a directed acyclic graph generated based on the processing flow of the task to be processed;

[0116] The business server 401 is specifically used to receive and parse the tasks to be processed to obtain task material data, task interface address and interface request parameters;

[0117] Identify the task type of the task to be processed according to the task interface address and the interface request parameter;

[0118] Based on the task type of the task to be processed, a corresponding task flow template is searched, and a directed acyclic graph of the task to be processed is generated according to the task material data and the task flow template.

[0119] In another embodiment of the present application, the task scheduler 402 is specifically used for:

[0120] Splitting the task to be processed based on the nodes in the directed acyclic graph of the task to be processed and the node connection order to obtain a plurality of asynchronous subtasks;

[0121] as well as,

[0122] Obtain the number of predecessor tasks of each asynchronous subtask, and monitor the number of predecessor tasks that are successfully executed for each asynchronous subtask in real time;

[0123] The task status of each asynchronous subtask is obtained according to the number of predecessor tasks of each asynchronous subtask and the number of predecessor tasks of each asynchronous subtask that are successfully executed.

[0124] In another embodiment of the present application, the task state includes a waiting state and a ready state;

[0125] The task scheduler 402 is also specifically used for:

[0126] When the number of predecessor tasks of an asynchronous subtask is greater than the number of successfully executed predecessor tasks, the task state of the asynchronous subtask is a waiting state;

[0127] When the number of predecessor tasks of an asynchronous subtask is equal to the number of successfully executed predecessor tasks, the task state of the asynchronous subtask is a ready state;

[0128] as well as,

[0129] According to the target subtask type corresponding to the asynchronous subtask in the ready state, assigning the asynchronous subtask in the ready state to a task processor capable of processing the target subtask type;

[0130] The model required for executing the asynchronous subtask in the task processor is called to process the allocated asynchronous subtask.

[0131] In another embodiment of the present application, the task status also includes a running status;

[0132] The task scheduler 402 is also specifically used for:

[0133] Determine whether the source task processor corresponding to the asynchronous subtask access request is capable of processing the target subtask type corresponding to the asynchronous subtask in the ready state;

[0134] If so, the task state of the asynchronous subtask in the ready state is first changed to the running state, and then the asynchronous subtask whose task state is changed to the running state is allocated to the source task processor.

[0135] In another embodiment of the present application, the task status also includes an execution failure status;

[0136] The task scheduler 402 is also specifically used for:

[0137] Receive heartbeat signals from the task processor that is processing the task;

[0138] Determine whether a heartbeat timeout occurs in the task processor that is processing the task according to the heartbeat signal, and if a heartbeat timeout occurs, change the task state of the asynchronous subtask being executed by the task processor from a running state to an execution failure state;

[0139] Furthermore, when the task status of the asynchronous subtask shows an execution failure status, the allocation and scheduling of the asynchronous subtask is stopped, and the pending task is terminated.

[0140] In another embodiment of the present application, the task status also includes a business failure status;

[0141] The task scheduler 402 is also specifically used for:

[0142] Determine whether a business logic failure occurs when the task processor executes the asynchronous subtask. If so, change the task state of the asynchronous subtask from the running state to the business failure state;

[0143] Furthermore, when the task status of the asynchronous subtask reaches a business failure state, the allocation and scheduling of the asynchronous subtask is stopped, and the pending task is terminated.

[0144] The embodiments of the present invention can simply and efficiently schedule distributed asynchronous tasks when executing relatively complex distributed asynchronous tasks that require the collaboration of multiple models and in which the various parts of the tasks have strong dependencies. It can also conveniently combine and match multiple model services, fully utilize service resources, and quickly and stably deploy projects.

[0145] Based on the same inventive concept, the embodiment of this specification also provides an asynchronous complex multi-model collaborative scheduling device, a memory and at least one processor, wherein the memory stores instructions;

[0146] The at least one processor calls the instructions in the memory to enable the asynchronous complex multi-model collaborative scheduling device to perform the steps of the asynchronous complex multi-model collaborative scheduling method described in any of the above embodiments.

[0147] Furthermore, the above-mentioned asynchronous complex multi-model collaborative scheduling device may also include a transmission module and an input-output module, wherein the transmission module is connected to the above-mentioned processor, and the input-output module is connected to the above-mentioned processor.

[0148] The present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the asynchronous complex multi-model collaborative scheduling method described in any of the above embodiments.

[0149] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the exemplary embodiments described in the present invention can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation method of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the present invention. When the computer program is executed by a data processing device, the computer-readable medium is enabled to implement the asynchronous complex multi-model collaborative scheduling method described in any embodiment of the present invention.

[0150] In addition, the computer program for implementing the asynchronous complex multi-model collaborative scheduling method described in any of the above embodiments can be stored on one or more computer-readable media. The computer-readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0151] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0152] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0153] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0154] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An asynchronous complex multi-model collaborative scheduling method, characterized in that: The asynchronous complex multi-model collaborative scheduling method includes: Obtaining task dependency information of the task to be processed, wherein the task to be processed is a task that requires complex multi-model collaboration to complete, and the task dependency information is obtained after parsing and identifying the received task to be processed; Split the pending task into multiple asynchronous subtasks based on task dependency information; Receiving an asynchronous subtask access request and obtaining a source task processor corresponding to the asynchronous subtask access request, and determining a subtask type that can be processed by the source task processor based on complex multi-model information configured in the source task processor, wherein the asynchronous subtask access request is issued by an idle task processor at a preset interval; The task status of each asynchronous subtask is monitored, and each asynchronous subtask is assigned to a corresponding task processor for processing based on the subtask type that can be processed by the source task processor and the task status of each asynchronous subtask.

2. The asynchronous complex multi-model collaborative scheduling method according to claim 1, characterized in that: The task dependency information is a directed acyclic graph generated based on the processing flow of the tasks to be processed; Before obtaining the task dependency information of the task to be processed, the method further includes: Receive and parse the tasks to be processed to obtain the task material data, task interface address and interface request parameters; Identify the task type of the task to be processed according to the task interface address and the interface request parameter; Based on the task type of the task to be processed, a corresponding task flow template is searched, and a directed acyclic graph of the task to be processed is generated according to the task material data and the task flow template.

3. The asynchronous complex multi-model collaborative scheduling method according to claim 2, characterized in that: The step of splitting the task to be processed into multiple asynchronous subtasks according to the task dependency information includes: Splitting the task to be processed based on the nodes in the directed acyclic graph of the task to be processed and the node connection order to obtain a plurality of asynchronous subtasks; The monitoring of the task status of each asynchronous subtask comprises: Obtain the number of predecessor tasks of each asynchronous subtask, and monitor the number of predecessor tasks that are successfully executed for each asynchronous subtask in real time; The task status of each asynchronous subtask is obtained according to the number of predecessor tasks of each asynchronous subtask and the number of predecessor tasks of each asynchronous subtask that are successfully executed.

4. The asynchronous complex multi-model collaborative scheduling method according to claim 3, characterized in that: The task status includes a waiting state and a ready state; The step of obtaining the task status of each asynchronous subtask according to the number of predecessor tasks of each asynchronous subtask and the number of predecessor tasks of each asynchronous subtask that are successfully executed includes: When the number of predecessor tasks of an asynchronous subtask is greater than the number of successfully executed predecessor tasks, the task state of the asynchronous subtask is a waiting state; When the number of predecessor tasks of an asynchronous subtask is equal to the number of successfully executed predecessor tasks, the task state of the asynchronous subtask is a ready state; The step of allocating each asynchronous subtask to a corresponding task processor for processing based on the subtask type processable by the source task processor and the task status of each asynchronous subtask comprises: According to the target subtask type corresponding to the asynchronous subtask in the ready state, assigning the asynchronous subtask in the ready state to a task processor capable of processing the target subtask type; The model required for executing the asynchronous subtask in the task processor is called to process the allocated asynchronous subtask.

5. The asynchronous complex multi-model collaborative scheduling method according to claim 4, characterized in that: The task status also includes a running status; The step of allocating the asynchronous subtask in the ready state to a task processor capable of processing the target subtask type according to the target subtask type corresponding to the asynchronous subtask in the ready state comprises: Determine whether the source task processor corresponding to the asynchronous subtask access request is capable of processing the target subtask type corresponding to the asynchronous subtask in the ready state; If so, the task state of the asynchronous subtask in the ready state is first changed to the running state, and then the asynchronous subtask whose task state is changed to the running state is allocated to the source task processor.

6. The asynchronous complex multi-model collaborative scheduling method according to claim 5, characterized in that: The task status also includes an execution failure status; After changing the task state of the asynchronous subtask in the ready state to the running state, and then assigning the asynchronous subtask whose task state is changed to the running state to the source task processor, the method further includes: Receive heartbeat signals from the task processor that is processing the task; Determine whether a heartbeat timeout occurs in the task processor that is processing the task according to the heartbeat signal, and if a heartbeat timeout occurs, change the task state of the asynchronous subtask being executed by the task processor from a running state to an execution failure state; The task scheduling method further includes: When the task status of the asynchronous subtask shows an execution failure state, the allocation and scheduling of the asynchronous subtask is stopped, and the pending task is terminated.

7. The asynchronous complex multi-model collaborative scheduling method according to claim 6, characterized in that: The task status also includes a business failure status; After changing the task state of the asynchronous subtask in the ready state to the running state, and then assigning the asynchronous subtask whose task state is changed to the running state to the source task processor, the method further includes: Determine whether a business logic failure occurs when the task processor executes the asynchronous subtask. If so, change the task state of the asynchronous subtask from the running state to the business failure state; The task scheduling method further includes: When the task status of the asynchronous subtask reaches a business failure state, the allocation and scheduling of the asynchronous subtask is stopped, and the pending task is terminated.

8. An asynchronous complex multi-model collaborative scheduling system, characterized in that: The asynchronous complex multi-model collaborative scheduling system includes: The business server is used to receive the tasks to be processed, parse and identify them to obtain task dependency information of the tasks to be processed, and send the task dependency information to the task scheduler, wherein the tasks to be processed are tasks that require complex multi-model collaboration to complete; A task scheduler is used to obtain task dependency information of tasks to be processed; split the tasks to be processed into multiple asynchronous subtasks according to the task dependency information; split the tasks to be processed into multiple asynchronous subtasks according to the task dependency information; receive an asynchronous subtask access request and obtain a source task processor corresponding to the asynchronous subtask access request, determine the subtask type that can be processed by the source task processor based on the complex multi-model information configured in the source task processor, monitor the task status of each asynchronous subtask, and assign each asynchronous subtask to a corresponding task processor for processing based on the subtask type that can be processed by the source task processor and the task status of each asynchronous subtask; The task processor is used to send subtask access requests to the task scheduler at preset intervals when idle; and to process asynchronous subtasks assigned by the task scheduler.

9. An asynchronous complex multi-model collaborative scheduling device, characterized in that: The asynchronous complex multi-model collaborative scheduling device includes: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the asynchronous complex multi-model collaborative scheduling device to perform the steps of the asynchronous complex multi-model collaborative scheduling method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the program / instructions are executed by the processor, the steps of the asynchronous complex multi-model collaborative scheduling method as claimed in any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the asynchronous complex multi-model collaborative scheduling method as described in any one of claims 1 to 7 are implemented.

Images