AIGC engine scheduling method and equipment

Abstract

The invention provides an AIGC engine scheduling method and equipment, is applied to the technical field of artificial intelligence, and is suitable for equipment deployed with a multi-mode AIGC engine. The multi-mode AIGC engine comprises AIGC engines with different model functions, and the method comprises the steps that for any AIGC task to be processed, the AIGC task is divided into AIGC subtasks which can be processed through a single AIGC engine; screening out each target sub-task which conforms to the attribute information of each AIGC engine in the multi-mode AIGC engine from each AIGC sub-task; and determining a corresponding target AIGC engine for the at least one target sub-task according to the running state information of each AIGC engine in the multi-mode AIGC engine, and executing the at least one target sub-task in parallel through the corresponding target AIGC engine. According to the method and the device, the AIGC subtasks and the AIGC engines are respectively screened, and the corresponding AIGC engine is allocated to each AIGC subtask, so that AIGC multi-engine concurrent processing is realized. The number of processable tasks can be dynamically adjusted, and orderly execution of task processing is ensured.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to an AIGC engine scheduling method and device. Background Art

[0002] Artificial Intelligence Generated Content (AIGC) is a technology that can be applied to a variety of content creation fields, including but not limited to text, images, music, video, and audio.

[0003] In the prior art, the resource scheduling scheme for the AIGC multimodal engine adopts load balancing, which cannot reasonably allocate the resources of multiple AIGC multimodal engines, resulting in the accumulation of pending AIGC tasks. If the engine crashes, the AIGC tasks may be lost.

[0004] Therefore, how to coordinate AIGC multimodal engine resources in the process of AIGC task processing is the key. Summary of the invention

[0005] The embodiments of the present application provide an AIGC engine scheduling method and device for allocating a corresponding AIGC engine to each AIGC subtask, thereby realizing concurrent processing of multiple AIGC engines.

[0006] In a first aspect, an embodiment of the present application provides an AIGC engine scheduling method, which is applied to a device deployed with a multi-modal AIGC engine; the multi-modal AIGC engine includes an AIGC engine with different model functions, and the method includes:

[0007] For any AIGC task to be processed, dividing the AIGC task into AIGC subtasks that can be processed by a single AIGC engine;

[0008] Filtering out target subtasks that match the attribute information of each AIGC engine in the multimodal AIGC engine from each AIGC subtask;

[0009] According to the operation status information of each AIGC engine in the multimodal AIGC engine, a corresponding target AIGC engine is determined for at least one target subtask, and the at least one target subtask is executed in parallel by the corresponding target AIGC engine.

[0010] This application implements AIGC multi-engine concurrent processing by screening AIGC subtasks and AIGC engines respectively and assigning corresponding AIGC engines to each AIGC subtask. The number of tasks that can be processed can be adjusted dynamically to ensure the orderly execution of task processing; even if the system crashes, the currently processed task will have a completion status after the system restarts, without affecting other pending tasks.

[0011] Optionally, the attribute information of each AIGC engine in the multimodal AIGC engine includes a service identifier executable by the AIGC engine, a model function executable by the AIGC engine, and a model style executable by the AIGC engine;

[0012] Filtering out target subtasks that match the attribute information of each AIGC engine in the multimodal AIGC engine from each AIGC subtask includes:

[0013] According to the service identifier, the model function and the model style, target subtasks that meet the attribute information of each AIGC engine in the multimodal AIGC engine are screened out from each AIGC subtask.

[0014] AIGC engines are classified from multiple dimensions to ensure the flexibility of their configuration.

[0015] Optionally, determining a corresponding target AIGC engine for at least one target subtask according to the running status information of each AIGC engine in the multimodal AIGC engine includes:

[0016] According to the resource allocation principle of the AIGC engine, according to the operating status information of each AIGC engine in the multimodal AIGC engine and the operating mode of each AIGC engine in the multimodal AIGC engine, a corresponding target AIGC engine is determined for at least one target subtask, and the operating mode includes an exclusive mode for a single service identifier and a mixed mode for multiple service identifiers.

[0017] Optionally, if there is a first target subtask for which no corresponding target AIGC engine is determined, the number of elections for the first target subtask is incremented; the first target subtask is any one of the target subtasks;

[0018] If the number of elections for the first target subtask reaches the detection threshold, the first target subtask is processed according to the strategy.

[0019] By setting the detection threshold, it can be ensured that the target subtask is completed in time when there is no corresponding target AIGC engine processing, saving the resources of the AIGC engine.

[0020] Optionally, the detection threshold is the quotient of the scene processing time and the interval time for obtaining the first target subtask; the scene processing time is the target AIGC engine processing time that characterizes the first target task in a specified scene.

[0021] By setting the scene processing time, it is ensured that when the second target subtask is executed abnormally, the target AIGC engine occupied by it will be released, which is conducive to the reasonable allocation of AIGC engine resources.

[0022] Optionally, executing the at least one target subtask in parallel by a corresponding target AIGC engine includes:

[0023] Determine a scene processing duration that matches a second target subtask; the second target subtask is any one of the target subtasks;

[0024] When the duration for which the target AIGC engine corresponding to the second target subtask executes the second target subtask exceeds the matching scene processing duration, stopping execution of the second target subtask;

[0025] The running status information of the target AIGC engine corresponding to the second target subtask is set to an electable state.

[0026] Optionally, determining the scene processing duration matching the second target subtask includes:

[0027] Obtaining a historical processing time of a target AIGC engine corresponding to the second target subtask in executing a historical target subtask; the historical target subtask and the second target subtask have the same scenario;

[0028] The longer processing time between the historical processing time and the pre-configured processing time is determined as the scenario processing time matching the second target subtask.

[0029] By comparing and covering the historical processing duration of historical target subtasks with the same scenario as the second target subtask, the stability of the target AIGC engine when executing the second target subtask is ensured.

[0030] In a second aspect, an embodiment of the present application provides an AIGC engine scheduling device, including:

[0031] A processing module, configured to divide any AIGC task to be processed into AIGC subtasks that can be processed by a single AIGC engine;

[0032] A screening module, used to screen out target subtasks that meet the attribute information of each AIGC engine in the multimodal AIGC engine from each AIGC subtask;

[0033] An execution module is used to determine a corresponding target AIGC engine for at least one target subtask according to the operation status information of each AIGC engine in the multimodal AIGC engine, and to execute the at least one target subtask in parallel through the corresponding target AIGC engine.

[0034] This application implements AIGC multi-engine concurrent processing by screening AIGC subtasks and AIGC engines respectively and assigning corresponding AIGC engines to each AIGC subtask. The number of tasks that can be processed can be adjusted dynamically to ensure the orderly execution of task processing; even if the system crashes, the currently processed task will have a completion status after the system restarts, without affecting other pending tasks.

[0035] Optionally, the screening module is specifically used for:

[0036] The attribute information of each AIGC engine in the multimodal AIGC engine includes a service identifier executable by the AIGC engine, a model function executable by the AIGC engine, and a model style executable by the AIGC engine;

[0037] Filtering out target subtasks that match the attribute information of each AIGC engine in the multimodal AIGC engine from each AIGC subtask includes:

[0038] According to the service identifier, the model function and the model style, target subtasks that meet the attribute information of each AIGC engine in the multimodal AIGC engine are screened out from each AIGC subtask.

[0039] Optionally, the execution module is specifically used to:

[0040] The step of determining a corresponding target AIGC engine for at least one target subtask according to the operation status information of each AIGC engine in the multimodal AIGC engine includes:

[0041] According to the resource allocation principle of the AIGC engine, according to the operating status information of each AIGC engine in the multimodal AIGC engine and the operating mode of each AIGC engine in the multimodal AIGC engine, a corresponding target AIGC engine is determined for at least one target subtask, and the operating mode includes an exclusive mode for a single service identifier and a mixed mode for multiple service identifiers.

[0042] Optionally, the processing module is specifically used for:

[0043] If there is a first target subtask for which no corresponding target AIGC engine has been determined, the number of elections for the first target subtask is incremented; the first target subtask is any one of the target subtasks;

[0044] If the number of elections for the first target subtask reaches the detection threshold, the first target subtask is processed according to the strategy.

[0045] Optionally, the processing module is specifically used for:

[0046] The detection threshold is the quotient of the scene processing time and the interval time for obtaining the first target subtask; the scene processing time is the target AIGC engine processing time that characterizes the first target task in a specified scene.

[0047] Optionally, the screening module is specifically used for:

[0048] Determine a scene processing duration that matches a second target subtask; the second target subtask is any one of the target subtasks;

[0049] When the duration for which the target AIGC engine corresponding to the second target subtask executes the second target subtask exceeds the matching scene processing duration, stopping execution of the second target subtask;

[0050] The running status information of the target AIGC engine corresponding to the second target subtask is set to an electable state.

[0051] Optionally, the screening module is specifically used for:

[0052] The determining of the scene processing duration matching the second target subtask includes:

[0053] Obtaining a historical processing time of a target AIGC engine corresponding to the second target subtask in executing a historical target subtask; the historical target subtask and the second target subtask have the same scenario;

[0054] The longer processing time between the historical processing time and the pre-configured processing time is determined as the scenario processing time matching the second target subtask.

[0055] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes any of the AIGC engine scheduling methods described in the first aspect above.

[0056] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device. When the program runs on the computer device, the computer device executes any of the AIGC engine scheduling methods described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0058] Figure 1 A flowchart of an AIGC engine scheduling method provided by an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of a device architecture of a multi-modal AIGC engine provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of the structure of an AIGC engine scheduling device provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of the structure of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.

[0064] AIGC: (Artificial Intelligence Generated Content) refers to content generated by artificial intelligence technology. This technology can be applied to a variety of content creation fields, including but not limited to text, images, music, video and audio. The core of AIGC technology is to use machine learning and deep learning algorithms, especially the progress in natural language processing (NLP) and computer vision (CV), to automatically generate new content and creativity.

[0065] Multimodal: In the field of artificial intelligence and machine learning, it refers to the technology of combining and processing multiple types of data or information modalities. These modalities can include text, images, video, audio, touch, etc. The purpose of multimodal technology is to improve the system's understanding ability and performance by integrating data from different modalities.

[0066] AIGC Multimodal Engine: An engine that combines artificial intelligence generated content (AIGC) technology with multimodal data processing capabilities. This engine is able to understand and generate content across different media and formats, such as text, images, audio, and video. By integrating these different data modalities, the AIGC Multimodal Engine can provide a richer and more comprehensive user experience. For example: sdwebui, comfyui.

[0067] sd: Stable Diffusion, is a text-to-image generation model based on deep learning. It belongs to the Latent Diffusion Models and can generate high-quality, high-resolution and realistic images based on text input.

[0068] sdwebui: Browser interface for the Stable Diffusion model. Built on the Gradio library, it provides a friendly interface that allows users to generate images from text descriptions (txt2img) and modify existing images (img2img). It simplifies the interaction process with the Stable Diffusion model and integrates rich functions. It can meet higher degree of customized drawing needs by installing various plug-ins and importing multiple models, and generate a more controllable result.

[0069] comfyui: comfyui is an image generation tool built on Stable Diffusion technology. Through the node-based interface design, users can easily drag and drop and connect different models, modules and parameters to form a highly customized and reusable workflow without writing any code. It also supports the export and sharing of workflows, which not only speeds up project development, but also creates conditions for learning and communication between teams, greatly promoting the development and innovation of the creative industry.

[0070] like Figure 1 As shown, an AIGC engine scheduling method proposed in the present application is applicable to a device with a multi-modal AIGC engine deployed; the multi-modal AIGC engine includes an AIGC engine with different model functions, and the above method specifically includes the following steps:

[0071] Step S101 : for any AIGC task to be processed, divide the AIGC task into AIGC subtasks that can be processed by a single AIGC engine.

[0072] Specifically, after the AIGC task to be processed enters the multimodal AIGC engine, task division will be performed, and the AIGC task to be processed will be divided into multiple AIGC subtasks processed by a single AIGC engine, and the corresponding AIGC subtask will be processed by an AIGC engine with different model functions. An AIGC engine can process one type of AIGC subtask or two or more AIGC subtasks. In this application, if the AIGC task to be processed is an AIGC subtask that can be processed by a single AIGC engine, there is no need to divide the AIGC task to be processed.

[0073] For example, for a text-to-video task, it can be divided into a text-to-image subtask, a text-to-speech subtask, and a picture and speech generation video subtask, where the picture and speech generation video subtask depends on the results of the text-to-image subtask and the text-to-speech subtask.

[0074] Step S102, selecting target subtasks that meet the attribute information of each AIGC engine in the multimodal AIGC engine from each AIGC subtask.

[0075] Specifically, according to the attribute information of each AIGC engine in the multimodal AIGC engine, the AIGC subtasks that each AIGC engine can process are determined, and a target subtask that meets the requirements is selected from multiple AIGC subtasks for each AIGC engine. The number of AIGC subtasks is the same as the number of AIGC engines in the multimodal AIGC engine. If an AIGC engine in the multimodal AIGC engine is determined to be down according to the attribute information, the down AIGC engine will not be selected and assigned AIGC subtasks.

[0076] For example, the multimodal AIGC engine can be divided into a text-to-image engine, a text-to-speech engine, and a picture and speech generated video engine according to the attribute information of the AIGC engine. Then, the text-to-image subtask, the text-to-speech subtask, and the picture and speech generated video subtask are selected from each AIGC subtask.

[0077] For another example, if the executing AIGC engine deployed in the multimodal AIGC engine has 2 cultural image engines, 2 cultural speech engines, and 2 image and speech generation video engines, there are 3 cultural video processing tasks, and the 3 cultural video tasks are divided to generate: 3 cultural image tasks, 3 cultural speech tasks, and 3 image and speech generation videos; then 2 cultural image tasks, 2 cultural speech tasks, and 2 image and speech generation videos will be sent to the 2 cultural image engines, 2 cultural speech engines, and 2 image and speech generation video engines. If at this time one Wensheng image engine and one Wensheng voice engine crash, only one Wensheng image engine, one Wensheng voice engine, and two image and voice generation video engines are working, there are three Wensheng video processing tasks, and the three Wensheng video tasks are divided into: 3 Wensheng image tasks, 3 Wensheng voice tasks, and 3 image and voice generation videos; then 1 Wensheng image task, 1 Wensheng voice task, and 2 image and voice generation videos will be sent to 1 Wensheng image engine, 1 Wensheng voice engine, and 2 image and voice generation video engines.

[0078] Step S103: determining a corresponding target AIGC engine for at least one target subtask according to the operation status information of each AIGC engine in the multimodal AIGC engine, and executing at least one target subtask in parallel through the corresponding target AIGC engine.

[0079] Specifically, for any AIGC engine, if the AIGC engine is executing a subtask, the subtask is exclusive to the AIGC engine and cannot execute other subtasks. It is necessary to wait until the AIGC engine is in an idle state before the corresponding type of subtask can be assigned to the AIGC engine. Therefore, before redistributing tasks, the operating status information of each AIGC engine in the multimodal AIGC engine must be considered, and the corresponding target AIGC engine is determined for each target subtask. The target AIGC engine executes the assigned target subtask. Among them, each target subtask determines the corresponding target AIGC engine, including but not limited to random election and the furthest executable election.

[0080] For example, a text-based video task is divided into a text-based image subtask, a text-based speech subtask, and a picture and speech generation video subtask. The text-based image subtask and the text-based speech subtask can be processed in parallel to obtain a text-based image subresult and a text-based speech subresult, respectively. Then, the picture and speech generation video subtask is processed based on the text-based image subresult and the text-based speech subresult. Therefore, the start time of each target AIGC engine executing the target subtask may be the same or different.

[0081] The present invention implements AIGC multi-engine concurrent processing by screening AIGC subtasks and AIGC engines respectively and assigning a corresponding AIGC engine to each AIGC subtask. The number of processable tasks can be adjusted dynamically to ensure the orderly execution of task processing; even if the system crashes, the currently processed task will have a completion status after the system is restarted, without affecting other tasks to be processed.

[0082] The attribute information of each AIGC engine in the multimodal AIGC engine includes but is not limited to the business identifier executable by the AIGC engine, the model function executable by the AIGC engine, and the model style executable by the AIGC engine. In the above step S102, each target subtask that meets the attribute information of each AIGC engine in the multimodal AIGC engine is selected from each AIGC subtask, including: selecting each target subtask that meets the attribute information of each AIGC engine in the multimodal AIGC engine from each AIGC subtask according to the business identifier, model function and model style.

[0083] Specifically, the service identifier is used to distinguish the tasks sent by each institution (user). The service identifiers carried by the AIGC tasks to be processed sent by the same institution (user) to the multimodal AIGC engine are the same. For example, institution 1 (user) sends a Vincent video task A and a Vincent image task B to the multimodal AIGC engine, and the service identifiers carried by these two tasks are the same; institution 1 (user) sends a Vincent video task A1 to the multimodal AIGC engine, and institution 2 (user) sends a Vincent video task A2 to the multimodal AIGC engine, and the service identifiers carried by Vincent video task A1 and Vincent video task A2 are different. The service identifier executable by the AIGC engine means that the institution (user) is allowed to use the AIGC engine. If the multimodal AIGC engine allows institution 1 (user) and institution 2 (user) to use it, the existing institution 3 (user) wants to send the AIGC task to be processed to the multimodal AIGC engine, and the multimodal AIGC engine will not accept the AIGC task to be processed sent by institution 3 (user).

[0084] The model functions that can be executed by the AIGC engine are different. The model used by the AIGC engine generates different content. For example, for the same paragraph of text, there are three AIGC engines that can convert the text into pictures, voice and video respectively.

[0085] The model styles that the AIGC engine can execute are different from the styles of the content generated by the model used by the AIGC engine. For example, for the same paragraph of text, there are 4 AIGC engines that can convert the text into two-dimensional style, Chinese style, ancient style and European and American style respectively.

[0086] AIGC engines are classified from multiple dimensions to ensure the flexibility of their configuration.

[0087] Determining the corresponding target AIGC engine for at least one target subtask is in accordance with the resource allocation principle of the AIGC engine. According to the operating status information of each AIGC engine in the multimodal AIGC engine and the operating mode of each AIGC engine in the multimodal AIGC engine, the corresponding target AIGC engine is determined for at least one target subtask. The operating mode includes an exclusive mode for a single service identifier and a mixed mode for multiple service identifiers.

[0088] For example, in the existing AIGC engines sdwebui1: and sdwebui2, the business identifier of organization 1 (user) is appid1; the business identifier of organization 2 (user) is appid2. Among them, the AIGC engines sdwebui1: and sdwebui2 can process tasks with business identifiers appid1 and appid2 respectively.

[0089] Exclusive mode means that institution 1 (user) and institution 2 (user) each have relatively high requirements for the execution efficiency of the AIGC engine, and the corresponding engines carry corresponding tasks. Then sdwebui1 processes tasks with the business identifier appid1, and sdwebui2 processes tasks with the business identifier appid2.

[0090] The hybrid mode means that institution 1 (user) and institution 2 (user) each have low requirements for the execution efficiency of the AIGC engine, and share the AIGC engines sdwebui1: and sdwebui2. That is, institution 1 (user) and institution 2 (user) can be processed by sdwebui1 or sdwebui2. The cost of calling the AIGC engine sdwebui1 or sdwebui2 is calculated based on a series of indicators such as the number of task requests and GPU consumption.

[0091] Determining the corresponding target AIGC engine for each target subtask consumes multi-modal AIGC engine resources, so a detection threshold needs to be set. When the number of elections for the corresponding target AIGC engine assigned to the target subtask reaches the detection threshold, the target subtask needs to be abandoned, and the limited modal AIGC engine resources are reasonably allocated to other target subtasks that can be processed. The specific process is: if there is a first target subtask that has not determined the corresponding target AIGC engine, the number of elections for the first target subtask is incremented; the first target subtask is any one of the target subtasks; if the number of elections for the first target subtask reaches the detection threshold, the first target subtask is processed according to the strategy.

[0092] Specifically, the detection threshold is the quotient of the scene processing time and the interval time for obtaining the first target subtask. The detection threshold is used to dynamically evaluate the processing capacity of the first target subtask by the engine; the scene processing time is the target AIGC engine processing time that characterizes the first target task in a specified scene. For example, if the interval time for obtaining the first target subtask is 5s, the scene processing time of the first target task A is 60s, and the scene processing time of the first target task B is 40s, then the detection threshold of the first target task A is 12, and the detection threshold of the first target task B is 8. For the first target task A, if the number of elections of the corresponding target AIGC engine assigned to the first target task A reaches 12 times, the first target task A needs to be processed according to the strategy, which may be to discard the first target task A. In some possible embodiments, the first target task A can be returned to the multimodal AIGC engine to make the first target task A wait for the next election.

[0093] By setting the detection threshold, it can be ensured that the target subtask is completed in time when there is no corresponding target AIGC engine processing, saving the resources of the AIGC engine.

[0094] At least one target subtask is executed in parallel through the corresponding target AIGC engine, including: determining the scene processing duration that matches the second target subtask; the second target subtask is any one of the target subtasks; when the duration for the target AIGC engine corresponding to the second target subtask to execute the second target subtask exceeds the matching scene processing duration, stopping the execution of the second target subtask; setting the operating status information of the target AIGC engine corresponding to the second target subtask to an electable state.

[0095] Specifically, the scene processing time is the maximum processing time set for each target subtask in order to save AIGC engine resources. Therefore, the actual processing time of the second target subtask should be less than the scene processing time matched by the second target subtask; if the execution time of the target subtask in the target AIGC engine exceeds the matched scene processing time, the execution of the second target subtask is stopped, the target AIGC engine resources corresponding to the second target subtask are released, and the running status information of the target AIGC engine is updated, and the running status of the target AIGC engine is set to an electable state for the use of the target AIGC engine by other second target subtasks.

[0096] For example, the processing time of the scene matched by the second target subtask A is 60s, and the target AIGC engine A corresponding to the second target subtask actually takes 30s to process the second target subtask A. After the second target subtask A is processed, the resources of the target AIGC engine A will be released, and the running status information of the target AIGC engine A will be updated, and the running status information of the target AIGC engine A will be set to the electable state;

[0097] The scene processing time matched by the second target subtask B is 45s. The target AIGC engine B corresponding to the second target subtask has not been completed after processing the second target subtask B for 45s. At this time, the second target subtask B is interrupted, the resources of the target AIGC engine B are released, and the running status information of the target AIGC engine B is updated, and the running status information of the target AIGC engine B is set to the electable state.

[0098] By setting the scene processing time, it is ensured that when the second target subtask is executed abnormally, the target AIGC engine occupied by it will be released, which is conducive to the reasonable allocation of AIGC engine resources.

[0099] In some embodiments, it also includes obtaining the historical processing time of the target AIGC engine corresponding to the second target subtask in executing the historical target subtask; the historical target subtask and the second target subtask have the same scene; and the longer processing time between the historical processing time and the preconfigured processing time is determined as the scene processing time that matches the second target subtask.

[0100] For example, before the target AIGC engine corresponding to the second target subtask C executes the second target subtask C, the historical processing time of the historical target subtask with the same scenario as the second target subtask C will be obtained. If the historical processing time of the historical target subtask is 60s at this time, and the pre-configured processing time of the second target subtask C is 100s, the pre-configured processing time of the second target subtask C will not be replaced; if the pre-configured processing time of the second target subtask C is 50s, the pre-configured processing time of the second target subtask C will be replaced with 100s.

[0101] By comparing and covering the historical processing duration of historical target subtasks with the same scenario as the second target subtask, the stability of the target AIGC engine when executing the second target subtask is ensured.

[0102] like Figure 2As shown, a device architecture diagram of a multimodal AIGC engine deployed in the present application is provided, and the device includes a task loader, a preloader, a task coordinator, an engine memory and a multimodal engine architecture. The multimodal AIGC engine architecture is divided into three layers: a three-level AIGC engine, a two-level AIGC engine and a one-level AIGC engine. There is a detection counter at the three-level AIGC engine, and the two-level AIGC engine contains a state machine, which is used to record the status information of the first-level engine. The two-level AIGC engine also includes the IP, VIP, domain name, and MAC of each engine in the first-level engine; the first-level AIGC engine includes a program interrupter and a life counter, and sdwebui, sdwebui_vidieo, ast, and comfyul engines. It should be noted that the three-level engine, the two-level engine, and the first-level engine in the three-level engine can be horizontally expanded, and the number and type of engines in the first-level engine are not limited to those shown in the figure.

[0103] Step 1: Before the task loader initiates a task processing request, the task coordinator obtains the number of callable first-level AIGC engines through the service identifier and the AIGC engine type. The number of first-level AIGC engines is the number of AIGC tasks to be processed by the task loader.

[0104] Step 2: The organization (user) sends the AIGC task to be processed to the device deployed with the multimodal AIGC engine. The task loader divides the AIGC task into AIGC subtasks that can be processed by a single AIGC engine. At the same time, the task loader initiates a task detection request to the three-level AIGC engine. The three-level AIGC engine obtains the key task parameters of each AIGC task to be processed through the preloader: business identifier, engine type, model, and processing scenario. The three-level AIGC engine obtains the engine processing time of the scenario and the executable detection threshold of the scenario based on the key task parameters.

[0105] At the same time, the historical value of the engine processing time for this type of scene is retrieved from the engine memory through the business identifier of the AIGC task to be processed and the engine type, and compared with the time it takes the engine in the preloader to process the scene. If the time in the preloader is greater than the time in the memory, an overwrite operation is performed to update the data in the memory to the time provided by the preloader; otherwise, no operation is performed and the data in the memory remains unchanged.

[0106] Step 3: The third-level AIGC engine filters the service identification of the received task detection request. By identifying the service identification information contained in the task detection request, it ensures that only requests that meet the specific service identification can continue to be processed by the subsequent second-level AIGC engine, thereby improving the security and pertinence of the system; the request is further filtered by the AIGC engine, and the task detection request is screened and classified according to the preset second-level AIGC engine rules to ensure that only requests that meet the requirements of the specific second-level AIGC engine can enter the next stage; on this basis, a model filtering mechanism is introduced to perform more refined identification and judgment of the request, so as to accurately identify the second-level AIGC engine address of the request that meets the conditions.

[0107] Step 4: Enter the second-level AIGC engine, and select the only first-level AIGC engine that can execute each AIGC subtask according to the preset first-level AIGC engine rules. The first-level AIGC engine rules include but are not limited to random election, farthest executable election, etc., to ensure that each AIGC subtask can be evenly distributed to each first-level AIGC engine and fully utilize GPU resources; at the same time, obtain the first-level engine address so that the task can be further passed to the first-level AIGC engine for processing; the first-level AIGC engine maintains exclusivity during the task processing process, and the first-level AIGC engine will update the execution information to the state machine after the execution is completed.

[0108] If an executable first-level AIGC engine address is not selected in the second-level AIGC engine, the detection counter of the third-level AIGC engine is incremented to record the number of times the AIGC subtask is assigned to the first-level AIGC engine. If the AIGC subtask has not been able to select an executable first-level AIGC subtask engine address during continuous attempts, the detection counter will continue to increment until it reaches the scene executable detection threshold set in the previous step 2; when the detection counter reaches or exceeds the detection threshold, the task is determined to be a task that cannot be effectively assigned, and the abandonment mechanism is triggered to discard the AIGC subtask from the task loader to release engine resources and ensure the efficient operation of the multimodal engine.

[0109] Step 5: Enter the first-level AIGC engine, and through comprehensive consideration of the three key parameters of the task's business identifier, engine type, and model, elect an executable first-level AIGC engine, assign the AIGC subtask to the most suitable first-level AIGC engine, return the address of the first-level AIGC engine, and send the AIGC subtask to the corresponding first-level AIGC engine for execution. When the AIGC subtask starts to execute, the engine processing time information of the scene obtained in step 2 is written into the life counter, and the life counter is immediately started and starts to count in a decreasing manner. If the AIGC subtask has not been processed after the life counter reaches zero, the program interrupter is triggered, the first-level AIGC engine stops the execution of the AIGC subtask, triggers the abandonment mechanism, and abandons the AIGC subtask to release engine resources; if the first-level AIGC engine completes the processing of the AIGC subtask before the life counter reaches zero, the execution information of the state machine of the second-level engine is updated (other AIGC subtasks can enter the first-level AIGC engine, that is, the first-level AIGC engine can perform the next election).

[0110] This embodiment relies on a three-level engine through a task coordinator device to ensure that the number of scheduled tasks corresponds to the configuration of the AIGC engine executing the task, that is, the number of scheduled tasks is determined by the total number of first-level AIGC engines executing the task, ensuring that the tasks are executed in an orderly manner without omission or disorder.

[0111] like Figure 3 As shown, an AIGC engine scheduling device 300 is provided for an embodiment of the present application, including:

[0112] A processing module 301 is used for dividing any AIGC task to be processed into AIGC subtasks that can be processed by a single AIGC engine;

[0113] A screening module 302, configured to screen out target subtasks that meet the attribute information of each AIGC engine in the multimodal AIGC engine from each AIGC subtask;

[0114] The execution module 303 is used to determine a corresponding target AIGC engine for at least one target subtask according to the operation status information of each AIGC engine in the multimodal AIGC engine, and execute the at least one target subtask in parallel through the corresponding target AIGC engine.

[0115] This application implements AIGC multi-engine concurrent processing by screening AIGC subtasks and AIGC engines respectively and assigning corresponding AIGC engines to each AIGC subtask. The number of tasks that can be processed can be adjusted dynamically to ensure the orderly execution of task processing; even if the system crashes, the currently processed task will have a completion status after the system restarts, without affecting other pending tasks.

[0116] Optionally, the screening module 302 is specifically used for:

[0117] The attribute information of each AIGC engine in the multimodal AIGC engine includes a service identifier executable by the AIGC engine, a model function executable by the AIGC engine, and a model style executable by the AIGC engine;

[0118] Filtering out target subtasks that match the attribute information of each AIGC engine in the multimodal AIGC engine from each AIGC subtask includes:

[0119] According to the service identifier, the model function and the model style, target subtasks that meet the attribute information of each AIGC engine in the multimodal AIGC engine are screened out from each AIGC subtask.

[0120] Optionally, the execution module 303 is specifically used to:

[0121] The step of determining a corresponding target AIGC engine for at least one target subtask according to the operation status information of each AIGC engine in the multimodal AIGC engine includes:

[0122] According to the resource allocation principle of the AIGC engine, according to the operating status information of each AIGC engine in the multimodal AIGC engine and the operating mode of each AIGC engine in the multimodal AIGC engine, a corresponding target AIGC engine is determined for at least one target subtask, and the operating mode includes an exclusive mode for a single service identifier and a mixed mode for multiple service identifiers.

[0123] Optionally, the processing module 301 is specifically used for:

[0124] If there is a first target subtask for which no corresponding target AIGC engine has been determined, the number of elections for the first target subtask is incremented; the first target subtask is any one of the target subtasks;

[0125] If the number of elections for the first target subtask reaches the detection threshold, the first target subtask is processed according to the strategy.

[0126] Optionally, the processing module 301 is specifically used for:

[0127] The detection threshold is the quotient of the scene processing time and the interval time for obtaining the first target subtask; the scene processing time is the target AIGC engine processing time that characterizes the first target task in a specified scene.

[0128] Optionally, the screening module 302 is specifically used for:

[0129] Determine a scene processing duration that matches a second target subtask; the second target subtask is any one of the target subtasks;

[0130] When the duration for which the target AIGC engine corresponding to the second target subtask executes the second target subtask exceeds the matching scene processing duration, stopping execution of the second target subtask;

[0131] The running status information of the target AIGC engine corresponding to the second target subtask is set to an electable state.

[0132] Optionally, the screening module 302 is specifically used for:

[0133] The determining of the scene processing duration matching the second target subtask includes:

[0134] Obtaining a historical processing time of a target AIGC engine corresponding to the second target subtask in executing a historical target subtask; the historical target subtask and the second target subtask have the same scenario;

[0135] The longer processing time between the historical processing time and the pre-configured processing time is determined as the scenario processing time matching the second target subtask.

[0136] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 4 As shown, it includes at least one processor 401 and a memory 402 connected to the at least one processor. The specific connection medium between the processor 401 and the memory 402 is not limited in the embodiment of the present application. Figure 4 For example, the processor 401 and the memory 402 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0137] In the embodiment of the present application, the memory 402 stores instructions that can be executed by at least one processor 401. The at least one processor 401 can execute the steps of the above-mentioned multi-objective-based workflow scheduling method by executing the instructions stored in the memory 402.

[0138] Among them, the processor 401 is the control center of the computer device, which can use various interfaces and lines to connect various parts of the computer device, and process the AIGC tasks to be processed sent by the user by running or executing instructions stored in the memory 402 and calling the data stored in the memory 402, and select the corresponding target AIGC engine to execute for the multiple target subtasks of the AIGC tasks to be processed, and finally obtain the processing results of the AIGC tasks to be processed. Optionally, the processor 401 may include one or more processing units, and the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401. In some embodiments, the processor 401 and the memory 402 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0139] Processor 401 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0140] The memory 402 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 402 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 402 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 402 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0141] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned AIGC engine scheduling method.

[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0143] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0146] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An AIGC engine scheduling method, characterized in that: The method is applicable to a device equipped with a multi-modal AIGC engine; the multi-modal AIGC engine includes an AIGC engine with different model functions, and the method includes: For any AIGC task to be processed, dividing the AIGC task into AIGC subtasks that can be processed by a single AIGC engine; Filtering out target subtasks that match the attribute information of each AIGC engine in the multimodal AIGC engine from each AIGC subtask; According to the operation status information of each AIGC engine in the multimodal AIGC engine, a corresponding target AIGC engine is determined for at least one target subtask, and the at least one target subtask is executed in parallel by the corresponding target AIGC engine.

2. The method according to claim 1, characterized in that The attribute information of each AIGC engine in the multimodal AIGC engine includes a service identifier executable by the AIGC engine, a model function executable by the AIGC engine, and a model style executable by the AIGC engine; Filtering out target subtasks that match the attribute information of each AIGC engine in the multimodal AIGC engine from each AIGC subtask includes: According to the service identifier, the model function and the model style, target subtasks that meet the attribute information of each AIGC engine in the multimodal AIGC engine are screened out from each AIGC subtask.

3. The method according to claim 1, characterized in that The step of determining a corresponding target AIGC engine for at least one target subtask according to the operation status information of each AIGC engine in the multimodal AIGC engine includes: According to the resource allocation principle of the AIGC engine, according to the operating status information of each AIGC engine in the multimodal AIGC engine and the operating mode of each AIGC engine in the multimodal AIGC engine, a corresponding target AIGC engine is determined for at least one target subtask, and the operating mode includes an exclusive mode for a single service identifier and a mixed mode for multiple service identifiers.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: If there is a first target subtask for which no corresponding target AIGC engine has been determined, the number of elections for the first target subtask is incremented; the first target subtask is any one of the target subtasks; If the number of elections for the first target subtask reaches the detection threshold, the first target subtask is processed according to the strategy.

5. The method according to claim 4, characterized in that The detection threshold is the quotient of the scene processing time and the interval time for obtaining the first target subtask; the scene processing time is the target AIGC engine processing time that characterizes the first target task in a specified scene.

6. The method according to any one of claims 1 to 3, characterized in that: The executing the at least one target subtask in parallel by the corresponding target AIGC engine includes: Determine a scene processing duration that matches a second target subtask; the second target subtask is any one of the target subtasks; When the duration for which the target AIGC engine corresponding to the second target subtask executes the second target subtask exceeds the matching scene processing duration, stopping execution of the second target subtask; The running status information of the target AIGC engine corresponding to the second target subtask is set to an electable state.

7. The method according to claim 6, characterized in that The determining of the scene processing duration matching the second target subtask includes: Obtaining a historical processing time of a target AIGC engine corresponding to the second target subtask in executing a historical target subtask; the historical target subtask and the second target subtask have the same scenario; The longer processing time between the historical processing time and the pre-configured processing time is determined as the scenario processing time matching the second target subtask.

8. An AIGC engine scheduling device, characterized in that: include: A processing module, configured to divide any AIGC task to be processed into AIGC subtasks that can be processed by a single AIGC engine; A screening module, used to screen out target subtasks that meet the attribute information of each AIGC engine in the multimodal AIGC engine from each AIGC subtask; An execution module is used to determine a corresponding target AIGC engine for at least one target subtask according to the operation status information of each AIGC engine in the multimodal AIGC engine, and to execute the at least one target subtask in parallel through the corresponding target AIGC engine.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of any one of the methods of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 7.

Images