Artificial intelligence image and video processing workflow construction and scheduling method

Abstract

The invention provides an artificial intelligence image and video processing workflow construction and scheduling method, which comprises the following steps of: packaging a ComfyUI (User Interface) into a container mirror image by using Docker so as to simplify the installation and deployment step of the ComfyUI; constructing a task workflow scheduling system by using an intelligent agent based on a large language model; when a user uses a ComfyUI function, only a segment of text description needs to be input, the text description is submitted to a scheduler, and the intelligent body uses the task workflow scheduling system to process a ComfyUI task. According to the method, a proper and idle ComfyUI node can be selected for scheduling according to a decision based on an LLM + Agent mechanism, and meanwhile, a workflow result is obtained and displayed to an upper-layer user, so that the utilization rate of the ComfyUI is increased.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a method for constructing and scheduling an artificial intelligence image and video processing workflow. Background Art

[0002] Most of the existing AI image and video processing workflow construction and scheduling work is manually deployed, which is troublesome and repetitive. In addition, AI image and video processing workflow construction and scheduling are mainly deployed on personal computers or a cloud host, which are generally only used by individuals and have low utilization rates. ComfyUI is currently difficult to deploy and can only be used by a single person in a visual way. Summary of the invention

[0003] In response to the above problems, the present invention provides a method for constructing and scheduling an AI image and video processing workflow based on a ComfyUI carrier.

[0004] The method for constructing and scheduling an artificial intelligence image and video processing workflow provided by the present invention comprises the steps of:

[0005] S1. Use Docker to package ComfyUI into a container image to simplify the installation and deployment steps of ComfyUI;

[0006] S2. Use an agent based on a large language model to build a task workflow scheduling method;

[0007] S3. When the user uses the ComfyUI function, he only needs to enter a text description, which is submitted to the scheduler. The intelligent experience will use the task workflow scheduling system to process the ComfyUI task.

[0008] further,

[0009] The step S1 comprises:

[0010] Install the standard Docker or Podman container runtime environment;

[0011] Download the ComfyUI code package;

[0012] Create a Dockerfile container image build file;

[0013] Use the Docker or Podman build tool, the ComfyUI code package, and the Dockerfile container image build file to build the ComfyUI container image;

[0014] Upload the built ComfyUI container image to the Docker image repository or other third-party container image repository.

[0015] further,

[0016] The ComfyUI container image only contains the environment of ComfyUI runtime, and does not include the workflow carrier content during specific execution.

[0017] further,

[0018] The step S2 comprises the steps of:

[0019] S2a, placing all workflow carrier contents in a shared storage, and then mounting the shared storage to all GPU host nodes that will run ComfyUI instances. One GPU host node is a physical machine with at least one GPU card, each of which is used to run a Docker container, in which the ComfyUI program runs. A physical machine has several GPU cards, which means it runs several ComfyUI instances.

[0020] S2b, installing a Docker or Podman container runtime environment on multiple GPU host nodes, and then allocating workflows based on the ComfyUI container image;

[0021] S2c, submitting the type, metadata information and other natural language descriptions of the ComfyUI workflow carrier to the large language model for learning;

[0022] S2d. On the scheduler node, deploy an agent based on the large language model to access the large language model. One scheduler node is a host, which is used to assign a task to an idle ComfyUI instance. There is at least one scheduler node.

[0023] further,

[0024] The workflow carrier content is configuration files and related model files;

[0025] The type and metadata information of the ComfyUI workflow carrier include access address, supported models and templates.

[0026] further,

[0027] In the step S2b, the workflow based on the ComfyUI container image distribution includes the following steps:

[0028] Pull the container image produced in step S1 from the Docker image repository or other third-party container image repository, so that the multiple GPU host nodes directly enter the ComfyUI runtime environment;

[0029] According to the type of the workflow, the graphics processor cards of different graphics processor host nodes are allocated to different types of workflows, and each time a graphics processor card is allocated to a type of workflow, the type of workflow has a runtime instance for processing a work task;

[0030] According to the assigned workflow, a ComfyUI container instance is started on each graphics processor card, and the graphics processor card and workflow carrier content used are specified during startup.

[0031] further,

[0032] In the step S2d, the user only needs to call the interface provided by the agent to submit the ComfyUI task.

[0033] further,

[0034] In the step S2d, the calling includes the following steps:

[0035] The agent implemented based on the large language model submits the ComfyUI task description to the large language model for analysis;

[0036] The large language model provides a group of related ComfyUI running instance nodes, that is, the ComfyUI instances are classified, each type of ComfyUI instance processes an artificial intelligence processing task, the ComfyUI instances of the same type can process the same task, and ComfyUI instances of different types cannot process each other's tasks;

[0037] The agent selects an unassigned ComfyUI instance as ComfyUI instance A, then submits the current ComfyUI task to the ComfyUI instance A, and marks ComfyUI instance A as assigned;

[0038] If ComfyUI instance A returns that it does not support the current ComfyUI task or fails to process, the intelligent body will recall it, exclude the recalled ComfyUI instance A, and reselect other unassigned ComfyUI instances until a ComfyUI instance node that can normally process the current ComfyUI task is found.

[0039] further,

[0040] A labeling function is added when training the large language model, and the labels are all predefined, and the labels are used to specify specific and unambiguous information.

[0041] The method for constructing and scheduling an AI image and video processing workflow provided by the present invention re-customizes and packages ComfyUI, simplifies the ComfyUI deployment cycle, and constructs a task workflow construction and scheduling mechanism at the same time. The upper layer can be used by multiple people at the same time. When submitting tasks during use, the system will automatically construct a ComfyUI workflow, and according to the decision of the agent (Agent) (i.e., LLM+Agent) mechanism implemented based on the large language model (LLM), select a suitable and idle ComfyUI node for scheduling, and obtain the results of the workflow at the same time, and display them to the upper layer users; thereby improving the utilization rate of ComfyUI.

[0042] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flow chart of a method for constructing and scheduling an artificial intelligence image and video processing workflow according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions. The terms "first", "second", "third", etc. in the specification and claims of this application or the above-mentioned figures are used to distinguish different objects, not to describe a specific order or a primary and secondary relationship. "Multiple" appearing in this application refers to more than two (including two).

[0047] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0048] Figure 1 This is a flow chart of the method for constructing and scheduling AI image and video processing workflows provided by the present invention. Figure 1 , the method for constructing and scheduling the AI ​​image and video processing workflow includes the following steps.

[0049] S1. Use Docker to package ComfyUI into a container image to simplify the installation and deployment steps of ComfyUI, including the following steps:

[0050] S1a. Install the standard Docker or Podman container runtime environment.

[0051] S1b. Download the ComfyUI code package from GitLab or other source code repository.

[0052] S1c. Create a Dockerfile container image build file.

[0053] S1d. Use Docker or Podman build tools, ComfyUI code package, and the above Dockerfile container image build file to build the ComfyUI container image to facilitate and simplify the distribution and batch deployment of the ComfyUI workflow package.

[0054] S1e. Upload the built ComfyUI container image to the Docker image repository or other third-party container image repository for subsequent use.

[0055] The constructed ComfyUI container image only contains the environment of ComfyUI runtime, and does not include the workflow carrier content during specific execution. This mode is intended to simplify the construction and distribution of the ComfyUI runtime environment and reuse the existing workflow carrier content. It should be noted that the workflow carrier content is the configuration file of the ComfyUI workflow and its related model files. The files, especially the model files, are generally large, such as several GB or even more than ten GB. Such files are not suitable for packaging in container images.

[0056] S2. Build a task workflow scheduling method based on LLM+Agent, which includes the following steps:

[0057] S2a. Put all workflow carrier contents (i.e. configuration files and related model files) in shared storage, and then mount the shared storage to all GPU host nodes that will run ComfyUI instances. A host node can be a physical machine with at least one GPU card. Each GPU card is used to run a Docker container, in which the ComfyUI program runs. In other words, a physical machine with several GPU cards can run several ComfyUI instances (each ComfyUI instance can process a workflow task independently). There can be multiple such physical machines, each of which is called a GPU host node;

[0058] S2b, on multiple (more than 1) graphics processing unit (GPU) host nodes, install the Docker or Podman container runtime environment, and then distribute the workflow based on the ComfyUI container image, including the following steps:

[0059] S2bA, pulling the container image produced in step S1 from a Docker image repository or other third-party container image repository, so that the multiple GPU host nodes directly enter the ComfyUI runtime environment;

[0060] S2bB, staff or administrators can allocate GPU cards of different GPU host nodes to different types of workflows according to the type of workflow. Each time a GPU card is allocated to a certain type of workflow, the workflow of that type will have a runtime instance to process a work task.

[0061] S2bC. According to the workflow assigned in step S2bB, a ComfyUI container instance is started on each GPU card. When starting, the GPU card used and the workflow carrier content (including the configuration file and the related model path) are specified.

[0062] S2c, submit the type of ComfyUI workflow carrier, metadata information (including access address, supported models, templates, etc.) and other natural language descriptions and other related information to LLM for learning;

[0063] S2d, deploy LLM+Agent on the scheduler node. Agent is used to access LLM. Users only need to call the interface provided by the Agent to submit ComfyUI tasks. A scheduler node can be a host, which is used to assign a task to a suitable, idle ComfyUI instance through certain algorithms or strategies. There is at least one scheduler node. The calling method includes the following steps:

[0064] S2dA and LLM+Agent submit the ComfyUI task description to LLM for analysis;

[0065] S2dB and LLM provide a group of related ComfyUI running instance nodes, such as the access address of the ComfyUI running instance, that is, to classify the ComfyUI instances. Each type of ComfyUI instance can handle a type of AI processing task. For example, some plug-ins encapsulated in some ComfyUI instances support face fusion photo, while other plug-ins encapsulated in some ComfyUI instances are used to process video generation, and other ComfyUI instances may be music generation, etc.; ComfyUI instances of the same type can handle the same tasks, and ComfyUI instances of different types cannot process each other's tasks.

[0066] S2dC and LLM+Agent select an unassigned ComfyUI instance and record it as ComfyUI instance A, then submit the current ComfyUI task to ComfyUI instance A, and mark ComfyUI instance A as assigned;

[0067] S2dD. If ComfyUI instance A returns that it does not support the current ComfyUI task or fails to process, the Agent will recall it, exclude the recalled ComfyUI instance A, and reselect other unassigned ComfyUI instances until a ComfyUI instance node that can normally process the current ComfyUI task is found.

[0068] S2dE. If all ComfyUI instance nodes do not support ComfyUI tasks or the processing fails, the caller task is returned as not supported or the processing fails.

[0069] The Agent will check the task results in real time at intervals: if the task has been completed, it will notify the caller of the task results. Among them, a ComfyUI instance only processes one task at a time (the purpose of this is to simplify scheduling and deployment). When a ComfyUI instance is processing a task (that is, the ComfyUI instance is considered to be assigned to the task, or the task is assigned to the ComfyUI instance), it will no longer process other tasks. Therefore, the Agent must exclude the ComfyUI instance that is processing a task and find an idle ComfyUI instance to process a new task.

[0070] S3. When the user uses the ComfyUI function, he only needs to enter a text description, and then the text description is submitted to the scheduler. LLM+Agent will use the task workflow scheduling system constructed in the above step S2, including the scheduler, to process the ComfyUI task. This method greatly reduces the threshold for users to use ComfyUI and improves convenience. For example, if a user wants to make a wedding photo of himself, he can enter "Help me make a Chinese classical wedding photo with a man on the left and a woman on the right. Our personal photos are https: / / xxxx / p1.jpg and https: / / xxxx / p2.jpg respectively." At this time, LLM will parse out the personal photo model, (Chinese classical + man on the left, woman on the right) wedding photo template, the download address of the two personal photos and other information, and filter out the corresponding ComfyUI workflow running instance such as the access address based on the model and template information. Then, the Agent will use this information as parameters to call the processing interface of the ComfyUI workflow instance to complete the submission of the task. After the task is completed, a download address of the finished wedding photo will be returned to the user, such as: https: / / yyyy / result.jpg.

[0071] In addition, although LLM can recognize some text descriptions, in some cases, it will fail to recognize or fail to recognize part of the information. Therefore, as an enhancement, the present invention adds a labeling function when training LLM (these labels are predefined, and when training LLM, they are built-in and learned into the model so that LLM can quickly understand and recognize them). Labels are used to specify some specific and unambiguous information, such as: @image:https: / / xxxx / p1.jpg specifies an image information and the corresponding image data address; @image-2:https: / / xxxx / p2.jpg is the same as @image, but represents the second image. That is, the label is interpreted as a specific semantics and will not cause ambiguity.

[0072] As a workflow package, ComfyUI supports the design and use of multiple workflows. We encapsulate it into Docker as the atomic unit of workflow inference computing power. At the upper layer, the LLM+Agent mechanism is used to decide which workflows to combine, distribute and call to complete a series of media content production tasks such as image generation, image style conversion, image editing, and image to video conversion, so as to achieve a highly flexible and integrated AIGC system.

[0073] The advantages of the present invention are: it proposes a unitized encapsulation of workflow reasoning nodes, customizes and simplifies the ComfyUI installation and deployment steps, and uses Docker technology to package it into a container image, which is convenient and fast to distribute, and can quickly deploy ComfyUI working nodes: users can quickly start a ComfyUI working node with a minimum environment and minimum configuration through our customized ComfyUI container image, and the working node can be used alone; it realizes an Agent upper-layer task dispatching system that uses a large language model to distribute task decisions. The scheduling algorithm implemented by the large language model can accurately schedule multiple tasks to multiple ComfyUI working nodes, which can maximize the full use of ComfyUI resources and greatly improve the resource utilization of ComfyUI. It can understand user intentions through LLM + Function Calling; it supports the design and expansion of multiple solutions, without the need for algorithm engineers to perform redundant programming. Through the dragging and connection of plug-in nodes by designers, and the low-code design capability of ComfyUI, an AIGC workflow task can be built and put into production.

[0074] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing and scheduling an artificial intelligence image and video processing workflow, characterized in that: Includes steps: S1. Use Docker to package ComfyUI into a container image to simplify the installation and deployment steps of ComfyUI; S2. Use an agent based on a large language model to build a task workflow scheduling method; S3. When the user uses the ComfyUI function, he only needs to enter a text description, which is submitted to the scheduler. The intelligent experience will use the task workflow scheduling system to process the ComfyUI task.

2. The method for constructing and scheduling an artificial intelligence image and video processing workflow according to claim 1, characterized in that: The step S1 comprises: Install the standard Docker or Podman container runtime environment; Download the ComfyUI code package; Create a Dockerfile container image build file; Use the Docker or Podman build tool, the ComfyUI code package, and the Dockerfile container image build file to build the ComfyUI container image; Upload the built ComfyUI container image to the Docker image repository or other third-party container image repository.

3. The method for constructing and scheduling an artificial intelligence image and video processing workflow according to claim 2, characterized in that: The ComfyUI container image only contains the environment of ComfyUI runtime, and does not include the workflow carrier content during specific execution.

4. The method for constructing and scheduling an artificial intelligence image and video processing workflow according to claim 3, characterized in that: The step S2 comprises the steps of: S2a, placing all workflow carrier contents in a shared storage, and then mounting the shared storage to all GPU host nodes that will run ComfyUI instances. One GPU host node is a physical machine with at least one GPU card, each of which is used to run a Docker container, in which the ComfyUI program runs. A physical machine has several GPU cards, which means it runs several ComfyUI instances. S2b, installing a Docker or Podman container runtime environment on multiple GPU host nodes, and then allocating workflows based on the ComfyUI container image; S2c, submitting the type, metadata information and other natural language descriptions of the ComfyUI workflow carrier to the large language model for learning; S2d. On the scheduler node, deploy an agent based on the large language model to access the large language model. One scheduler node is a host, which is used to assign a task to an idle ComfyUI instance. There is at least one scheduler node.

5. The method for constructing and scheduling an artificial intelligence image and video processing workflow according to claim 4, characterized in that: The workflow carrier content is configuration files and related model files; The type and metadata information of the ComfyUI workflow carrier include access address, supported models and templates.

6. The method for constructing and scheduling an artificial intelligence image and video processing workflow according to claim 5, characterized in that: In the step S2b, the workflow based on the ComfyUI container image distribution includes the following steps: Pull the container image produced in step S1 from the Docker image repository or other third-party container image repository, so that the multiple GPU host nodes directly enter the ComfyUI runtime environment; According to the type of the workflow, the graphics processor cards of different graphics processor host nodes are allocated to different types of workflows, and each time a graphics processor card is allocated to a type of workflow, the type of workflow has a runtime instance for processing a work task; According to the assigned workflow, a ComfyUI container instance is started on each graphics processor card, and the graphics processor card and workflow carrier content used are specified during startup.

7. The method for constructing and scheduling an artificial intelligence image and video processing workflow according to claim 6, characterized in that: In the step S2d, the user only needs to call the interface provided by the agent to submit the ComfyUI task.

8. The method for constructing and scheduling an artificial intelligence image and video processing workflow according to claim 7, characterized in that: In the step S2d, the calling includes the following steps: The agent implemented based on the large language model submits the ComfyUI task description to the large language model for analysis; The large language model provides a group of related ComfyUI running instance nodes, that is, the ComfyUI instances are classified, each type of ComfyUI instance processes an artificial intelligence processing task, the ComfyUI instances of the same type can process the same task, and ComfyUI instances of different types cannot process each other's tasks; The agent selects an unassigned ComfyUI instance as ComfyUI instance A, then submits the current ComfyUI task to the ComfyUI instance A, and marks ComfyUI instance A as assigned; If ComfyUI instance A returns that it does not support the current ComfyUI task or fails to process, the intelligent body will recall it, exclude the recalled ComfyUI instance A, and reselect other unassigned ComfyUI instances until a ComfyUI instance node that can normally process the current ComfyUI task is found.

9. The method for constructing and scheduling an artificial intelligence image and video processing workflow according to claim 8, characterized in that: A labeling function is added when training the large language model, and the labels are all predefined, and the labels are used to specify specific and unambiguous information.

Images