Artificial intelligence generation system and deployment method thereof

Abstract

The invention relates to an artificial intelligence generation system and a deployment method thereof. The system comprises a client component used for receiving generation intention information input by a user and used for realizing workflow operation, and sending a generation request to a cloud; the intermediate node assembly comprises a plurality of intermediate nodes, a part or all of the intermediate nodes form a specific workflow for operating one or more models based on generation intention information of the user, and all and part of the intermediate nodes can be selected based on specific requirements of the user and are actively adjusted by the user, so that the user experience is improved. Therefore, an exclusive intermediate node meeting the requirements of the user is formed; and the back-end reasoning component is used as a constituent node of the specific workflow, is deployed at the cloud, receives data from the intermediate node constituting the specific workflow, performs reasoning, obtains a generation result to be formed by generating intention information, and sends the generation result to the client component.

Description

Technical Field

[0001] The present disclosure relates to an artificial intelligence generation system and a deployment method thereof. More specifically, the present disclosure relates to a method for making the deployment of an artificial intelligence generation system fine-grained. Background Art

[0002] Currently, cloud platform companies that provide generation graph APIs on the market mainly have two forms. One is together.ai, replicate, siliconflow, which provide APIs at the granularity of the model itself. For example, the model related to the together.ai image API. The other is cloud platform companies that provide services in a workflow hosting manner, that is, the user uploads a comfyui workflow by themselves, or the user selects an existing workflow on the cloud platform, and end-to-end deployment is automatically completed. For example, thinkdiffusion.com, https: / / fal.ai / workflows. Whether it is the former or the latter, they all adopt the way of workflows (pipelines). The so-called "workflow" is an interface for multiple links to cooperate to complete the generation of images and videos, and it is not as highly standardized as the existing LLM (when both the input and output are text). For example, in the most basic SDXL model, there are 3 models and 7 nodes. The model service provided by Maas is to provide users with a workflow composed of 7 nodes. In contrast, the LLM service provided by Maas now is a single node in comfyUI, with 1 input and 1 output.

[0003] Therefore, in terms of generating graphs, this workflow is still in the process of rapid evolution, and therefore it is also very non-standardized. In addition, existing workflow development tools, such as diffusers and ComfyUI, are themselves rapidly iterating, and new workflows (pipelines) are also increasing, so the commonly used ComfyUI does not have a version management function for the time being. In particular, the models used for workflows cannot be standardized. For example, major Maas companies have focused on the SD (Stable Diffusion) series of models. However, in actual industrial scenarios, many C-station training models (Maggie series, mmmmix series) are used. This results in the model not being standardized, making it impossible for users to obtain the personalized results they want when using it. Moreover, in order to obtain better results, users need to use some plug-ins when using these artificial intelligence platforms to obtain better results, but these plug-ins are also not standardized. As a result, industry insiders have come to realize that "ComfyUI without plug-ins and ComfyUI with plug-ins installed are simply two software." More importantly, ComfyUI itself does not have a very comprehensive workflow capability. Workflows (or nodes) such as ipadaptor, InstantID, super-resolution, and cutout are all contributed by community members, not ComfyUI, so there is no standardized plug-in. The generation graph model is constantly innovating, and this kind of personal porting has never stopped. This situation leads to everyone providing APIs in units of models (workflows), which makes it difficult for users to use them in production environments. Therefore, the models and workflows that users want are non-standard, and it is difficult to find directly corresponding APIs on Maas. Therefore, various AIGC platforms (such as Toast and Unbounded AI) now almost do everything from workflow to terminal interface by themselves, resulting in a small chance of using Maas API. For example, the ComfyUI workflow hosting platform also hosts the workflow as a whole, that is, "either none or all", which still requires users to use cloud computing resources at the granularity of workflow. Although the user's workflow can be run directly on the cloud by "pre-installing common plug-ins", it is not realistic to implement the specificization of user tasks by "pre-installing common plug-ins" because plug-ins change quickly.

[0004] Therefore, people need an artificial intelligence image generation system or video generation system that is more flexible to use and can meet the specific requirements of users, eliminating the defects of the rigid workflow and model provided by the platform. Summary of the invention

[0005] An object of the present invention is to solve at least the above problems. Specifically, the present disclosure provides an artificial intelligence generation system, including: a client component, configured to receive generation intention information input by a user for implementing a workflow operation and send a generation request to the cloud; an intermediate node component, including a plurality of intermediate nodes, some or all of the intermediate nodes form a specific workflow for running one or more models based on the user's generation intention information, wherein all or some of the intermediate nodes can be selected based on the user's specific needs and actively adjusted by the user to form exclusive intermediate nodes that meet the user's own needs; a backend inference component, deployed in the cloud as a constituent node of the specific workflow, receiving data from the intermediate nodes constituting the specific workflow, performing inference, obtaining a generation result to be formed by the generation intention information, and sending it to the client component.

[0006] According to the artificial intelligence generation system of the present disclosure, wherein the exclusive intermediate node includes one or more child nodes, wherein the child nodes for performing operations are deployed in the cloud, and the child nodes for directly and actively adjusting are deployed locally on the client.

[0007] According to the artificial intelligence generation system of the present disclosure, wherein the model is a text generation model, a graph generation model or a video generation model.

[0008] According to the artificial intelligence generation system of the present disclosure, wherein the client component receives the generation result from the backend inference component, performs extraction and parsing, and presents the generation result to the user.

[0009] According to the artificial intelligence generation system of the present disclosure, it further includes: an intermediate bridging component, configured to convert the request sent by the client component into a protocol format required by the backend inference component based on a predetermined specific protocol and send it to the backend inference component for the backend inference component to perform inference calculations, and convert the generation result of the backend inference component into a protocol format required by the client component based on a predetermined specific protocol and return it to the client component.

[0010] According to another aspect of the present disclosure, there is provided a deployment method for an artificial intelligence generation system, including: deploying a client component locally on a user side for receiving generation intention information input by the user for implementing workflow operation and sending a generation request to the cloud; deploying an intermediate node component in the cloud, the intermediate node component including a plurality of intermediate nodes, and some or all of the intermediate nodes forming a specific workflow for running one or more models based on the user's generation intention information, wherein all or some of the intermediate nodes can be selected based on the user's specific requirements and actively adjusted by the user so as to form exclusive intermediate nodes that meet the user's own requirements; deploying a backend inference component in the cloud, the backend inference component being a constituent node of the specific workflow, receiving data from the intermediate nodes constituting the specific workflow, performing inference, obtaining a generation result to be formed by the generation intention information, and sending it to the client component.

[0011] According to the deployment method for an artificial intelligence generation system of the present disclosure, wherein the exclusive intermediate node includes one or more child nodes, and the child nodes for performing operations are deployed in the cloud, while the child nodes for directly and actively making adjustments are deployed locally on the client side.

[0012] According to the deployment method for an artificial intelligence generation system of the present disclosure, wherein the model is a text generation model, a graph generation model or a video generation model.

[0013] According to the deployment method for an artificial intelligence generation system of the present disclosure, wherein the client component receives the generation result from the backend inference component, performs extraction and parsing, and presents the generation result to the user.

[0014] According to the deployment method for an artificial intelligence generation system of the present disclosure, it further includes: an intermediate bridging component deployed in the cloud, which is used to convert the request sent by the client component into a protocol format required by the backend inference component based on a predetermined specific protocol and send it to the backend inference component so that the backend inference component can perform inference calculations, and convert the generation result of the backend inference component into a protocol format required by the client component based on the predetermined specific protocol and return it to the client component.

[0015] With the artificial intelligence generation system and its deployment method according to the present disclosure, the existing method of overall deployment in the cloud at the workflow granularity is refined, reducing the granularity of the APIs provided by the existing Maas, so that APIs are provided on a node-by-node basis. On the one hand, this enables users to deploy computing tasks for generating images or videos in the cloud. In this way, ordinary users do not need to download the required models locally, nor do they need to consider the deployment of expensive GPU devices locally, thus saving the terminal and eliminating the local hardware threshold and the time for environment configuration. On the other hand, since the present disclosure provides Maas services on a node-by-node basis that constitutes the workflow, when a user executes a computing task for generating an image or a video on the cloud that relies on the Maas service, the user can replace only the individual nodes required for the specific user needs, which can also save the user's time for environment configuration. For example, when a user wants to add a super-resolution effect when generating an image, instead of downloading multiple super-resolution models as before, now it can be replaced with a cloud node. For the applicant, through the ComfyUI community, it is easier to reach the real needs and high-frequency feedback of users.

[0016] Other advantages, objectives, and features of the present invention will be partially reflected by the following description and partially understood by those skilled in the art through the research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a schematic diagram of the principle of the artificial intelligence generation system according to the present disclosure.

[0018] Figure 2 Shown is a flowchart of the deployment method in the artificial intelligence generation system according to the present disclosure.

[0019] Figure 3 Shown is a flowchart of the second embodiment of the deployment method in the artificial intelligence generation system according to the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The following further elaborates on the present invention in detail in conjunction with embodiments and drawings, so that those skilled in the art can implement it with reference to the text of the specification.

[0021] Here, the exemplary embodiments will be described in detail, and their examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0022] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a", "the", and "said" used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0023] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, one of two possible objects may be referred to as the first intermediate node or the second intermediate node hereinafter. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0024] To enable those skilled in the art to better understand this disclosure, the following further describes this disclosure in detail in conjunction with the accompanying drawings and specific embodiments.

[0025] Figure 1 Shown is a schematic diagram of the principle of an artificial intelligence generation system according to this disclosure. As Figure 1 shown, the artificial intelligence generation system is mainly applied in the field of image generation or video generation. By inputting generation intention information by the user, the artificial intelligence generation system configures a model combination based on various existing models and outputs a generation result that meets the user's needs. As mentioned above, the environment configuration of the artificial intelligence generation system requires a large number of computing devices, so the actual computing process is deployed in the cloud. The configuration environment required for the computing process requires a large number of GPU devices, which is a cost that is difficult for ordinary users to bear. However, for a Maas service company or platform that can configure a large number of GPU devices, requiring users to use cloud computing resources at the granularity of workflows currently cannot fully provide the personalized workflows required by users. For this reason, the applicant provides as Figure 1The artificial intelligence generation system 100 shown includes a client component 110, an intermediate node component 120, and a backend inference component 130. The client component 110 is deployed locally by the user and is used to receive the generation intention information input by the user for implementing the workflow operation, and send a generation request to the cloud or the Maas service platform. The generation request contains the user's generation intention information. The Maas service platform can configure a corresponding generation model based on the generation request, perform operations to obtain a generation result. Specifically, the intermediate node component 120 includes a plurality of intermediate nodes 120-1, 120-2, 120-3... 120-N. Some or all of the intermediate nodes form a specific workflow for running one or more models based on the user's generation intention information. All or some of the intermediate nodes can be selected based on the user's specific needs and actively adjusted by the user to form a dedicated intermediate node 120-X that meets the user's own needs. In this way, in the intermediate node component 120, instead of deploying the entire workflow in the cloud, the models that make up various workflows are deployed in the cloud as nodes, so that the nodes that make up the workflow are combined into the required workflow when receiving a user request, and at the same time, the nodes that make up the required workflow are in a state where they can be replaced or selected, thus enabling the user to save the local hardware threshold. At the same time, since it is deployed in the cloud at the granularity of nodes, when the user issues different requirements, only individual ones can be replaced, thus saving the user's time for environment configuration. For example, when the user wants to add the effect of super-resolution, under the existing technical conditions, the user originally had to download multiple super-resolution models, but in the case of adopting the technology of this application, only the cloud node needs to be replaced, that is, a dedicated intermediate node that meets the user's own needs is formed. The backend inference component 130 deployed in the cloud, as a constituent node of the specific workflow, receives data from the intermediate nodes that make up the specific workflow, performs inference, obtains the generation result to be formed by the generation intention information, and sends it to the client component 110. The specific inference process of the backend inference component 130 is carried out in a conventional inference manner, so it will not be elaborated here.

[0026] The client component 110 usually exists in the form of a plugin and can be fully compatible with the existing workflows of ComfyUI. ComfyUI is a node-based user interface where you can build a workflow for generating images by connecting different nodes. The ComfyUI workflow is already in common use and will not be elaborated here. Adapted to the ComfyUI user interface, the client component 110 sends a request to the server (cloud), and after the cloud computing ends and returns the result, extracts and parses the result of the cloud computing and outputs it for presentation to the user.

[0027] As described above, the dedicated intermediate node 120-X includes one or more child nodes (not shown), where the child nodes for performing operations are deployed in the cloud, and the child nodes for directly and actively making adjustments are deployed locally on the client. Optionally, the child nodes for directly and actively making adjustments can also be deployed in the cloud.

[0028] Furthermore, the model constituting each node can be a text generation model, a graph generation model, or a video generation model.

[0029] As Figure 1 shown, the artificial intelligence generation system of the present disclosure further includes an intermediate bridging component 140, which is used to convert the request sent by the client component into the protocol format required by the backend inference component based on a predetermined specific protocol, and send it to the backend inference component 130, so that the backend inference component 130 performs inference calculations, and converts the generation result of the backend inference component into the protocol format required by the client component based on a predetermined specific protocol and returns it to the client component 110. The user can determine the specific protocol type according to their own needs.

[0030] Figure 2 Shown is a flowchart of a deployment method for an artificial intelligence generation system according to the present disclosure. As Figure 2 shown, first, at step S210, the client component 110 is deployed locally on the user to receive the generation intention information input by the user for implementing the workflow operation, and send a generation request to the cloud. Then, at step S220, the intermediate node component 120 is deployed or configured in the cloud. The intermediate node component includes a plurality of intermediate nodes 120-1, 120-2, 120-3... 120-N. Some or all of the intermediate nodes form a specific workflow for running one or more models based on the user's generation intention information. All or some of the intermediate nodes can be selected based on the user's specific needs and actively adjusted by the user to form a dedicated intermediate node 120-X that meets the user's own needs. Finally, at step S230, the backend inference component 130 is deployed in the cloud. The backend inference component 130, as a constituent node of the specific workflow, receives data from the intermediate nodes constituting the specific workflow, performs inference, obtains the generation result to be formed by the generation intention information, and sends it to the client component 110.

[0031] The exclusive intermediate node 120-X includes one or more child nodes, where the child nodes for performing operations are deployed in the cloud, while the child nodes for directly and actively making adjustments are deployed locally on the client side. The exclusive intermediate node 120-X is adjusted by the user to form a dedicated model that meets the user's needs, thus meeting the flexible needs of the user. It is precisely by adopting this fine-grained node deployment method that the rigidity of the current artificial intelligence generation system deployed in a workflow manner is eliminated, providing users with more flexible options. As described above, the model is a text generation model, a graph generation model, or a video generation model. And the client component receives the generation result from the backend inference component, performs extraction and parsing, and presents the generation result to the user.

[0032] Figure 3 Shown is a flowchart of a second embodiment of a deployment method for an artificial intelligence generation system according to the present disclosure. In addition to being the same as the Figure 3 steps shown, it further includes, at step S240, deploying an intermediate bridging component 140 in the cloud. The intermediate bridging component is used to convert the request sent by the client component into the protocol format required by the backend inference component based on a predetermined specific protocol, and send it to the backend inference component so that the backend inference component can perform inference calculations, and convert the generation result of the backend inference component into the protocol format required by the client component based on a predetermined specific protocol and return it to the client component.

[0033] Through the artificial intelligence generation system and its deployment method according to the present disclosure, the existing method of overall deployment in the cloud in the form of a workflow is fine-grainedly deployed, reducing the granularity of the API provided by the existing Maas, so that the API service is provided in units of nodes. On the one hand, this enables users to deploy the computing tasks of generating graphs or videos in the cloud. In this way, ordinary users do not need to download the required model to the local, nor do they need to consider the deployment of expensive GPU devices locally, thus saving the local side and eliminating the local hardware threshold and the time for environment configuration. On the other hand, since the present disclosure provides Maas services in units of the nodes that make up the workflow, when users rely on the cloud providing Maas services to execute the computing tasks of generating graphs or videos, they can replace only the individual nodes required by the user for specific user needs, which can also save the time for users to configure the environment. For example, if a user wants to add a super-resolution effect when generating a graph, instead of downloading multiple super-resolution models as before, now it can be replaced with a cloud node. For the applicant, through the ComfyUI community, it is easier to reach the real needs and high-frequency feedback of users.

[0034] The basic principles of the present disclosure have been described above in connection with specific embodiments. However, it should be noted that for those of ordinary skill in the art, all or any steps or components of the method and apparatus of the present disclosure can be implemented in any computing device (including a processor, a storage medium, etc.) or a network of computing devices in the form of hardware, firmware, software, or a combination thereof, which can be achieved by those of ordinary skill in the art using their basic programming skills after reading the description of the present disclosure.

[0035] Therefore, the object of the present disclosure can also be achieved by running a program or a set of programs on any computing device. The computing device can be a well-known general-purpose device. Therefore, the object of the present disclosure can also be achieved only by providing a program product containing program code for implementing the method or apparatus. That is to say, such a program product also constitutes the present disclosure, and a storage medium storing such a program product also constitutes the present disclosure. Obviously, the storage medium can be any well-known storage medium or any storage medium developed in the future.

[0036] It should also be noted that in the apparatus and method of the present disclosure, obviously, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present disclosure. And the steps of performing the above series of processes can naturally be executed in chronological order according to the described order, but it is not necessary to be executed in chronological order. Some steps can be executed in parallel or independently of each other.

[0037] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present disclosure should be included within the protection scope of the present disclosure.

Claims

1. An artificial intelligence generation system, comprising: The client component is used to receive the generation intention information input by the user for realizing the workflow operation, and send the generation request to the cloud; An intermediate node component includes a plurality of intermediate nodes, some or all of which form a specific workflow for running one or more models based on the user's generation intention information, wherein all or some of the intermediate nodes can be selected based on the user's specific needs and can be actively adjusted by the user to form an exclusive intermediate node that meets the user's own needs; The backend reasoning component, as a constituent node of a specific workflow, is deployed in the cloud, receives data from the intermediate nodes that constitute the specific workflow, performs reasoning, obtains the generation results to be generated by the generation intention information, and sends them to the client component.

2. The artificial intelligence generation system according to claim 1, wherein: The dedicated intermediate node includes one or more sub-nodes, wherein the sub-nodes for executing calculations are deployed in the cloud, and the sub-nodes for directly and actively making adjustments are deployed locally on the client.

3. The artificial intelligence generation system according to claim 1, wherein: The model is a text generation model, a graph generation model or a video generation model.

4. The artificial intelligence generation system according to claim 1, wherein: The client component receives the generated results from the backend reasoning component, performs extraction and parsing, and presents the generated results to the user.

5. The artificial intelligence generation system according to claim 1, further comprising: The intermediate bridging component is used to convert the request issued by the client component into the protocol format required by the backend reasoning component based on a predetermined specific protocol, and send it to the backend reasoning component so that the backend reasoning component can perform reasoning calculations, and convert the generated results of the backend reasoning component into the protocol format required by the client component based on the predetermined specific protocol and return it to the client component.

6. A deployment method for an artificial intelligence generation system, comprising: The client component is deployed locally on the user's computer to receive the generation intention information input by the user for implementing the workflow operation and send a generation request to the cloud. Deploy an intermediate node component in the cloud, wherein the intermediate node component includes a plurality of intermediate nodes, and some or all of the intermediate nodes form a specific workflow for running one or more models based on the user's generation intention information, wherein all or some of the intermediate nodes can be selected based on the user's specific needs and can be actively adjusted by the user to form an exclusive intermediate node that meets the user's own needs; A backend reasoning component is deployed in the cloud. The backend reasoning component serves as a constituent node of a specific workflow, receives data from intermediate nodes constituting the specific workflow, performs reasoning, obtains a generation result to be formed by generating intention information, and sends the result to the client component.

7. The deployment method for an artificial intelligence generation system according to claim 6, wherein: The dedicated intermediate node includes one or more sub-nodes, wherein the sub-nodes for executing calculations are deployed in the cloud, and the sub-nodes for directly and actively making adjustments are deployed locally on the client.

8. The deployment method for an artificial intelligence generation system according to claim 6, wherein: The model is a text generation model, a graph generation model or a video generation model.

9. The deployment method for an artificial intelligence generation system according to claim 6, wherein: The client component receives the generated results from the backend reasoning component, performs extraction and parsing, and presents the generated results to the user.

10. The deployment method for an artificial intelligence generation system according to claim 6, further comprising: An intermediate bridging component deployed in the cloud is used to convert the request issued by the client component into the protocol format required by the backend reasoning component based on a predetermined specific protocol, and send it to the backend reasoning component so that the backend reasoning component can perform reasoning calculations, and convert the generated results of the backend reasoning component into the protocol format required by the client component based on the predetermined specific protocol and return it to the client component.

Images