System and method for reconstructing workflow at server

Abstract

The invention relates to a system and a method for reconstructing a workflow at a server. The system comprises a native workflow acquisition component which is deployed at a client and is used for acquiring a native workflow to be used by a user based on a request of the user; the native node traversal component is deployed at the client and is used for traversing and analyzing the acquired native nodes in the native workflow; the directed acyclic graph generation component is deployed at a client and abstracted into a directed acyclic graph by taking the obtained native node as a node; the workflow regeneration assembly is deployed at the server side and regenerates a new workflow based on the directed acyclic graph, and the new workflow at least comprises computing nodes, needing large model computing, of the original workflow, so that the nodes of the new workflow are deployed at the server side, and the model file stored at the server side is directly utilized at the server side.

Description

Technical Field

[0001] The present disclosure relates to the field of computer information processing, and in particular to a system and method for reconstructing a workflow on a server side. Background Art

[0002] With the rapid development of artificial intelligence, models that generate images or videos from text are becoming increasingly popular. However, due to insufficient local computing power, these models often run very slowly, resulting in a poor user experience. Therefore, leveraging network computing power and public computing power to quickly run models and obtain results has become a common user need.

[0003] ComfyUI is a commonly used workflow tool that uses a graphical interface to lower the barrier to entry for AI image generation. However, while regular users know they can leverage public computing power to accelerate AI image generation, they often don't know which ComfyUI native nodes can be converted to cloud nodes and deployed in the cloud. Furthermore, some cloud node developers have deployed numerous cloud nodes as plug-ins, allowing users to leverage these cloud nodes to accelerate AI image generation. However, the average user doesn't know the corresponding relationships between these native nodes and certain cloud nodes, making direct application difficult.

[0004] Currently, some nodes based on cloud computing resources have emerged. However, these nodes can only output text or images, tending to encapsulate complete workflows as cloud nodes. This makes them incompatible with the usage habits of native workflow tools (such as ComfyUI). This over-encapsulation approach reduces the node's compatibility with common workflow tools. However, there are reasons why most cloud nodes choose this "over-packaging" approach. The inputs and outputs of certain nodes in common native workflow tools are difficult to effectively "transmit" to the cloud. For example, from the user's perspective, a node that loads a model outputs a "MODEL." However, this is not actually the actual model parameters, but rather a "callable object." A reference to this callable object is transmitted to subsequent nodes, which then call it when performing actual computations. However, this "callable object" is difficult to transmit across the network because no instance of the object exists on the remote server. This approach will change user habits with conventional workflow tools and increase the difficulty of using conventional workflow tools that utilize server-side computing power.

[0005] Therefore, people need a system and method for reconstructing workflows on the server side, which can enable ordinary users to place their main computing power needs in the cloud instead of locally, so as to reduce local computing overhead and not change the user's usage habits, thereby using remote server computing power without feeling, and accelerating the remote deployment of the model and the efficiency of model operation.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0007] In view of this, the present disclosure provides a system or plug-in for reconstructing the workflow on the server side, which can facilitate ordinary users to directly and unconsciously deploy the main computing nodes on the server side without changing the user's habitual usage. In this way, the AI ​​image generation process can be simply realized through the plug-ins of some workflow tools.

[0008] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0009] According to one aspect of the present disclosure, a system for reconstructing workflows on a server side is proposed, including: a native workflow acquisition component, deployed on a client side, which acquires the native workflow to be used by the user based on the user's request; a native node traversal component, deployed on the client side, which traverses and analyzes the native nodes in the acquired native workflow; a directed acyclic graph generation component, deployed on the client side, which abstracts the acquired native nodes into a directed acyclic graph with the acquired native nodes as nodes; and a workflow regeneration component, deployed on the server side, which regenerates a new workflow based on the directed acyclic graph, wherein the new workflow at least includes computing nodes of the native workflow that require large model calculations, so that the nodes of the new workflow are deployed on the server side, so that the model files stored on the server side can be directly utilized on the server side.

[0010] According to the system for reconstructing workflows on the server side disclosed in the present invention, the directed acyclic graph generation component generates a directed acyclic graph through the following steps: scanning all native nodes in the native workflow to identify the computing nodes and data nodes that need to perform tasks in the cloud; and constructing dependencies between native nodes by analyzing the connections between native nodes, and eliminating closed-loop deadlocks by detecting circular dependencies.

[0011] According to the system for reconstructing workflows on a server side disclosed herein, the directed acyclic graph generation component converts the abstracted directed acyclic graph into JSON structured data and transmits it to the workflow regeneration component on the server side.

[0012] According to the system for reconstructing workflows on the server side disclosed in the present invention, the workflow regeneration component includes the following in regenerating a new workflow: replacing computing nodes in a directed acyclic graph that require large model calculations with equivalent cloud nodes; trimming auxiliary nodes that are only used for client display; marking inputs to be uploaded from the client and results to be returned.

[0013] The system for reconstructing a workflow on a server side according to the present disclosure further includes: a scheduling component deployed on the server side, which optimizes the directed acyclic graph to insert acceleration nodes.

[0014] According to another aspect of the present disclosure, a method for reconstructing a workflow on a server side is provided, comprising: obtaining a native workflow to be used by a user based on a user's request through a native workflow acquisition component deployed on a client side; traversing and analyzing native nodes in the obtained native workflow through a native node traversal component deployed on a client side; abstracting a directed acyclic graph with the obtained native nodes as nodes through a directed acyclic graph generation component deployed on a client side; and regenerating a new workflow based on the directed acyclic graph through a workflow regeneration component deployed on a server side, wherein the new workflow includes at least computing nodes of the native workflow that require large model calculations, so that the nodes of the new workflow are deployed on the server side so that the model files stored on the server side can be directly utilized on the server side.

[0015] According to the method for reconstructing a workflow on the server side disclosed in the present invention, the step of generating a directed acyclic graph includes: scanning all native nodes in the native workflow to identify the computing nodes and data nodes that need to perform tasks in the cloud; and constructing dependencies between native nodes by analyzing the connections between native nodes, and eliminating closed-loop deadlocks by detecting circular dependencies.

[0016] According to the method for reconstructing a workflow on a server side disclosed herein, the method further includes: converting the abstracted directed acyclic graph into JSON structured data and transmitting the data to a workflow regeneration component on the server side.

[0017] ‌ According to the method of reconstructing a workflow on the server side disclosed in the present invention, the regeneration of a new workflow includes: ‌ replacing the computing nodes in the directed acyclic graph that require large model calculations with equivalent cloud nodes; ‌ trimming the auxiliary nodes that are only used for client display; ‌ marking the inputs to be uploaded from the client and the results to be returned.

[0018] The method for reconstructing a workflow on a server side according to the present disclosure further includes optimizing the directed acyclic graph by a scheduling component deployed on the server side so as to insert an acceleration node.

[0019] According to the disclosed system and method for reconstructing workflows on the server side, since large file models stored on the server side are preferentially adapted during the reconstruction process, users do not need to download or store large model files (such as FLUX.1, SD 3.5, Kolors, and other basic models) on the local client. All models required for the large model computing nodes of the reconstructed workflow are hosted in the server-side cloud environment, which can reduce local storage pressure and is especially suitable for users of low-configuration devices. In addition, since client nodes are established at the same time during the reconstruction process, so that the server side receives lightweight data input and outputs result data to the client through this client node, it is possible to directly transmit small or lightweight data (including text prompts, latents tensors, images tensors, masks tensors, etc.) to the server side through the network in real time, ensuring the real-time and low-latency nature of the interaction, thereby eliminating the need for users to change their usage habits.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other objects, features and advantages of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the accompanying drawings. The drawings described below are only some embodiments of the present disclosure, and it is obvious to those skilled in the art that other drawings can be derived from these drawings without inventive effort.

[0022] Figure 1 It is a block diagram showing a first embodiment of a system for reconstructing a workflow on a server side according to the present disclosure, according to an exemplary embodiment.

[0023] Figure 2 It is a block diagram showing a second embodiment of a system for reconstructing a workflow on a server side according to the present disclosure, according to an exemplary embodiment.

[0024] Figure 3 The figure is a flowchart of a method for reconstructing a workflow on a server side according to the present disclosure, shown according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0026] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present disclosure.

[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0029] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Thus, the first computing device discussed below can be referred to as the second computing device without departing from the teachings of the present disclosure. As used herein, the term "and / or" includes any one and all combinations of one or more of the associated listed items.

[0030] Those skilled in the art will understand that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present disclosure, and therefore cannot be used to limit the scope of protection of the present disclosure.

[0031] Figure 1 FIG. 1 is a block diagram of an embodiment of a system 100 for reconstructing a workflow on a server side according to the present disclosure, according to an exemplary embodiment. Figure 1 As shown, the workflow conversion system 100 includes a native workflow acquisition component 110, a native node traversal component 120, a directed acyclic graph generation component 130, and a workflow regeneration component 140. Generally speaking, the system 100 for reconstructing workflows on the server side is similar to installing a workflow tool such as ComfyUI with a reconstruction system or a reconstruction plug-in, thereby functionally reconstructing native workflows on the server side. This allows users to seamlessly "cloudify" computing resources while maintaining the functionality of the workflow.

[0032] Specifically, the system 100 for reconstructing workflows on the server side includes: a client and a server. The native workflow acquisition component 110 deployed on the client side obtains the native workflow that the user wants to use based on the user's request. Subsequently, the native node traversal component 120 deployed on the client side traverses and analyzes the native nodes in the obtained native workflow. The directed acyclic graph generation component 130 deployed on the client side abstracts the obtained native nodes into a directed acyclic graph using the obtained native nodes as nodes. Specifically, the directed acyclic graph generation component 130 first scans all native nodes in the native workflow to identify the computing nodes and data nodes that need to perform tasks in the cloud, and then constructs the dependency relationship between the native nodes by analyzing the connections between the native nodes, and eliminates closed-loop deadlocks by detecting circular dependencies, and finally forms a directed acyclic graph.

[0033] The DAG generation component 130 converts the abstracted DAG into JSON structured data and transmits the data to the workflow regeneration component 140 on the server side.

[0034] The workflow regeneration component 140 deployed on the server regenerates a new workflow based on the directed acyclic graph. This new workflow contains at least the compute nodes of the original workflow that require large-scale model computation. This allows the nodes of the new workflow to be deployed on the server, allowing the server to directly utilize the model files stored on the server. In the regenerated new workflow, the workflow regeneration component 140 replaces the compute nodes in the directed acyclic graph that require large-scale model computation with equivalent cloud nodes; trims auxiliary nodes that are only used for client display; and marks inputs to be uploaded from the client and results to be returned.

[0035] Figure 2 1 is a block diagram of a second embodiment of a system for reconstructing a workflow on a server side according to the present disclosure, according to an exemplary embodiment. Figure 1 The difference is that it also includes a scheduling component 150. Therefore, the scheduling component 150 deployed on the server optimizes the directed acyclic graph to insert the acceleration node.

[0036] Figure 3 FIG is a flowchart of a method for reconstructing a workflow on a server side according to an exemplary embodiment of the present disclosure. Figure 3As shown, first, in step S310, the native workflow to be used by the user is obtained based on the user's request through the native workflow acquisition group 110 deployed on the client. Secondly, in step S320, the native nodes in the obtained native workflow are traversed and analyzed through the native node traversal component deployed on the client. Subsequently, in step S330, the directed acyclic graph generation component deployed on the client is used to abstract the obtained native nodes into a directed acyclic graph with the obtained native nodes as nodes. Specifically, the directed acyclic graph generation component 130 first scans all native nodes in the native workflow to identify the computing nodes and data nodes that need to perform tasks in the cloud, and then constructs the dependency relationship between the native nodes by analyzing the connections between the native nodes, and eliminates closed-loop deadlocks by detecting circular dependencies, and finally forms a directed acyclic graph. The directed acyclic graph generation component 130 converts the abstracted directed acyclic graph into JSON structured data and transmits it to the workflow regeneration component 140 on the server.

[0037] Finally, in step S440, the workflow regeneration component 140 deployed on the server regenerates a new workflow based on the directed acyclic graph. This new workflow contains at least the computational nodes of the original workflow that require large-scale model computation. This allows the nodes of the new workflow to be deployed on the server, allowing the server to directly utilize the model files stored on the server. This regeneration of the new workflow includes: replacing the computational nodes in the directed acyclic graph that require large-scale model computation with equivalent cloud nodes; trimming auxiliary nodes used only for client display; and marking inputs to be uploaded from the client and results to be returned.

[0038] The method for reconstructing a workflow on a server side according to the present disclosure further includes optimizing the directed acyclic graph by a scheduling component deployed on the server side so as to insert an acceleration node.

[0039] Taking a ComfyUI workflow for generating images from text as an example, a native workflow contains seven native ComfyUI nodes. The entire workflow includes: a load checkpoint node (LOADCHECKPOINT), which is used to load the pre-trained model weights and configuration to ensure that the model has initialized parameters when generating images; and two text description encoding nodes (CLIPText Encode), which are used to encode text prompts into vectors for alignment with image features. It uses the CLIP (Contrastive Language-Image Pre-training) model to convert text into the model's input format; the Empty Latent node is used to generate an empty latent space vector, which is often used as the initial latent representation for generating images; the Key Sampling node (KSampler) is used to sample from the latent space and generate multiple latent vectors, which are used to generate diverse images; the Variational Autoencoder (VAE) node is used to decode the latent space vector into an image. The VAE encodes the image into a latent space representation through an encoder and then restores it to an image through a decoder; and the Image Save node (Save Image) is used to save the generated image to a specified path or format for subsequent processing or display. Together, these nodes form a complete workflow for generating images from text, enabling the transition from text description to image generation.

[0040] In layman's terms, workflows are abstracted as DAG structure transmission: the workflows built by users in workflow tools (such as document-to-graph or graph-to-graph processes) will be abstracted by the plug-in into a directed acyclic graph (DAG) structure. The DAG defines the dependencies and execution order between task nodes, and is sent to the server as a whole over the network for efficient parsing and processing. The server rebuilds and executes the workflow: After receiving the DAG, the server will rebuild the complete workflow instance and call the self-developed scheduler for execution. The scheduler optimizes resource allocation (such as GPU computing parallelization), handles model loading, data flow and task coordination, and finally generates results (such as images) and returns them to the user client.

[0041] In summary, according to the workflow conversion system and method disclosed in the present invention, by determining the independence of native nodes, migratable native nodes are displayed, and through the cloud node library provided by the system for various native nodes that can be converted into cloud nodes, selectable cloud nodes are provided to users, and the system automatically provides users with fusible guidance based on the adjacent series relationship between native nodes, thereby providing ordinary users with a user-friendly workflow conversion interface. As a result, it is greatly facilitated for users to use network or cloud computing power based on commonly used workflow tools to improve the speed and efficiency of AI generation processing, thereby bringing practical convenience of smart life to users.

[0042] Those skilled in the art will appreciate that the modules described above can be distributed in the device according to the description of the embodiment, or can be modified accordingly to be used in one or more devices that are different from the embodiment. The modules of the above embodiment can be combined into one module or further divided into multiple submodules.

[0043] Through the description of the above embodiments, it will be readily understood by those skilled in the art that the example embodiments described herein can be implemented via software or via a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard drive) or on a network and includes several instructions for enabling a computing device (such as a personal computer, server, mobile terminal, or network device) to execute the methods according to the embodiments of the present disclosure.

[0044] While the exemplary embodiments of the present disclosure have been specifically illustrated and described above, it should be understood that the present disclosure is not limited to the detailed structures, configurations, or implementations described herein; rather, the present disclosure is intended to encompass various modifications and equivalent configurations within the spirit and scope of the appended claims.

Claims

1. A system for reconstructing a workflow on a server side, comprising: The native workflow acquisition component is deployed on the client and obtains the native workflow that the user wants to use based on the user's request; The native node traversal component is deployed on the client and traverses and analyzes the native nodes in the acquired native workflow; The directed acyclic graph generation component is deployed on the client and uses the obtained native nodes as nodes to abstract them into a directed acyclic graph. as well as The workflow regeneration component is deployed on the server side and regenerates a new workflow based on a directed acyclic graph. The new workflow at least includes the computing nodes of the original workflow that require large model calculations, so that the nodes of the new workflow are deployed on the server side so that the model files stored on the server side can be directly used on the server side.

2. The system for reconstructing a workflow on a server side according to claim 1, wherein the directed acyclic graph generation component generates a directed acyclic graph by the following steps: Scan all native nodes in the native workflow to identify compute nodes and data nodes that need to execute tasks in the cloud; and By analyzing the connections between native nodes, we build dependencies between native nodes and eliminate closed-loop deadlocks by detecting circular dependencies.

3. The system for reconstructing workflows on a server side according to claim 1, wherein the directed acyclic graph generation component converts the abstracted directed acyclic graph into JSON structured data and transmits the data to the workflow regeneration component on the server side.

4. The system for reconstructing a workflow on a server side according to claim 3, wherein the workflow regeneration component comprises: Replace the compute nodes in the directed acyclic graph that require large model computations with equivalent cloud nodes; ‌ Clip auxiliary nodes for client display only; Marks the input to be uploaded from the client and the result to be sent back.

5. The system for reconstructing a workflow on a server side according to claim 4, further comprising: The scheduling component is deployed on the server side and optimizes the directed acyclic graph to facilitate the insertion of acceleration nodes.

6. A method for reconstructing a workflow on a server, comprising: The native workflow acquisition component deployed on the client obtains the native workflow that the user wants to use based on the user's request; Traverse and analyze the native nodes in the obtained native workflow through the native node traversal component deployed on the client; By deploying a directed acyclic graph generation component on the client, the obtained native nodes are used as nodes and abstracted into a directed acyclic graph; as well as By deploying a workflow regeneration component on the server side, a new workflow is regenerated based on a directed acyclic graph. The new workflow at least includes the computing nodes of the original workflow that require large model calculations, so that the nodes of the new workflow are deployed on the server side so that the model files stored on the server side can be directly used on the server side.

7. The method for reconstructing a workflow on a server side according to claim 6, wherein the step of generating a directed acyclic graph comprises: Scan all native nodes in the native workflow to identify the compute nodes and data nodes that need to execute tasks in the cloud; as well as By analyzing the connections between native nodes, we build dependencies between native nodes and eliminate closed-loop deadlocks by detecting circular dependencies.

8. The method for reconstructing a workflow on a server side according to claim 6, further comprising: The abstracted directed acyclic graph is converted into JSON structured data and transmitted to the workflow regeneration component on the server.

9. The method for reconstructing a workflow on a server side according to claim 8, wherein regenerating a new workflow comprises: Replace the compute nodes in the directed acyclic graph that require large model computations with equivalent cloud nodes; ‌ Clip auxiliary nodes for client display only; Marks the input to be uploaded from the client and the result to be sent back.

10. The method for reconstructing a workflow on a server side according to claim 9, further comprising optimizing the directed acyclic graph by a scheduling component deployed on the server side so as to insert an acceleration node.

Images