Intelligent workflow generation method and device, equipment and storage medium

Abstract

The invention discloses an intelligent workflow generation method and device, equipment and a storage medium, and the method comprises the steps: obtaining multi-modal demand information inputted by a user, carrying out the semantic understanding of the multi-modal demand information, and obtaining a semantic tag set and a workflow node link; according to the semantic tag set and the workflow node link, screening a basic template matched with the multi-modal demand information from a plurality of workflow generation templates, and expanding nodes in the basic template to obtain a target template; and acquiring hardware resource information of the user equipment, adjusting the node configuration information in the target template, and generating a target workflow file according to the adjusted target template. According to the technical scheme provided by the embodiment of the invention, the technical difficulty of a workflow generation scheme can be reduced, the generation efficiency of the workflow and the quality of a workflow generation result are improved, the problem that the template cannot normally operate due to the fact that the hardware of the user equipment is not supported or the plug-in is missing is avoided, and the workflow can smoothly fall to the ground.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for generating intelligent workflows. Background Technology

[0002] Currently, in the field of Artificial Intelligence Generated Content (AIGC), workflow generation solutions are a technology proposed to accelerate application deployment and automate the process from data preparation to application workflow generation.

[0003] In existing technologies, workflow generation solutions can be divided into two categories: one is where users manually break down requirements into combinations of tool nodes and rely on these combinations to build workflows; the other is where user-required workflows are built using fixed workflow generation templates.

[0004] However, the first type of solution lacks semantic understanding capabilities. Non-technical users are prone to selecting the wrong node due to unfamiliarity with the node functions, and the deviation can only be discovered after the entire workflow is executed. Manually adjusting the node combination requires a lot of time and human resources. The second type of solution usually only matches a fixed template according to the user's needs (video or image), completely ignoring the dependencies of different templates on plugins and resource constraints, which causes the template to fail to function properly due to the user's device hardware not being supported or the lack of plugins. Summary of the Invention

[0005] This invention provides an intelligent workflow generation method, apparatus, device, and storage medium, which can reduce the technical difficulty of workflow generation solutions, improve workflow generation efficiency and the quality of workflow generation results, avoid problems such as templates failing to run properly due to user device hardware incompatibility or missing plugins, ensure smooth workflow implementation, and realize the intelligentization of workflow generation solutions.

[0006] According to one aspect of the present invention, an intelligent workflow generation method is provided, comprising:

[0007] Obtain multimodal requirement information input by the user, perform semantic understanding on the multimodal requirement information, and obtain a set of semantic tags and workflow node links;

[0008] Based on the semantic tag set and workflow node links, a basic template matching the multimodal requirement information is selected from multiple pre-built workflow generation templates, and the nodes in the basic template are expanded to obtain the target template; each workflow generation template pre-records node configuration information, resource requirement tags, and plugin dependency information.

[0009] Obtain the hardware resource information of the user device, adjust the node configuration information in the target template according to the hardware resource information, and generate the target workflow file according to the adjusted target template.

[0010] Optionally, semantic understanding is performed on the multimodal requirement information to obtain a set of semantic tags and workflow node links, including:

[0011] Using a pre-defined semantic analysis model, the multimodal demand information is semantically understood to obtain multiple initial semantic tags corresponding to the multimodal demand information.

[0012] Establish an initial mapping relationship between the multiple initial semantic tags and preset workflow nodes;

[0013] The multiple initial semantic tags are displayed to the user through a visual interface, and the user's modification operations on the multiple initial semantic tags are received;

[0014] Based on the modification operation, the multiple initial semantic tags and initial mapping relationships are iteratively optimized, and the final set of semantic tags and workflow node links are determined based on the optimization results.

[0015] Optionally, based on the semantic tag set and workflow node links, a basic template matching the multimodal requirement information is selected from multiple pre-built workflow generation templates, including:

[0016] Determine the semantic similarity between the set of semantic tags and the template tags in each workflow generation template;

[0017] Obtain the user device's environment information, and determine the compatibility of each workflow generation template based on the environment information and the plugin dependency information in each workflow generation template;

[0018] Obtain the hardware resource information of the user device, and determine the resource compatibility of each workflow generation template based on the hardware resource information and the resource requirement tags in each workflow generation template.

[0019] Based on the semantic similarity, compatibility, and resource adaptability of the templates generated by each workflow, a basic template matching the multimodal requirement information is determined.

[0020] Optionally, based on the semantic similarity, compatibility, and resource adaptability of the templates generated for each workflow, a basic template matching the multimodal requirement information is determined, including:

[0021] Retrieve the historical success rate and update time recorded in each workflow generation template;

[0022] The score for each workflow generation template is determined based on its semantic similarity, compatibility, resource adaptability, historical success rate, and update time.

[0023] Based on the scores corresponding to the templates generated by each workflow, a basic template that matches the multimodal requirement information is determined.

[0024] Optionally, after obtaining the target template, the following may also be included:

[0025] Obtain the node configuration information in the target template, and compare the semantic tag set with the node configuration information;

[0026] Based on the comparison results and the parameter constraints corresponding to each node, the node configuration information is adjusted.

[0027] Optionally, the node configuration information in the target template may be adjusted based on the hardware resource information, including:

[0028] Based on the node configuration information and resource requirement tags in the target template, determine the resource requirement vector corresponding to each node in the target template;

[0029] Based on the hardware resource information, determine the hardware state vector corresponding to the user equipment;

[0030] Based on the resource requirement vector corresponding to each node in the target template and the hardware state vector corresponding to the user equipment, the resource satisfaction degree of the target template under the user equipment is determined.

[0031] Based on the resource satisfaction level, the node configuration information in the target template is adjusted.

[0032] Optionally, before generating the target workflow file based on the adjusted target template, the following steps are also included:

[0033] In response to a user's modification request triggered by the target template through a visual interface, the target template is adjusted according to the modification request.

[0034] According to another aspect of the present invention, an intelligent workflow generation apparatus is provided, comprising:

[0035] The semantic understanding module is used to acquire multimodal demand information input by the user, perform semantic understanding on the multimodal demand information, and obtain a set of semantic tags and workflow node links;

[0036] The template filtering module is used to filter basic templates that match the multimodal requirement information from multiple pre-built workflow generation templates based on the semantic tag set and workflow node links, and to expand the nodes in the basic templates to obtain target templates; each workflow generation template pre-records node configuration information, resource requirement tags, and plugin dependency information.

[0037] The template adjustment module is used to obtain the hardware resource information of the user device, adjust the node configuration information in the target template according to the hardware resource information, and generate a target workflow file based on the adjusted target template.

[0038] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0039] At least one processor; and a memory communicatively connected to said at least one processor;

[0040] The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the intelligent workflow generation method according to any embodiment of the present invention.

[0041] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the intelligent workflow generation method according to any embodiment of the present invention.

[0042] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the intelligent workflow generation method described in any embodiment of the present invention.

[0043] The technical solution provided in this invention involves acquiring multimodal requirement information input by the user, performing semantic understanding on the multimodal requirement information to obtain a set of semantic tags and workflow node links, selecting a basic template that matches the multimodal requirement information from multiple pre-constructed workflow generation templates based on the semantic tag set and workflow node links, and expanding the nodes in the basic template to obtain a target template. Each workflow generation template pre-records node configuration information, resource requirement tags, and plugin dependency information, acquires the hardware resource information of the user's device, and adjusts the node configuration information in the target template based on the hardware resource information. This technology, which generates target workflow documents based on the adjusted target template, provides a way for non-technical personnel to quickly generate workflows, reducing the technical difficulty of workflow generation. Compared to existing technologies where users manually break down requirements into tool nodes, this method improves workflow generation efficiency. By combining semantic analysis technology to understand user requirements, the quality of workflow generation results can be improved, rework rates can be reduced, and resource compatibility between user devices and target templates can be achieved. This avoids problems such as templates failing to function properly due to unsupported user device hardware or missing plugins, thus ensuring smooth workflow implementation and achieving intelligent workflow generation solutions.

[0044] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of an intelligent workflow generation method provided according to an embodiment of the present invention;

[0047] Figure 2 This is a flowchart of another intelligent workflow generation method provided by an embodiment of the present invention;

[0048] Figure 3 This is a flowchart of another intelligent workflow generation method provided by an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of an intelligent workflow generation device according to an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the intelligent workflow generation method of this invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0053] Figure 1 This is a flowchart illustrating an intelligent workflow generation method provided in an embodiment of the present invention. This embodiment is applicable to situations where workflows are automatically generated based on user-input requirements. The method can be executed by an intelligent workflow generation device, which can be implemented in hardware and / or software and configured in an electronic device. Figure 1 As shown, the method includes:

[0054] Step 110: Obtain multimodal requirement information input by the user, perform semantic understanding on the multimodal requirement information, and obtain a set of semantic tags and workflow node links.

[0055] In this embodiment, the multimodal requirement information can be the user's requirements for constructing a specific workflow. The multimodal requirement information may include user-inputted text information, image information, and voice information, etc., and this embodiment does not impose any limitations on this.

[0056] After obtaining the multimodal requirement information, a preset semantic recognition model can be used to perform semantic understanding on the multimodal requirement information to obtain multiple semantic tags corresponding to the multimodal requirement information, and a semantic tag set can be established based on the multiple semantic tags. Specifically, the semantic tags can be data tags specified in the multimodal requirement information, such as people, scenes, meeting rooms, outputs, optimizations, and styles.

[0057] In this step, after determining the semantic tag set in the above manner, each semantic tag can be mapped to a specific node in the workflow node library, and the workflow node link can be determined based on the mapping results and the dependencies between the nodes.

[0058] Specifically, the workflow node library includes multiple tool nodes for building workflows, such as ModelLoader node, LORA node, Defoum node, KSampler node, etc.

[0059] Step 120: Based on the semantic tag set and workflow node links, select a basic template that matches the multimodal requirement information from multiple pre-built workflow generation templates, and expand the nodes in the basic template to obtain the target template; each workflow generation template pre-records node configuration information, resource requirement tags and plugin dependency information.

[0060] In this embodiment, optionally, multiple workflow generation templates can be pre-established before obtaining the multimodal requirement information input by the user, as shown in Table 1.

[0061] Each workflow generation template pre-records node configuration information, resource requirement tags, and plugin dependency information. Node configuration information may include the tool nodes used under the corresponding workflow generation template, as well as the parameter configuration information of each tool node. Resource requirement tags can be the hardware resources required by each tool node during the normal operation of the corresponding workflow generation template. Plugin dependency information can be the additional plugin tools that the corresponding workflow generation template depends on during normal operation.

[0062] In this step, specifically, the semantic tag set can be matched with the template name and / or scene tag in each workflow generation template, the workflow node link can be matched with the node configuration information in each workflow generation template, and the hardware resources and plugin installation information of the user device can be matched with the plugin dependency information in each workflow generation template. Finally, a basic template that matches the multimodal requirement information is obtained. Then, the nodes in the basic template are expanded and adjusted according to the user's semantic tag set so that the basic template specifically meets the user's needs, thereby obtaining the target template.

[0063] Table 1

[0064]

[0065] For example, if the plugin tool specified in the base template is not installed on the user's device, the user can be prompted with a message "XX plugin not installed" to prompt the user to install the plugin tool.

[0066] Step 130: Obtain the hardware resource information of the user device, adjust the node configuration information in the target template according to the hardware resource information, and generate the target workflow file according to the adjusted target template.

[0067] In this step, the parameter configuration information of each tool node in the target template can be optimized and adjusted based on the hardware resource information of the user's device to ensure that the target template can run normally.

[0068] In a specific embodiment, taking the aforementioned T101 template as an example, assuming the memory requirements of each node in the template are: Model Loader (GPU 3GB) + LORA (GPU 1GB) + ControlNet (GPU 2GB) + KSampler (GPU 4GB) + Defoum (GPU 6GB), the peak memory requirement is 8GB. If the user device's hardware resource information shows a total of 8GB of memory (currently 2GB occupied, 6GB remaining), meaning the peak memory in the hardware resources is insufficient to support the normal operation of the T101 template, then the parameter configuration information of each node needs to be optimized to reduce the memory usage of the T101 template.

[0069] Optionally, the resolution corresponding to Deferum can be reduced from 1080P to 720P (reducing VRAM usage from 6GB to 4GB), or a model fragmentation loading strategy can be enabled for the KSampler node (reducing VRAM usage by 1GB). The optimized T101 template requires 3+1+2+3+4=13GB of VRAM, with actual VRAM usage ≤6GB, thus ensuring the T101 template runs normally on user devices.

[0070] In addition, after optimizing and adjusting the parameter configuration information of the nodes in the target template in the above manner, a resource adaptation report can be generated based on the adjustment results. This report is used to show the comparison of optimized hardware parameters and is then presented to the user. After the user confirms the report, the adjusted target template is used to generate a target workflow file in .workflow format.

[0071] Specifically, the target workflow file may include node information (including node identifier, type, etc.), port connection information (clearly defining the data flow between nodes), node parameter configuration information, and metadata corresponding to the workflow name. Additionally, operation guides for the target workflow file can be provided to users, explaining how to import the .workflow format target workflow file and plugin installation steps, etc.

[0072] In this embodiment, optionally, before using the target template to generate the target workflow file, the performance of the target template can be verified, and the target template can be optimized again based on the verification results until the performance of the target template meets the preset requirements.

[0073] In one embodiment of this example, a core node sampling strategy can be adopted, that is, only the core node links that affect the workflow generation effect in the target template (such as Model Loader node, KSampler node, and Detailer node, etc.) are executed, while non-core nodes (such as Text Render and watermark nodes) are skipped, thereby generating a preview result of the target template (such as 10-20 frames of small sample images or a 5-second video). Then, the similarity between the preview result and the semantic tag set is determined by a pre-built multimodal effect evaluation model. If the similarity is greater than or equal to a preset value (such as 0.9), the target template is used to generate a complete target workflow file. If the similarity is less than the preset value, the node parameter configuration information in the target template is automatically adjusted backtracking, and the preview result is regenerated until the similarity is greater than or equal to the preset value.

[0074] In a specific embodiment, the similarity of the preview results with the semantic tag set in terms of character features, scene features, and style features can be calculated separately, and the similarity of the character features, scene features, and style features can be weighted and fused according to a preset weight ratio to obtain the final similarity.

[0075] This involves comparing the match between the age, clothing, and actions of the individuals in the preview results and the set of semantic tags to determine the similarity of their characteristics. The similarity of scene features is determined by calculating the coverage and fidelity of the preview results with the scene elements (such as meeting rooms, product display stands, etc.) in the semantic tag set. The similarity of style features is determined by comparing the preview results with the style features in the semantic tag set using a pre-defined style classification model. Finally, the final similarity between the preview results and the semantic tag set is determined using the following formula. :

[0076] .

[0077] The advantage of this setup is that by filtering core nodes in the target template for verification, the accuracy of the verification results can be improved while maintaining verification efficiency. This ensures that the consistency between the target template preview result and the complete execution result reaches over 93%, thereby identifying 92% of performance deviation issues in advance. By previewing and verifying the target template, the workflow rework rate can be reduced from 35% in the traditional approach to 8%, thus saving resources consumed in workflow generation.

[0078] The technical solution provided by this invention provides a way for non-technical personnel to quickly generate workflows. This is achieved by acquiring multimodal requirement information input by the user, performing semantic understanding on the multimodal requirement information to obtain a set of semantic tags and workflow node links, selecting a basic template matching the multimodal requirement information from multiple pre-built workflow generation templates based on the semantic tag set and workflow node links, and expanding the nodes in the basic template to obtain a target template. Furthermore, it involves acquiring hardware resource information of the user's device, adjusting the node configuration information in the target template based on the hardware resource information, and generating a target workflow file based on the adjusted target template. This reduces the technical difficulty of workflow generation solutions compared to existing technologies where users manually break down requirements into tool node combinations, thus improving workflow generation efficiency. Combining semantic analysis technology with semantic understanding of user requirement information improves the quality of workflow generation results, reduces rework rates, and ensures resource compatibility between the user's device and the target template. This avoids problems such as templates failing to function properly due to unsupported user device hardware or missing plugins, thereby ensuring smooth workflow implementation and achieving intelligent workflow generation solutions.

[0079] Figure 2 A flowchart of another intelligent workflow generation method provided in an embodiment of the present invention is shown below. Figure 2 As shown, the method includes:

[0080] Step 210: Obtain multimodal demand information input by the user, and perform semantic understanding on the multimodal demand information through a preset semantic analysis model to obtain multiple initial semantic tags corresponding to the multimodal demand information.

[0081] Step 220: Establish the initial mapping relationship between the multiple initial semantic tags and the preset workflow nodes.

[0082] Step 230: Display the multiple initial semantic tags to the user through a visual interface, and receive the user's modification operations on the multiple initial semantic tags.

[0083] In this step, specifically, the multiple initial semantic tags and the dependencies between the mapped nodes can be displayed to the user through a visual interface, so that the user can edit and modify the multiple initial semantic tags and the mapped nodes.

[0084] In addition, users can upload supplementary reference images (such as product appearance images) through the interface. Upon receiving the reference image, the initial semantic tags and mapped nodes can be automatically updated based on the reference image.

[0085] Step 240: Based on the modification operation, iteratively optimize the multiple initial semantic tags and initial mapping relationships, and determine the final semantic tag set and workflow node links based on the optimization results.

[0086] In this embodiment, optionally, after obtaining the multimodal requirement information input by the user, for textual requirements, a text semantic model can be used to process the textual requirements to obtain a 512-dimensional text vector. For image-related needs, a lightweight visual feature model can be used to extract features such as style and objects from the input image, outputting a 512-dimensional image vector. For voice-related needs, speech commands can be converted into 128-dimensional speech vectors using speech-to-text models and emotion classification models. Simultaneously, intent analysis is performed on multimodal demand information to obtain demand intent vectors. Then, according to the weight values ​​of the vectors corresponding to each mode. The multimodal vectors are weighted and fused to obtain the semantic analysis results before the user's modification operation. Among them, the weight values ​​of the vectors corresponding to each mode. It can be determined using the following formula:

[0087] .

[0088] in, Cosine similarity is used to measure the correlation between a single modal vector and an intent vector. User-corrected coefficients, user-corrected modes Uncorrected modes ; To adjust the weighting coefficients; , .

[0089] In this embodiment, after a user performs modification operations on multiple initial semantic tags, the semantic analysis result after the modification operations can be determined in the manner described above. and calculate and Difference between :

[0090] .

[0091] if Then, the feature fusion and node mapping process described above will be re-executed; if In this case, only the node mapping results and node parameter configuration information are locally updated to reduce computational overhead. In this embodiment, 0.2 is a preset difference threshold based on the actual data processing procedure.

[0092] The advantage of this setup is that iterative optimization of semantic tags and mapping relationships can resolve issues arising from misunderstandings of user needs. It also improves the accuracy of semantic tag sets and workflow node link matching results.

[0093] Step 250: Determine the semantic similarity between the semantic tag set and the template tags in each workflow generation template; obtain the environmental information of the user device, and determine the compatibility of each workflow generation template based on the environmental information and the plugin dependency information in each workflow generation template.

[0094] Specifically, in this step, the cosine similarity between the semantic tag set and each template tag can be calculated, and this result can be multiplied by 100 to obtain the semantic similarity. .

[0095] Simultaneously, based on the user's device environment information and the plugin dependency information in each workflow generation template, the compatibility of each workflow generation template is determined. For example, the T101 template requires ControlNet and Defoum plugins. If these are already installed on the user's device, then confirm. It is 100.

[0096] Step 260: Obtain the hardware resource information of the user device, and determine the resource compatibility of each workflow generation template based on the hardware resource information and the resource requirement tags in each workflow generation template.

[0097] In this step, the resource requirement tags in each workflow generation template are compared with the hardware resource information of the user's device, and the resource matching degree is determined based on the comparison results. For example, assuming the user's device currently has 8GB of remaining video memory, and the T101 template requires 8GB of video memory, then the resource matching degree of the T101 template needs to be determined. If the value is 100, and the T203 template requires 12GB of video memory, then the resource matching degree of the T203 template is determined. It is 60.

[0098] Step 270: Based on the semantic similarity, compatibility, and resource adaptability of the templates generated by each workflow, determine the basic template that matches the multimodal requirement information.

[0099] In one embodiment of this example, a basic template matching the multimodal requirement information is determined based on the semantic similarity, compatibility, and resource adaptability of each workflow generation template. This includes: obtaining the historical success rate and update time recorded in each workflow generation template; determining the score corresponding to each workflow generation template based on the semantic similarity, compatibility, resource adaptability, historical success rate, and update time; and determining the basic template matching the multimodal requirement information based on the score corresponding to each workflow generation template.

[0100] In this embodiment, the pre-established multiple workflow generation templates can be as shown in Table 2.

[0101] In addition to recording node configuration information, resource requirement tags, and plugin dependency information, each workflow generation template also records the historical success rate and the last update time.

[0102] In this embodiment, the implementation rate of each workflow generation template can be determined based on its historical success rate. For example, assuming a success rate of 92%, then the degree of implementation is determined. If the score is 92 and the success rate is 88%, then the implementation rate is determined. The value is 88. Additionally, the update level of each workflow template can be determined based on its update time. For example, assuming the most recent update time of template T101 is less than 30 days from the current time, then the update level is determined. If the value is 100, and the most recent update time of template T203 is more than 30 days from the current time, then the update level is determined. It is 90.

[0103] Table 2

[0104]

[0105] Finally, the templates corresponding to each workflow can be generated. , , , as well as Perform weighted fusion to determine the score corresponding to each workflow generation template. :

[0106] .

[0107] Then, the workflow generation template with the higher score is selected as the base template to match the multimodal requirement information. In the above formula, 0.4, 0.2, 0.2, 0.1, and 0.1 are the weight values ​​preset in this embodiment based on the actual data processing process.

[0108] Step 280: Expand the nodes in the basic template to obtain the target template, obtain the hardware resource information of the user device, adjust the node configuration information in the target template according to the hardware resource information, and generate the target workflow file according to the adjusted target template.

[0109] The technical solution provided by this invention reduces the technical difficulty of workflow generation by acquiring multimodal requirement information, obtaining initial semantic tags through semantic understanding of the multimodal requirement information using a semantic analysis model, establishing an initial mapping relationship between the initial semantic tags and workflow nodes, receiving user modification operations on the initial semantic tags, iteratively optimizing the initial semantic tags and initial mapping relationship based on the modification operations, determining the final semantic tag set and workflow node links based on the optimization results, determining the semantic similarity between the semantic tag set and the template tags in each workflow generation template, determining the compatibility of each workflow generation template, determining the resource adaptability of each workflow generation template, determining a basic template matching the multimodal requirement information based on semantic similarity, compatibility, and resource adaptability, expanding the nodes in the basic template to obtain a target template, adjusting the node configuration information in the target template, and generating a target workflow file based on the adjusted target template. This approach improves the efficiency and quality of workflow generation, avoids problems such as templates failing to function properly due to user device hardware incompatibility or missing plugins, ensures smooth workflow implementation, and achieves intelligent workflow generation.

[0110] Figure 3 A flowchart of another intelligent workflow generation method provided in an embodiment of the present invention is shown below. Figure 3 As shown, the method includes:

[0111] Step 310: Obtain multimodal requirement information input by the user, perform semantic understanding on the multimodal requirement information, and obtain a set of semantic tags and workflow node links.

[0112] Step 320: Based on the semantic tag set and workflow node links, select a basic template that matches the multimodal requirement information from multiple pre-built workflow generation templates, and expand the nodes in the basic template to obtain the target template.

[0113] Step 330: Obtain the node configuration information in the target template, compare the semantic tag set with the node configuration information, and adjust the node configuration information according to the comparison results and the parameter constraints corresponding to each node.

[0114] In one embodiment of this example, assuming that the user requirement in the semantic tag set is "smoothness priority, such as 60fps", and the default parameter of a certain node in the template is 30fps, with a constraint range of [10,60] fps, then the node's parameter can be adjusted to 60fps according to this constraint range.

[0115] In addition, nodes with the highest semantic relevance can be selected from the candidate nodes in the template as nodes for subsequent use. For example, if the user's requirement in the semantic tag set is "business lively style", then the node with the highest relevance "office_lively.safetensors" can be selected from the candidate LORA node list in the template.

[0116] Additionally, regular expressions and standard numerical ranges can be used to determine the validity of node parameters in the template, and invalid parameter values ​​can be adjusted to ensure that the parameter values ​​are within the standard numerical range.

[0117] In one specific embodiment, Table 3 illustrates the comparison results before and after adjusting the node configuration information in the T101 template according to the semantic tag set.

[0118] Step 340: Based on the node configuration information and resource requirement tags in the target template, determine the resource requirement vector corresponding to each node in the target template, obtain the hardware resource information of the user equipment, and determine the hardware state vector corresponding to the user equipment based on the hardware resource information.

[0119] In this step, a resource requirement vector corresponding to each node in the target template can be established based on the resource requirements of each node in the target template. .in, This indicates the GPU memory requirements (e.g., ModelLoader requires 3GB of video memory, and Defoum requires 6GB of video memory). Indicates the CPU core requirement (e.g., Text Render requires 2 cores, Detailer requires 4 cores). Indicate memory requirements (e.g., KSampler requires 8GB of memory, LORA requires 2GB of memory). This indicates the latency requirements of the nodes (e.g., ≤200ms for real-time preview and ≤500ms for background generation). It can also construct the hardware state vector corresponding to the user device in real time. The hardware resources corresponding to the user equipment may include local hardware resources of the device, as well as edge hardware or cloud hardware resources connected to the device, etc., and this embodiment does not impose any limitations on this.

[0120] Step 350: Determine the resource satisfaction level of the target template under the user equipment based on the resource requirement vector corresponding to each node in the target template and the hardware status vector corresponding to the user equipment; adjust the node configuration information in the target template according to the resource satisfaction level.

[0121] In this step, the resource satisfaction level of the target template under the user device can be determined according to the following formula. :

[0122] .

[0123] Specifically, if In this case, there is no need to optimize the target template, and the target template can be directly applied in the subsequent execution process.

[0124] if In such cases, lightweight optimization strategies can be used to process the node configuration information in the target template. For example, without affecting the core effect of the template, the model precision can be reduced or unnecessary functions can be turned off. Taking the Defolum node as an example, the smoothness can be reduced from 1080P to 720P. Taking the KSampler node as an example, the floating-point data precision can be reduced from FP16 to FP8, while the real-time preview function can be turned off.

[0125] if In this case, a deep optimization strategy combined with cross-node scheduling can be used to process the node configuration information in the target template, such as reducing node resolution, splitting the model, or scheduling some nodes to edge hardware or cloud hardware connected to the device.

[0126] Table 3

[0127]

[0128] The advantage of this setup is that by associating the resource requirements of template nodes with the hardware status, the node configuration can be dynamically optimized, avoiding hardware memory overflow and resource waste, and ensuring that the workflow can be successfully implemented.

[0129] Step 360: In response to a user's modification request triggered by the target template through a visual interface, adjust the target template according to the modification request.

[0130] In the prior art, the parameters of fixed workflow generation templates are usually unchanging, which cannot meet the personalized configuration needs of users. To address this, this embodiment proposes a visual interactive template editing method, which allows users to intuitively modify the target template, lowers the operation threshold, and keeps the workflow generation scheme consistent with the user's operating habits.

[0131] Specifically, the target template can be displayed to the user through a visual interface. Users can add nodes to the target template and configure node parameters, as well as delete one or more nodes, such as redundant watermark nodes. Furthermore, users can preview the generated result of the target template and apply it to subsequent processes after confirming the preview.

[0132] Step 370: Generate the target workflow document based on the adjusted target template.

[0133] The technical solution provided by this invention improves workflow generation efficiency and quality by acquiring multimodal requirement information input by the user, performing semantic understanding on the multimodal requirement information to obtain a set of semantic tags and workflow node links, selecting a basic template that matches the multimodal requirement information from multiple pre-built workflow generation templates, expanding the nodes in the basic template to obtain a target template, acquiring the node configuration information in the target template, comparing the set of semantic tags with the node configuration information, adjusting the node configuration information according to the comparison results and the parameter constraints corresponding to each node, determining the resource requirement vector corresponding to each node in the target template, determining the hardware state vector corresponding to the user device, and determining the resource satisfaction of the target template under the user device; adjusting the node configuration information in the target template according to the resource satisfaction, responding to the modification request triggered by the user through the visual interface, adjusting the target template according to the modification request, and generating a target workflow file based on the adjusted target template. This technical means can improve the workflow generation efficiency and the quality of the workflow generation results, avoid the problem that the template cannot run normally due to the user device hardware not supporting it or missing plugins, ensure the smooth implementation of the workflow, and realize the intelligentization of the workflow generation solution.

[0134] Based on the above embodiments, in order to better describe this solution, the following preferred embodiments can be referred to:

[0135] Obtain multimodal requirement information from user input This includes text, images, and voice;

[0136] Calling multimodal semantic understanding interaction algorithms to... The process involves parsing and interactive correction to output a precise set of semantic tags. and node dependencies ;by As input, a five-dimensional template adaptive matching algorithm is invoked to retrieve and score templates from the template library, and then the optimal template is selected. Initial parameter configuration of its nodes .

[0137] Combine the optimal template Parameter constraints corresponding to each node Configuring initial parameters Optimize and generate compliance parameters .

[0138] Obtain the hardware status of the user device Call the resource dynamic collaborative adaptation algorithm to calculate exist Resource satisfaction :

[0139] like Then the final configuration parameters of each node in the optimal template are determined. ;

[0140] like Then, a lightweight optimization strategy (such as reducing precision, disabling the preview function, etc.) will be implemented. Further adjustments were made. ;

[0141] like If necessary, a deep optimization strategy (such as reducing resolution, model fragmentation, etc.) or cross-device node scheduling will be implemented. Further adjustments were made. ;

[0142] Then based on , as well as Generate workflow files in .workflow format. Call the workflow effect verification algorithm to... A lightweight preview and semantic consistency check were performed to obtain the similarity evaluation results. and verification report .

[0143] Among them, if Then, based on the type of deviation, the parameter configuration of the nodes in the optimal template is adjusted backtracking, and the similarity is re-evaluated. Continue until the preset requirements are met. Finally, output the final workflow file. and verification report .

[0144] Figure 4This invention provides a schematic diagram of the structure of an intelligent workflow generation device, which is applied in electronic devices, such as... Figure 4 As shown, the device includes: a semantic understanding module 410, a template filtering module 420, and a template adjustment module 430.

[0145] The semantic understanding module 410 is used to acquire multimodal demand information input by the user, perform semantic understanding on the multimodal demand information, and obtain a set of semantic tags and workflow node links;

[0146] The template filtering module 420 is used to filter basic templates that match the multimodal requirement information from a number of pre-built workflow generation templates based on the semantic tag set and workflow node links, and to expand the nodes in the basic templates to obtain target templates; each workflow generation template pre-records node configuration information, resource requirement tags and plugin dependency information.

[0147] The template adjustment module 430 is used to obtain the hardware resource information of the user device, adjust the node configuration information in the target template according to the hardware resource information, and generate a target workflow file according to the adjusted target template.

[0148] The technical solution provided by this invention provides a way for non-technical personnel to quickly generate workflows. This is achieved by acquiring multimodal requirement information input by the user, performing semantic understanding on the multimodal requirement information to obtain a set of semantic tags and workflow node links, selecting a basic template matching the multimodal requirement information from multiple pre-built workflow generation templates based on the semantic tag set and workflow node links, and expanding the nodes in the basic template to obtain a target template. Furthermore, it involves acquiring hardware resource information of the user's device, adjusting the node configuration information in the target template based on the hardware resource information, and generating a target workflow file based on the adjusted target template. This reduces the technical difficulty of workflow generation solutions compared to existing technologies where users manually break down requirements into tool node combinations, thus improving workflow generation efficiency. Combining semantic analysis technology with semantic understanding of user requirement information improves the quality of workflow generation results, reduces rework rates, and ensures resource compatibility between the user's device and the target template. This avoids problems such as templates failing to function properly due to unsupported user device hardware or missing plugins, thereby ensuring smooth workflow implementation and achieving intelligent workflow generation solutions.

[0149] Based on the above embodiments, the semantic understanding module 410 includes:

[0150] The initial analysis unit is used to perform semantic understanding on the multimodal requirement information through a preset semantic analysis model to obtain multiple initial semantic tags corresponding to the multimodal requirement information; and to establish an initial mapping relationship between the multiple initial semantic tags and preset workflow nodes.

[0151] The interactive optimization unit is used to display the multiple initial semantic tags to the user through a visual interface, receive the user's modification operations on the multiple initial semantic tags, iteratively optimize the multiple initial semantic tags and the initial mapping relationship according to the modification operations, and determine the final semantic tag set and workflow node links according to the optimization results.

[0152] Template filtering module 420 includes:

[0153] The adaptability determination unit is used to determine the semantic similarity between the semantic tag set and the template tags in each workflow generation template; acquire the environmental information of the user device, and determine the compatibility of each workflow generation template based on the environmental information and the plugin dependency information in each workflow generation template; acquire the hardware resource information of the user device, and determine the resource adaptability of each workflow generation template based on the hardware resource information and the resource requirement tags in each workflow generation template; and determine the basic template that matches the multimodal requirement information based on the semantic similarity, compatibility, and resource adaptability of each workflow generation template.

[0154] The scoring unit is used to obtain the historical success rate and update time recorded in each workflow generation template; determine the score corresponding to each workflow generation template based on the semantic similarity, compatibility, resource adaptability, historical success rate and update time; and determine the basic template that matches the multimodal requirement information based on the score corresponding to each workflow generation template.

[0155] The node constraint analysis unit is used to obtain the node configuration information in the target template, compare the semantic tag set with the node configuration information, and adjust the node configuration information according to the comparison results and the parameter constraints corresponding to each node.

[0156] Template adjustment module 430 includes:

[0157] The resource satisfaction determination unit is configured to: determine the resource requirement vector corresponding to each node in the target template based on the node configuration information and resource requirement tags in the target template; determine the hardware state vector corresponding to the user equipment based on the hardware resource information; determine the resource satisfaction of the target template under the user equipment based on the resource requirement vector corresponding to each node in the target template and the hardware state vector corresponding to the user equipment; and adjust the node configuration information in the target template based on the resource satisfaction.

[0158] An interactive editing unit is used to respond to a user's modification request triggered by the target template through a visual interface, and to adjust the target template according to the modification request.

[0159] The above-described apparatus can execute the methods provided in all the foregoing embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the above methods. Technical details not described in detail in the embodiments of the present invention can be found in the methods provided in all the foregoing embodiments of the present invention.

[0160] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0161] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0162] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0163] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as intelligent workflow generation methods.

[0164] In some embodiments, the intelligent workflow generation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the intelligent workflow generation method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the intelligent workflow generation method by any other suitable means (e.g., by means of firmware).

[0165] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0166] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0167] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0168] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0169] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0170] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0171] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0172] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method of intelligent workflow generation, characterized by, The method comprises the following steps: obtaining multi-modal requirement information input by a user, performing semantic understanding on the multi-modal requirement information to obtain a semantic tag set and a workflow node link; screening a basic template matched with the multi-modal requirement information from a plurality of pre-constructed workflow generation templates according to the semantic tag set and the workflow node link, and expanding nodes in the basic template to obtain a target template; wherein node configuration information, resource requirement tags and plug-in dependency information are pre-recorded in each workflow generation template; obtaining hardware resource information of a user device, adjusting the node configuration information in the target template according to the hardware resource information, and generating a target workflow file according to the adjusted target template.

2. The method of claim 1, wherein, The method comprises the following steps: performing semantic understanding on the multi-modal requirement information by a pre-set semantic analysis model to obtain a plurality of initial semantic tags corresponding to the multi-modal requirement information; establishing an initial mapping relationship between the plurality of initial semantic tags and pre-set workflow nodes; displaying the plurality of initial semantic tags to the user through a visual interface, and receiving a modification operation of the user on the plurality of initial semantic tags; iteratively optimizing the plurality of initial semantic tags and the initial mapping relationship according to the modification operation, and determining a final semantic tag set and a workflow node link according to an optimization result.

3. The method of claim 1, wherein, The method comprises the following steps: determining semantic similarity between the semantic tag set and template tags in each workflow generation template; obtaining environment information of a user device, and determining a compatibility corresponding to each workflow generation template according to the environment information and plug-in dependency information in each workflow generation template; obtaining hardware resource information of a user device, and determining a resource adaptation degree corresponding to each workflow generation template according to the hardware resource information and resource requirement tags in each workflow generation template; determining a basic template matched with the multi-modal requirement information according to the semantic similarity, the compatibility and the resource adaptation degree corresponding to each workflow generation template.

4. The method of claim 3, wherein, The method comprises the following steps: obtaining a historical success rate and an update time recorded in each workflow generation template; determining a score corresponding to each workflow generation template according to the semantic similarity, the compatibility, the resource adaptation degree, the historical success rate and the update time corresponding to each workflow generation template; determining a basic template matched with the multi-modal requirement information according to the score corresponding to each workflow generation template.

5. The method of claim 1, wherein, After obtaining the target template, the method further comprises the following steps: obtaining node configuration information in the target template, and comparing the semantic tag set with the node configuration information; adjusting the node configuration information according to a comparison result and a parameter constraint condition corresponding to each node.

6. The method of claim 1, wherein, According to the hardware resource information, the node configuration information in the target template is adjusted, comprising: According to the node configuration information in the target template and the resource demand label, the resource demand vector corresponding to each node in the target template is determined; According to the hardware resource information, the hardware state vector corresponding to the user equipment is determined; According to the resource demand vector corresponding to each node in the target template and the hardware state vector corresponding to the user equipment, the resource satisfaction degree corresponding to the target template under the user equipment is determined; According to the resource satisfaction degree, the node configuration information in the target template is adjusted.

7. The method of claim 1, wherein, Before generating the target workflow file according to the adjusted target template, it further comprises: In response to the modification request triggered by the user through the visual interface to the target template, the target template is adjusted according to the modification request.

8. An intelligent workflow generation apparatus, characterized by comprising: The device comprises: The semantic understanding module is used to obtain the multi-modal demand information input by the user, and to obtain the semantic label set and the workflow node link by performing semantic understanding on the multi-modal demand information; The template screening module is used to screen the basic template matched with the multi-modal demand information from the plurality of workflow generation templates pre-constructed according to the semantic label set and the workflow node link, and to obtain the target template by expanding the nodes in the basic template; Each workflow generation template pre-records node configuration information, resource demand labels and plug-in dependency information; The template adjustment module is used to obtain the hardware resource information of the user equipment, to adjust the node configuration information in the target template according to the hardware resource information, and to generate the target workflow file according to the adjusted target template.

9. An electronic device, comprising: The electronic device comprises: At least one processor; and The memory is in communication connection with the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the intelligent workflow generation method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to enable the processor to execute the intelligent workflow generation method in any one of claims 1-7 when executed.

Images