Method for building automatic process based on natural language capability of multi-modal large model

Through the multimodal large model and graph neural network generation process framework, the shortcomings of multimodal appeal input and process planning in the existing technology are solved, the diversity of appeals and the coherence of processes are achieved, and accurate automated process requirements are generated.

CN120447871APending Publication Date: 2025-08-08WUXI RONGZHI TECH CO LTD +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202510534958.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-04-27
Publication Date
2025-08-08

AI Technical Summary

Technical Problem

The existing automated process construction methods cannot realize multimodal appeal input, resulting in narrow appeal entry, semantic understanding and demand analysis, and process planning lacks dependence and constraint judgment, reducing process coherence.

Method used

The multimodal large model is used for semantic understanding and requirement analysis, through multimodal data acquisition, appeal analysis and processing, process planning, function division and unit coordination, combined with the graph neural network generation process framework, and the process sequence is optimized using task dependency analysis and resource constraint optimization algorithm.

Benefits of technology

It realizes efficient integration of multimodal demands, improves the diversity and accuracy of demands, generates accurate automated process requirements, and enhances the coherence and adaptability of the process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120447871A_ABST
    Figure CN120447871A_ABST
Patent Text Reader

Abstract

The invention discloses a method for building an automatic process based on natural language capability of a multi-modal large model. The method comprises the following steps: S1, collecting multi-modal data; s2, analyzing and processing the demands; s3, an appeal process is planned; s4, automatic task planning; s5, function division; s6, unit coordination; s7, verifying the integrity of the process parameters; s8, generating an actuator; according to the method, the dynamic memory matrix is established to realize long-distance dependence modeling, and learning can be performed through a memory matrix attenuation factor to adapt to different scene information life cycles; matrix processing is carried out on the important features, the dependency strength between contexts is determined according to the relation probability, adaptive association demands can be achieved conveniently, and accurate automatic process demands are obtained; the process planning model generates a process framework based on a process topology generator of a graph neural network, a task dependency analyzer and a resource constraint optimization algorithm, and optimizes the process framework according to a preliminary process framework and a final process framework.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to process automation, and more specifically, to a method for building an automated process based on the natural language capabilities of a multimodal large model. Background Art

[0002] The construction of traditional automation processes usually includes the following stages:

[0003] Demand Analysis: Communicate with customers to clarify automation goals and expected success; collect and analyze detailed information on business processes, including bottlenecks and improvement points of current processes;

[0004] Solution Design: Based on the results of the demand analysis, design the framework and steps of the automation process; determine the required technology stack and tools; and develop a detailed implementation plan and timetable;

[0005] Technology selection: select appropriate automation tools and platforms; determine development languages and frameworks;

[0006] Prototyping: Build a preliminary prototype of the automated process; conduct small-scale tests to prove the feasibility of the concept;

[0007] Coding implementation: write code according to the design plan; implement each module and function in the automation process;

[0008] Integration testing: Perform unit testing on the developed automation process to ensure that each part works properly; perform integration testing to ensure that the various modules can work together;

[0009] User acceptance testing: Involve end users in testing to ensure that the automated process meets business requirements; collect user feedback and make necessary adjustments;

[0010] Deployment and rollout: Deploy the automated process to the production environment; monitor the deployment process to ensure a smooth transition;

[0011] Training and Documentation: Provide users with operational training on automated processes; write user manuals and maintenance documentation.

[0012] However, the existing automated process has the following problems:

[0013] The input function for demands is single and multimodal demand input cannot be realized, resulting in the narrowness of demand input. In addition, semantic understanding and demand analysis of input demands cannot be realized, resulting in unclear demands. In addition, in the existing technology, dependencies and constraints cannot be determined for process planning defects, resulting in barriers between processes and reducing the coherence of the processes. Summary of the Invention

[0014] One object of the present invention is to provide a new technical solution for a method of building an automated process based on the natural language capabilities of a multimodal large model.

[0015] According to a first aspect of the present invention, a method for automating a natural language capability building process based on a multimodal large model is provided, which is applied to a multimodal large model including a natural language capability building capability, and includes the following steps:

[0016] S1. Multimodal data collection: Collect multimodal data requests from users, and use the natural language capabilities of the multimodal large model to perform semantic understanding and demand analysis to obtain demand analysis results;

[0017] S2. Analysis and processing of demands: semantic understanding and demand analysis of user demands through the natural language capabilities of the multimodal large model;

[0018] S3. Demand process planning: Receive the demand analysis results and extract the user's actual automation process requirements, and then pass the extracted automation process requirements to the process planning model;

[0019] S4. Automated task planning: Input the automated process requirements into the process planning model to generate automated task planning, plan a preliminary process framework, and process the entire workflow to obtain the final process framework;

[0020] S5. Functional division: Based on the final process framework of the process planning model, divide the final process framework into several independent yet mutually cooperating functional units;

[0021] S6. Unit coordination: Use process orchestration algorithms to orchestrate the coordination work within the functional unit and between it and other units to generate automated processes;

[0022] S7. Verify process parameter integrity: Determine whether the automation process is complete;

[0023] S8. Executor generation: Submit the orchestrated automation process to the executor.

[0024] Optionally, the appeal in S1 includes text data and voice data;

[0025] The speech data is first pre-processed by performing noise reduction and endpoint detection to obtain processed speech data, and then the processed speech data is subjected to feature extraction and acoustic model matching through MFCC to convert the processed speech data into accurate text appeals.

[0026] Optionally, the multimodal large model in S2 performs semantic understanding and demand analysis on the appeal:

[0027] The semantic understanding first extracts features from the text data to obtain the feature matrix A of the text data, and then understands the context of the text through the feature matrix to obtain the memory matrix M of the text. t ;

[0028]

[0029] Where w i It represents the weight of the i-th feature in the feature matrix, λ represents the decay factor of the feature in the text, A(Q,K) represents the query of the feature in the feature matrix, (Q,K) represents the query key of the memory, and n represents the total number of features in the feature matrix.

[0030] Optionally, the requirement analysis is performed on the feature matrix A and the memory matrix M t Fusion, the memory matrix M t Perform relationship probability calculation and analysis with the feature matrix A;

[0031]

[0032] in, Represented as feature matrix A and memory matrix M t The mapping relationship between them;

[0033]

[0034] Among them, h i h j (M t ) is represented as the memory matrix M t The position in the memory matrix of , and i, j represent the horizontal and vertical positions in the memory matrix respectively, b r A is represented as the loss function of the feature matrix A;

[0035] Obtain automation process requirements in text through relationship probability:

[0036] The feature matrix A and the memory matrix M t to integrate;

[0037]

[0038] in, Indicates that the feature matrix A and memory matrix M are realized through relationship probability t For fusion, Ls represents the connective for knowledge injection, and h represents the automation process requirement.

[0039] Optionally, the process planning model generates a process framework based on a process topology generator, a task dependency analyzer, and a resource constraint optimization algorithm of a graph neural network;

[0040] Build the initial process topology graph through the GNN topology generator;

[0041] Apply a multi-dimensional dependency analyzer to analyze task timing relationships;

[0042] Use mixed integer programming for resource-constrained optimization;

[0043] Generate a process framework with execution strategy.

[0044] Optionally, the process topology generator of the graph neural network is calculated as follows:

[0045]

[0046] Among them, ReLU is represented as the activation function, w1 and w2 are represented as learnable weight matrices, H is represented as the vector of automation process requirements, and h v The node itself is transformed to meet the automation process requirements. It is represented as the node neighbor aggregation of automation process requirements, and u∈N(v) is represented as the set of neighbor automation process requirements;

[0047] p=σ(MLP([H i ][H i+1 ]));

[0048] Among them, p represents the order of automation process requirements, which is the preliminary process framework, MLP ([H i ][H i+1 ]) represents the splicing automation process requirement, and σ represents the Sigmoid function.

[0049] Optionally, the task dependency analyzer calculates as follows:

[0050]

[0051] Among them, CP represents the critical path of task dependency, x j It is represented as the task dependency strength, path is represented as the path set, which is all possible paths from the starting point to the end point; argmax represents the path selection;

[0052] The task dependency analyzer is used to calculate the dependency strength between the sequences of the automation process requirements, and the final sequence planning result of the automation process requirements is determined based on the dependency strength.

[0053] Optionally, the resource constraint optimization algorithm is calculated as follows:

[0054]

[0055] in, is represented as the objective function, T is represented as the scheduling of all automation process requirements, c t represents the scheduling cost of executing the automation process demand, x t Expressed as a binary decision variable, x t =1 means that the schedule t required by the automated process is selected for execution, otherwise it is not executed;

[0056] It is represented as resource constraint, K represents the total number of resources, R K It is represented as the scheduling set of automation process requirements that need to occupy resource type k, b k Expressed as the maximum available quantity of resource type k;

[0057] It is represented as the dependency constraint of the automation process requirement, pred(j) is represented as the set of predecessor automation process requirements of automation process requirement j, and d i It is expressed as the duration or processing time of the automation process requirement i;

[0058] x t ∈{0,1} represents a variable type constraint, forcing x t is a binary variable.

[0059] Optionally, in the above S7, if the automated process is judged and necessary parameter information is missing, the automated process needs to be arranged again; if user parameters are missing, it is necessary to interact with the user to obtain information.

[0060] Optionally, the steps of determining the automated process are as follows:

[0061] Check parameters: Ensure that the required parameters in the automation process remain intact;

[0062] Parameter type check: Check all parameter types in the automation process to ensure that all parameters have the same type.

[0063] Parameter retrieval: When necessary parameter information is missing, the system parameters are automatically retrieved and the automated process is orchestrated again;

[0064] User parameter detection: When user parameters are missing, an interactive completion interface is triggered for the user parameters, and the user completes the parameters based on the missing parameter information;

[0065] Create a tracking log: Create a parameter version tracking log based on the information from the parameter inspection.

[0066] Beneficial effects of the present invention:

[0067] The present invention can improve the diversity of demands through multimodal demand input, integrate text and voice bimodal input, and establish a dynamic memory matrix to achieve long-distance dependency modeling. The memory matrix attenuation factor can be learned to adapt to the information life cycle of different scenarios and improve the updating of demands. The present invention also collects important features in demands through semantic understanding and demand analysis, and processes these important features into a matrix. The dependency strength between contexts is determined based on relationship probability, which facilitates the realization of adaptive related demands and obtains accurate automated process requirements.

[0068] The process planning model generates a process framework based on the process topology generator, task dependency analyzer, and resource constraint optimization algorithm of the graph neural network, and optimizes the process framework according to the preliminary process framework and the final process framework;

[0069] The process topology generator based on the graph neural network automatically captures the implicit dependencies between tasks through the message passing mechanism, without the need for manual definition of rules. The task dependency analyzer calculates the dependency strength between the order of automated process requirements, and determines the final sequence planning result of the automated process requirements based on the dependency strength. The resource constraint optimization algorithm constrains the process framework and determines the process sequence in combination with the dependency relationship, that is, technical collaboration obtains the final process framework.

[0070] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0072] Figure 1 A flowchart illustrating the steps of a method for building an automated process based on natural language capabilities of a multimodal large model in one embodiment. DETAILED DESCRIPTION

[0073] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention.

[0074] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0075] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0076] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0077] like Figure 1 As shown, a method for automating the process of building natural language capabilities based on a multimodal large model is applied to a multimodal large model that includes the ability to build natural language capabilities, including the following steps:

[0078] S1. Multimodal data collection: Collect multimodal data requests from users, and use the natural language capabilities of the multimodal large model to perform semantic understanding and demand analysis to obtain demand analysis results;

[0079] S2. Analysis and processing of demands: semantic understanding and demand analysis of user demands through the natural language capabilities of the multimodal large model;

[0080] S3. Demand process planning: Receive the demand analysis results and extract the user's actual automation process requirements, and then pass the extracted automation process requirements to the process planning model;

[0081] S4. Automated task planning: Input the automated process requirements into the process planning model to generate automated task planning, plan a preliminary process framework, and process the entire workflow to obtain the final process framework;

[0082] S5. Functional division: Based on the final process framework of the process planning model, divide the final process framework into several independent yet mutually cooperating functional units;

[0083] S6. Unit coordination: Use process orchestration algorithms to orchestrate the coordination work within the functional unit and between it and other units to generate automated processes;

[0084] S7. Verify process parameter integrity: Determine whether the automation process is complete;

[0085] S8. Executor generation: Submit the orchestrated automation process to the executor.

[0086] In this embodiment, preferably, the appeal in S1 includes text data and voice data;

[0087] The speech data is first pre-processed by noise reduction and endpoint detection to obtain processed speech data, and then the processed speech data is subjected to feature extraction and acoustic model matching through MFCC to convert the processed speech data into accurate text appeals;

[0088] It should be noted that through multimodal demand input, the diversity of demands can be improved, and the robustness in noisy environments can be improved through the MFCC acoustic model and context feature matrix.

[0089] In this embodiment, preferably, the multimodal large model in S2 performs semantic understanding and demand analysis of the demands:

[0090] The semantic understanding first extracts features from the text data to obtain the feature matrix A of the text data, and then understands the context of the text through the feature matrix to obtain the memory matrix M of the text. t ;

[0091]

[0092] Where w i It represents the weight of the i-th feature in the feature matrix, λ represents the attenuation factor of the feature in the text, A(Q,K) represents the query of the features in the feature matrix, and (Q,K) represents the query key of the memory, and n represents the total number of features in the feature matrix;

[0093] It should be noted that the dynamic memory matrix realizes long-distance dependency modeling, the attenuation factor adapts to the life cycle of information in different scenarios, improves the update of demands, and for the feature matrix A and the memory matrix M t Make connections and improve the relevance between contextual appeals.

[0094] In this embodiment, preferably, the demand analysis is performed on the feature matrix A and the memory matrix M t Fusion, the memory matrix M t Perform relationship probability calculation and analysis with the feature matrix A;

[0095]

[0096] in, Represented as feature matrix A and memory matrix M t The mapping relationship between them;

[0097]

[0098] Among them, h i h j (M t ) is represented as the memory matrix M t The position in the memory matrix of , and i, j represent the horizontal and vertical positions in the memory matrix respectively, b r A is represented as the loss function of the feature matrix A;

[0099] Obtain automation process requirements in text through relationship probability:

[0100] The feature matrix A and the memory matrix M t to integrate;

[0101]

[0102] in, Indicates that the feature matrix A and memory matrix M are realized through relationship probability t For fusion, Ls represents the connective for knowledge injection, and h represents the automation process requirement;

[0103] It should be noted that by the feature matrix A and the memory matrix M t Mapping is performed between them to associate the features between the contexts, and the memory matrix M is eliminated through the loss function t data loss, improve data accuracy, and combine the feature matrix A and memory matrix M t It is convenient to generate automated process requirements, and the automated process requirements are combined with the context to maintain the integrity of the demands and improve the integrity of the demands through conjunctions.

[0104] In this embodiment, preferably, the process planning model generates a process framework based on a process topology generator, a task dependency analyzer, and a resource constraint optimization algorithm of a graph neural network;

[0105] Build the initial process topology graph through the GNN topology generator;

[0106] Apply a multi-dimensional dependency analyzer to analyze task timing relationships;

[0107] Use mixed integer programming for resource-constrained optimization;

[0108] Generate a process framework with execution strategy;

[0109] It should be noted that the process topology generator through the graph neural network automatically captures the implicit dependencies between tasks through the message passing mechanism, without the need for manual definition of rules, and the task dependency analyzer calculates the dependency strength between the sequence of automation process requirements, and determines the final sequence planning result of the automation process requirements through the dependency strength, and the resource constraint optimization algorithm implements constraints on the process framework and determines the process sequence in combination with the dependency relationship, that is, the final process framework is obtained through technical collaboration.

[0110] In this embodiment, preferably, the process topology generator of the graph neural network is calculated as follows:

[0111]

[0112] Among them, ReLU is represented as the activation function, w1 and w2 are represented as learnable weight matrices, H is represented as the vector of automation process requirements, and hv The node itself is transformed to meet the automation process requirements. It is represented as the node neighbor aggregation of automation process requirements, and u∈N(v) is represented as the set of neighbor automation process requirements;

[0113] p=σ(MLP([H i ][H i+1 ]));

[0114] Among them, p represents the order of automation process requirements, which is the preliminary process framework, MLP ([H i ][H i+1 ]) represents the splicing automation process requirement, and σ represents the Sigmoid function;

[0115] It should be noted that the process topology generator of the graph neural network facilitates the optimization of the vector of automated process requirements, determines the relationship between the vector of automated process requirements itself and its neighbors, and realizes the preliminary generation process framework.

[0116] In this embodiment, preferably, the task dependency analyzer calculates as follows:

[0117]

[0118] Among them, CP represents the critical path of task dependency, x j It is represented as the task dependency strength, path is represented as the path set, which is all possible paths from the starting point to the end point; argmax represents the path selection;

[0119] Calculating the dependency strength between the sequences of automation process requirements through the task dependency analyzer, and determining the final sequence planning result of the automation process requirements based on the dependency strength;

[0120] It should be noted that the dependencies between the vectors of automated process requirements are used, and a comprehensive and detailed final process framework is generated based on the process path.

[0121] In this embodiment, preferably, the calculation of the resource constraint optimization algorithm is as follows:

[0122]

[0123] in, is represented as the objective function, T is represented as the scheduling of all automation process requirements, c t represents the scheduling cost of executing the automation process demand, x t Expressed as a binary decision variable, x t =1 means that the schedule t required by the automated process is selected for execution, otherwise it is not executed;

[0124] It is represented as resource constraint, K represents the total number of resources, R K It is represented as the scheduling set of automation process requirements that need to occupy resource type k, b k Expressed as the maximum available quantity of resource type k;

[0125] It is represented as the dependency constraint of the automation process requirement, pred(j) is represented as the set of predecessor automation process requirements of automation process requirement j, and d i It is expressed as the duration or processing time of the automation process requirement i;

[0126] x t ∈{0,1} represents a variable type constraint, forcing x t is a binary variable;

[0127] It should be noted that the resource constraint optimization algorithm constrains the process framework and determines the process sequence in combination with the dependency relationship, thereby constraining the automation process requirements and improving the accuracy of the automation process requirements.

[0128] In this embodiment, preferably, in the step S7, if the automated process is determined to be missing necessary parameter information, the automated process needs to be re-orchestrated; if the user parameters are missing, it is necessary to interact with the user to obtain information;

[0129] The steps of the automated judgment process are as follows:

[0130] Check parameters: Ensure that the required parameters in the automation process remain intact;

[0131] Parameter type check: Check all parameter types in the automation process to ensure that all parameters have the same type.

[0132] Parameter retrieval: When necessary parameter information is missing, the system parameters are automatically retrieved and the automated process is orchestrated again;

[0133] User parameter detection: When user parameters are missing, an interactive completion interface is triggered for the user parameters, and the user completes the parameters based on the missing parameter information;

[0134] Establishing tracking log: Establishing parameter version tracking log based on the information of parameter detection and inspection;

[0135] It should be noted that completing the missing parameter information can ensure that the process framework of the automation process is accurate.

[0136] Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should be understood by those skilled in the art that modifications may be made to the above embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for building an automated process based on the natural language capabilities of a multimodal large model, characterized in that: The application of a large multimodal model with natural language capabilities includes the following steps: S1. Multimodal data collection: Collect multimodal data requests from users, and use the natural language capabilities of the multimodal large model to perform semantic understanding and demand analysis to obtain demand analysis results; S2. Analysis and processing of demands: semantic understanding and demand analysis of user demands through the natural language capabilities of the multimodal large model; S3. Demand process planning: Receive the demand analysis results and extract the user's actual automation process requirements, and then pass the extracted automation process requirements to the process planning model; S4. Automated task planning: Input the automated process requirements into the process planning model to generate automated task planning, plan a preliminary process framework, and process the entire workflow to obtain the final process framework; S5. Functional division: Based on the final process framework of the process planning model, divide the final process framework into several independent yet mutually cooperating functional units; S6. Unit coordination: Use process orchestration algorithms to orchestrate the coordination work within the functional unit and between it and other units to generate automated processes; S7. Verify process parameter integrity: Determine whether the automation process is complete; S8. Executor generation: Submit the orchestrated automation process to the executor.

2. The method for building an automated process based on natural language capabilities of a multimodal large model according to claim 1, characterized in that: The appeal in S1 includes text data and voice data; The speech data is first pre-processed by performing noise reduction and endpoint detection to obtain processed speech data, and then the processed speech data is subjected to feature extraction and acoustic model matching through MFCC to convert the processed speech data into accurate text appeals.

3. The method for building an automated process based on natural language capabilities of a multimodal large model according to claim 1, characterized in that: The multimodal large model in S2 provides semantic understanding and demand analysis of demands: The semantic understanding first extracts features from the text data to obtain the feature matrix A of the text data, and then understands the context of the text through the feature matrix to obtain the memory matrix M of the text. t ; Where w i It represents the weight of the i-th feature in the feature matrix, λ represents the decay factor of the feature in the text, A(Q,K) represents the query of the feature in the feature matrix, (Q,K) represents the query key of the memory, and n represents the total number of features in the feature matrix.

4. The method for building an automated process based on natural language capabilities of a multimodal large model according to claim 3 is characterized by: The requirement analysis is performed on the feature matrix A and the memory matrix M t Fusion, the memory matrix M t Perform relationship probability calculation and analysis with the feature matrix A; in, Represented as feature matrix A and memory matrix M t The mapping relationship between them; Among them, h i h j (M t ) is represented as the memory matrix M t The position in the memory matrix of , and i, j represent the horizontal and vertical positions in the memory matrix respectively, b r A is represented as the loss function of the feature matrix A; Obtain automation process requirements in text through relationship probability: The feature matrix A and the memory matrix M t to integrate; in, Indicates that the feature matrix A and memory matrix M are realized through relationship probability t For fusion, Ls represents the connective for knowledge injection, and h represents the automation process requirement.

5. The method for building an automated process based on natural language capabilities of a multimodal large model according to claim 1 is characterized by: The process planning model generates a process framework based on a process topology generator, a task dependency analyzer, and a resource constraint optimization algorithm based on a graph neural network; Build the initial process topology graph through the GNN topology generator; Apply a multi-dimensional dependency analyzer to analyze task timing relationships; Use mixed integer programming for resource-constrained optimization; Generate a process framework with execution strategy.

6. The method for building an automated process based on natural language capabilities of a multimodal large model according to claim 5, characterized in that: The calculation of the process topology generator of the graph neural network is as follows: Among them, ReLU is represented as the activation function, w1 and w2 are represented as learnable weight matrices, H is represented as the vector of automation process requirements, and h v The node itself is transformed to meet the automation process requirements. It is represented as the node neighbor aggregation of automation process requirements, and u∈N(v) is represented as the set of neighbor automation process requirements; p=σ(MLP([H i ][H i+1 ])); Among them, p represents the order of automation process requirements, which is the preliminary process framework, MLP ([H i ][H i+1 ]) represents the splicing automation process requirement, and σ represents the Sigmoid function.

7. The method for building an automated process based on natural language capabilities of a multimodal large model according to claim 6, characterized in that: The task dependency analyzer calculates as follows: Among them, CP represents the critical path of task dependency, x j It is represented as the task dependency strength, path is represented as the path set, which is all possible paths from the starting point to the end point; argmax represents the path selection; The task dependency analyzer is used to calculate the dependency strength between the sequences of the automation process requirements, and the final sequence planning result of the automation process requirements is determined based on the dependency strength.

8. The method for building an automated process based on natural language capabilities of a multimodal large model according to claim 7, characterized in that: The resource constraint optimization algorithm is calculated as follows: in, is represented as the objective function, T is represented as the scheduling of all automation process requirements, c t represents the scheduling cost of executing the automation process demand, x t Expressed as a binary decision variable, x t =1 means that the schedule t required by the automated process is selected for execution, otherwise it is not executed; st It is represented as resource constraint, K represents the total number of resources, R K It is represented as the scheduling set of automation process requirements that need to occupy resource type k, b k Expressed as the maximum available quantity of resource type k; It is represented as the dependency constraint of the automation process requirement, pred(j) is represented as the set of predecessor automation process requirements of automation process requirement j, and d i It is expressed as the duration or processing time of the automation process requirement i; x t ∈{0,1} represents a variable type constraint, forcing x t is a binary variable.

9. The method for building an automated process based on natural language capabilities of a multimodal large model according to claim 1, characterized in that: In the above step S7, if the automation process is determined and necessary parameter information is missing, the automation process needs to be re-arranged; If user parameters are missing, it is necessary to interact with the user to obtain information.

10. The method for building an automated process based on natural language capabilities of a multimodal large model according to claim 9, characterized in that: The steps of the automated judgment process are as follows: Check parameters: Ensure that the required parameters in the automation process remain intact; Parameter type check: Check all parameter types in the automation process to ensure that all parameters have the same type. Parameter retrieval: When necessary parameter information is missing, the system parameters are automatically retrieved and the automated process is orchestrated again; User parameter detection: When user parameters are missing, an interactive completion interface is triggered for the user parameters, and the user completes the parameters based on the missing parameter information; Create a tracking log: Create a parameter version tracking log based on the information from the parameter detection and inspection.

Citation Information

Cited By

  • Live broadcast script generation method and device, storage medium and program product

    CN121658095A