A method for automatically generating machine vision solutions based on large language models

Abstract

The present invention proposes a method for automatically generating machine vision solutions based on a large language model. Its modules include: a workflow agent, a parameter agent, an algorithm agent, a workflow engine, and a pre-implemented operator library. The present invention solves many problems in the process of building traditional workflow-based machine vision solutions to a great extent, including: workflow construction is highly dependent on the personal experience of engineers, algorithm parameters are performance-sensitive and difficult to adjust, and the application iteration debugging cycle is long. In addition, the method proposed in the present invention has a variety of flexible deployment methods. The method can be deployed in carriers such as customer-site industrial computers, customer-site private cloud servers, and public cloud servers to complete the task of automatically generating machine vision solutions. It greatly improves the solution deployment efficiency of machine vision providers and reduces their personnel costs; it is of great significance to improving the efficiency of the "last mile" of the machine vision industry chain.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision applications, and in particular relates to a method for automatically generating a machine vision solution based on a large language model. Background Art

[0002] Machine vision is a rapidly developing branch of artificial intelligence, aiming to replace the human eye in tasks such as measurement, judgment, and recognition. The core of this technology lies in converting a target into an image signal using an image capture device (such as a CCD or CMOS camera). This signal is then transmitted to a specialized image processing system, which analyzes pixel distribution, brightness, color, and other information to convert it into a digitized signal. The image processing system then performs various operations to extract the target's features, ultimately using these features to make decisions or control the operation of field equipment.

[0003] The maturation of large language model technology in recent years has brought new development opportunities and challenges to machine vision software:

[0004] Currently, the primary users of machine vision software—machine vision solution providers—often need to visit customer sites, consuming significant manpower, resources, and time to build and adjust machine vision solutions. This problem remains unresolved, even with the advanced use of the aforementioned machine vision software. Machine vision engineers still need to use machine vision software to build workflows tailored to customer site requirements and repeatedly adjust and debug them to determine the correct steps and parameters.

[0005] After the machine vision software workflow is deployed at the customer's site, the information hidden behind it is very professional. Customers do not pay attention to the details of the algorithm, so they are often helpless when faced with the machine vision software. They can only contact the solution provider for after-sales service, which is relatively inefficient and increases the customer's cost of using the machine vision solution. Summary of the Invention

[0006] To address these challenges, this paper leverages the latest large language model technology to develop an automated machine vision solution generation technology. This technology addresses the high deployment cycles and personnel costs faced by machine vision solution providers at client sites. Furthermore, it reduces the cost of using machine vision solutions on client sites. By simply describing the on-site problem in human language, an intelligent agent can be invoked to modify current workflow parameters and even re-arrange the workflow. This technology has significant practical implications for improving the efficiency of the "last mile" of machine vision deployment.

[0007] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0008] A system for automatically generating machine vision solutions based on a large language model, including:

[0009] The multi-agent module is used to understand the input human language text based on the pre-prompt words and output the workflow representation and operator implementation code;

[0010] The pre-implemented operator library is used to store manually written pre-implemented operators and operator implementation codes output by the multi-agent module, and select operators from them to form workflows;

[0011] The workflow engine is used to parse the workflow representation output by the multi-agent module, load the operators in the pre-implemented operator library, run the workflow, and when it receives a machine vision application task trigger signal, parse the signal and trigger the corresponding workflow to complete the real-time reasoning of the machine vision application task.

[0012] The multi-agent module comprises:

[0013] The workflow agent is used to understand the input human language text based on the pre-heating prompt words, combine the operators in the pre-implemented operator library based on their specifications and functions, generate a workflow, and output a workflow representation that meets the user's needs and a queue of operators that lack intermediate elements;

[0014] The parameter agent is used to understand and fine-tune the input workflow representation based on the pre-trained prompt words, and output the workflow representation containing the policy that meets the user's desired parameter adjustment;

[0015] The operator agent is used to understand the basic information definition and explanatory description text of the missing operators in the input missing operator queue according to the preheating prompt words, define the description and structure of the missing operators, and generate and output the operator implementation code.

[0016] The pre-implemented operator library includes operator calls for 2D machine vision, 3D machine vision, deep learning, machine learning, reinforcement learning, large language model agents, mathematics, and tool types, among which:

[0017] 2D machine vision operators include image preprocessing, connected region analysis, result post-processing, and measurement submodules;

[0018] 3D machine vision operators include point cloud information processing and coordinate transformation submodules;

[0019] Deep learning operators include image classification, object detection, instance segmentation, semantic segmentation, pose estimation, and motion prediction submodules;

[0020] Machine learning operators include linear regression, random forest, and support vector machine submodules;

[0021] Reinforcement learning operators include Q-learning, policy-based methods, and Actor-Critic submodules;

[0022] Large language model agent operators include RAG, autogen, and conversational large model submodules;

[0023] Mathematical operators include matrix operations and digital definition submodules;

[0024] Tool operators include image acquisition, signal sending, timing, result conversion, and input and output sub-modules.

[0025] The workflow engine includes: application SDK, REST API service, and machine vision application software package.

[0026] The workflow representation is a structured text representation of a workflow, which is used to accurately describe the workflow using text and includes the following elements or objects:

[0027] Global timeout: This is a required field and is used to describe the overall timeout for algorithm execution.

[0028] Task List: This is a required field and is used to describe specific tasks.

[0029] The objects in the task list array must contain the following fields:

[0030] Task Name: This is a required field and is used to describe the task name. In a workflow, the task name cannot be repeated and is guaranteed to be auto-incremented by using a hyphen followed by a number.

[0031] Operator dependency: This is a required field and can be an empty array. It is used to describe other tasks that this task depends on. Fill in the task names of other tasks.

[0032] Operator Name: This is a required field and is used to describe the operator name for the specific execution of the task. The task name is unique in this session and the current generation algorithm, but multiple operator names can be used as needed.

[0033] Operator configuration items: optional, used to describe the configuration of the task and inject configuration items into the operator;

[0034] Asynchronous or not: It is optional and defaults to no. It is used to describe whether it is an asynchronous task.

[0035] Operator execution condition: This is an optional item used to describe the operator execution condition. The condition is given in the form of a key-value pair and supports five condition judgments: >=, <=, >, <, =. The condition value supports four types: int, double, float, and string.

[0036] Operator timeout: This is an optional parameter used to describe the operator timeout.

[0037] The missing operator queue is used to describe the operators and their descriptions that are still missing in the pre-implemented operator library in order to complete the task specified by the current user. It includes the following elements or objects:

[0038] Operator name: used to describe the unique identifier of the operator in the pre-implemented operator library. The operator name reflects its basic function.

[0039] Configuration items: An operator can have multiple configuration items. Each configuration item has a name, type, and initial value. The actual value of the configuration item is given an initial value by the workflow agent in the workflow representation and adjusted by the parameter agent. In the missing operator queue, the workflow agent is required to provide the name, type, and initial value of the configuration item to be added to the operator, and add a human language text explanation for each configuration item.

[0040] Input: An operator can have multiple inputs. Each input has an integer index that increases from 0, a type, and a default value. The actual value of the input cannot be given by any agent. Instead, the workflow engine calculates it in real time based on the data of upstream and downstream operators during inference. In the absence of an operator queue, the workflow agent is required to provide the index, type, and default value of the new operator input and add a human language text explanation for each input.

[0041] Output item: An operator can only have one output item, which has a type attribute. The actual value of the output item is calculated in real time by the workflow engine based on the data of upstream and downstream operators during reasoning. If there is a lack of an operator queue, the workflow agent is required to provide the type attribute of the new operator output item and add a human language text explanation.

[0042] The operator implementation code includes:

[0043] Operator declaration: declare the operator by giving the operator name, input items, output items, and textual explanations of each item;

[0044] Input declaration: A code snippet that declares and instantiates each input item in an operator from an upstream operator;

[0045] Configuration item declaration: a code snippet that declares and instantiates each configuration item in the operator from the workflow representation;

[0046] Result calculation: The code segment that calculates the result based on the actual values of input items and configuration items;

[0047] Output item declaration and return: Return the result obtained by the result calculation code segment.

[0048] A method for automatically generating a machine vision solution based on a large language model includes the following steps:

[0049] The multi-agent module understands the input human language text based on the preheating prompt words and outputs the workflow representation and operator implementation code;

[0050] The pre-implemented operator library stores manually written pre-implemented operators and operator implementation codes output by the multi-agent module, and selects operators from them to form workflows;

[0051] The workflow engine parses the workflow representation output by the multi-agent module, loads the operators in the pre-implemented operator library, and runs the workflow. When it receives a machine vision application task trigger signal, it parses the signal and triggers the corresponding workflow to complete the real-time reasoning of the machine vision application task.

[0052] The multi-agent module performs the following steps:

[0053] The workflow agent understands the input human language text based on the pre-heating prompt words, combines the operators in the pre-implemented operator library based on their specifications and functions, generates a workflow, and outputs a workflow representation that meets the user's needs and a queue of operators that are missing intermediate elements;

[0054] The parameter agent understands and fine-tunes the input workflow representation based on the preheating prompt words, and outputs a workflow representation that contains the policy that meets the user's required parameter adjustments;

[0055] The operator agent understands the basic information definition and explanatory description text of the missing operators in the input missing operator queue based on the preheating prompt words, defines the description and structure of the missing operators, and generates and outputs the operator implementation code.

[0056] The present invention has the following beneficial effects and advantages:

[0057] 1. This invention effectively shortens the average deployment cycle and personnel costs consumed by machine vision solution providers at customer sites.

[0058] 2. This invention effectively reduces the cost of using machine vision solutions for customers.

[0059] 3. The present invention effectively improves the application efficiency of machine vision solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0062] In response to the problems existing in the prior art as analyzed above, the present invention provides a complete set of automatic generation methods for machine vision solutions based on large language models. The algorithm includes Figure 1 Modules and elements shown.

[0063] like Figure 1 The algorithm consists of five modules: the workflow agent, the parameter agent, the operator agent, the pre-implemented operator library, and the workflow engine, as well as three elements: the workflow representation, the missing operator queue, and the operator implementation code. The following sections describe the technical details of these five modules and three elements.

[0064] Module Introduction

[0065] Module 1 Workflow Agent

[0066] The workflow agent is specifically characterized by: preheating prompt words, input link, core model, and output link.

[0067] The warm-up prompt word is a piece of human language text pre-input into the core model conversation; its purpose is to enable the workflow agent to understand its own role and the correct format of its output link in advance; at the same time, it allows the core model to understand the specifications and functions of the operators already in the pre-implemented operator library, and tends to let the core model give priority to selecting operators in it to combine workflows.

[0068] The input link is the human language text input into the core model in real time, and it is also the user's input text in the entire method; its purpose is to enable the workflow agent to understand the specific details of the machine vision application process required by the user.

[0069] The core model, as the core component of the workflow agent, can be any conversational agent on the market; its purpose is to understand the human language in the input link and output a workflow representation that meets the user's needs.

[0070] The output link is the real-time output of the core model. Its purpose is to output the machine vision application generated by the intelligent agent based on its own understanding in the form of an intermediate element workflow representation so that it can be loaded and processed in subsequent steps; at the same time, the output intermediate elements lack the operator queue.

[0071] Module 2 Parameter Agent

[0072] The parameter agent is specifically characterized by: preheating prompt words, input link, core model, and output link.

[0073] The warm-up prompt words are a piece of human language text pre-input into the core model session; the purpose is to let the parameter intelligent agent understand its own role and the correct format of its output link in advance.

[0074] The input link is the representation text input to the core model in real time, and the workflow representation output by the workflow agent is used as input.

[0075] The core model, as the core component of the workflow agent, can be any conversational agent on the market. Its purpose is to understand the workflow representation of the input link, output a strategy that meets the parameter adjustments required by the user, and fine-tune the input workflow representation.

[0076] The output link is the real-time output of the core model. Its purpose is to output the machine vision application generated by the intelligent agent based on its own understanding in the form of a workflow representation so that it can be loaded and processed in subsequent steps.

[0077] Module 3 Operator Agent

[0078] The algorithm agent is specifically characterized by: preheating prompt words, input link, core model, and output link.

[0079] The warm-up prompt words are a piece of human language text pre-input into the core model conversation; the purpose is to let the algorithm agent understand its own role and the correct format of its output in advance.

[0080] The input link is used as a real-time input to the core model representation text, and the basic information definition and explanation description text of the missing operators in the missing operator queue generated in the output link of the module 1 workflow agent are input into the core model.

[0081] The core model, as the core component of the workflow agent, can be any conversational agent on the market. Its purpose is to understand the description and structural definition of the missing operators in the missing operator queue of the input link; and generate the operator implementation in the target programming language code based on this information.

[0082] The output link serves as the real-time output of the core model. Its purpose is to update the missing operator implementations generated by the intelligent agent based on its own understanding to the pre-implementation operator library of module 4, so that it can be called by subsequent links.

[0083] Module 4 Pre-implemented Operator Library

[0084] To enable the workflow agent in Module 1 to more efficiently understand tasks, the present invention provides a library of pre-implemented operators, manually compiled based on industry experience, and allows it to select operators from this library to form workflows. The pre-implemented operator library for machine vision tasks includes, but is not limited to, operator calls for 2D machine vision, 3D machine vision, deep learning, machine learning, reinforcement learning, large language model agents, mathematics, and tools. 2D machine vision operators include modules such as image preprocessing, connected region analysis, result post-processing, and measurement; 3D machine vision operators include sub-modules such as point cloud information processing and coordinate transformation; deep learning operators include sub-modules such as image classification, target detection, instance segmentation, semantic segmentation, posture estimation, and motion prediction; machine learning operators include sub-modules such as linear regression, random forest, and support vector machine; reinforcement learning operators include sub-modules such as Q learning, policy-based methods, and actor-critic; large language model intelligent agent operators include sub-modules such as RAG, autogen, and conversational large models; mathematical operators include sub-modules such as matrix operations and digital definitions; tool operators include sub-modules such as image acquisition, signal transmission, timing, result conversion, input and output. By pre-implementing the operator library itself, it can basically cover the needs of most machine vision solutions on the market. It is worth noting that some machine vision tasks still require users to customize operators based on this to meet special needs, which is also the core significance and value of this invention.

[0085] All operators are implemented in high-level programming languages and can be directly loaded and executed by the workflow engine. Furthermore, the pre-implemented operator library is updated in real time, allowing new operators to be added manually or through the output phase of the operator agent in module 3, serving as pre-implemented operators for the pre-prompt word phase of the workflow agent in module 1.

[0086] Module 5 Workflow Engine

[0087] The workflow engine is the core component responsible for parsing the workflow representation of the parameter agent's final output, loading operators from the pre-implemented operator library, and ultimately executing the workflow. The workflow engine should be flexibly deployable in various formats based on user needs, including but not limited to application SDKs, REST API services, and machine vision application software packages, and deployed in the customer's production environment. When a machine vision application task trigger signal arrives, it parses the signal and triggers the corresponding workflow, completing the real-time reasoning of the machine vision application task.

[0088] Elements Introduction

[0089] Element 1 Workflow Representation

[0090] A workflow representation is a structured text representation of a workflow. This can be implemented using structured text representation formats such as JSON, XML, YAML, and HTML. Its purpose is to accurately describe the workflow using text so that the agent can output the actual tasks it understands as a workflow.

[0091] A workflow typically contains multiple task nodes, each of which is an instance of an operator within the workflow. Operators provide the specific implementation of a task node, including inputs, outputs, configuration options, and implementation code. They can be directly loaded and called by the workflow engine.

[0092] In order to more accurately describe the structure of a workflow and the properties of its operators, a workflow representation must include the following elements or objects:

[0093] Global Timeout: Required. Defaults to 5000 milliseconds, which is the overall timeout for algorithm execution.

[0094] Task list: Required. Add all task descriptions inside. The objects in the array must contain the following fields:

[0095] Task Name: Required. Task names must be unique within a workflow and must be incremented by a hyphen followed by a number.

[0096] Operator dependencies: Required, can be an empty array. Task dependencies, an array. Fill in the names of other tasks that this task depends on.

[0097] Operator Name: Required. This is the name of the operator that executes the task. Task names are unique within this session and the current generation algorithm, but multiple operator names are allowed as needed.

[0098] Operator configuration items: Optional. Task configuration, configuration items injected into the operator.

[0099] Is Asynchronous?: Optional, defaults to No. Whether this is an asynchronous task. Notifies the workflow engine in module 5 to execute this operator asynchronously. This means that the downstream operator will be called immediately after the task is published, without waiting for the operator to return a result.

[0100] Operator execution condition: Optional. The operator execution condition is specified as a key-value pair. Five conditional judgments are supported: >=, <=, >, <, =. Condition values can be int, double, float, and string.

[0101] Operator timeout: Optional, defaults to 500 milliseconds, the operator timeout.

[0102] Element 2: Lack of operator queue

[0103] The missing operator queue, one of the outputs generated by the workflow agent in Module 1, describes the operators and their descriptions that are missing from the pre-implemented operator library to complete the current user-specified task. The list of these operators and their descriptions is the missing operator queue.

[0104] The description of an operator must include at least the following elements or objects:

[0105] Operator name: The unique identifier of the operator in the pre-implemented operator library. The name of the operator should generally reflect its basic function.

[0106] Configuration Items: An operator can have multiple configuration items, each with a name, type, and initial value. The actual values of these configuration items can be assigned initial values by the workflow agent in the workflow representation and adjusted by the parameter agent. In the missing operator queue, the workflow agent is required to provide the name, type, and initial value of the configuration item for the operator to be added, and to add a human-language text explanation for each configuration item.

[0107] Inputs: An operator can have multiple inputs, each with an integer index starting at 0 and increasing in value, a type, and a default value. Unlike configuration items, the actual value of an input cannot be specified by any agent. Instead, it is calculated in real time by the workflow engine during inference based on data from upstream and downstream operators. In the absence of an operator queue, the workflow agent is required to provide the index, type, and default value of the operator input to be added. A human-language textual explanation is also provided for each input.

[0108] Output: An operator can have only one output, which has a type attribute. Similar to inputs, the output's value is calculated in real time by the workflow engine during inference based on data from upstream and downstream operators. In the absence of an operator queue, the workflow agent is required to provide the type attribute of the operator output to be added, along with a human-language textual explanation.

[0109] Element 3 operator implementation code

[0110] The operator implementation code is the output generated by the operator agent and is a high-level language code that can be directly parsed and loaded by the workflow engine. It must include the following parts:

[0111] Operator declaration: declare the operator based on the given operator name, input items, output items, text explanations, and other information.

[0112] Input declaration: A code snippet that declares and instantiates each input item in an operator from an upstream operator.

[0113] Configuration item declaration: A code snippet that declares and instantiates each configuration item in an operator from a workflow representation.

[0114] Result calculation: The code segment that calculates the result based on the actual values of input items and configuration items.

[0115] Output item declaration and return: Return the result obtained by the result calculation code segment.

[0116] Introduction to module implementation method

[0117] Modules 1-3: Workflow Agent, Parameter Agent, Operator Agent

[0118] All intelligent agents use the "Wenxin Yiyan" product launched by Baidu.

[0119] Module 4 Pre-implemented Operator Library

[0120] To maximize the efficiency of as many operators as possible, we've created a library of pre-implemented operators written in C / C++. These operators include operator types and some other types. The pre-implemented operator library contains 152 operators.

[0121] Module 5 Workflow Engine

[0122] The workflow engine responsible for concatenating operators is implemented in Python to achieve better workflow representation scalability and support operator diversity.

[0123] 2.2 Introduction to element implementation methods

[0124] Elements 1-2 workflow representation, lack of operator queue

[0125] Workflows are represented using HTML structured text, primarily for the convenience of front-end rendering during application. Operator queues are described in simplified Chinese.

[0126] Element 3 operator implementation code

[0127] The operator agent uses Python to write the implementation code of the missing operators to minimize the difficulty of understanding the core model.

Claims

1. A system for automatically generating machine vision solutions based on a large language model, characterized in that: include: The multi-agent module is used to understand the input human language text based on the pre-prompt words and output the workflow representation and operator implementation code; The pre-implemented operator library is used to store manually written pre-implemented operators and operator implementation codes output by the multi-agent module, and select operators from them to form workflows; The workflow engine is used to parse the workflow representation output by the multi-agent module, load operators from the pre-implemented operator library, and run the workflow. When it receives a trigger signal for a machine vision application task, it parses the signal and triggers the corresponding workflow to complete the real-time reasoning of the machine vision application task. The multi-agent module comprises: The workflow agent is used to understand the input human language text based on the pre-heating prompt words, combine the operators in the pre-implemented operator library based on their specifications and functions, generate a workflow, and output a workflow representation that meets the user's needs and a queue of operators that lack intermediate elements; A parameter agent, which is used to understand and fine-tune the input workflow representation based on the pre-trained prompt words, and output a workflow representation containing a policy that meets the user's desired parameter adjustments; The operator agent is used to understand the basic information definition and explanatory description text of the missing operators in the input missing operator queue based on the preheating prompt words, define the description and structure of the missing operators, and generate and output the operator implementation code; The pre-implemented operator library includes operator calls for 2D machine vision, 3D machine vision, deep learning, machine learning, reinforcement learning, large language model agents, mathematics, and tool types, among which: 2D machine vision operators include image preprocessing, connected region analysis, result post-processing, and measurement submodules; 3D machine vision operators include point cloud information processing and coordinate conversion submodules; Deep learning operators include image classification, object detection, instance segmentation, semantic segmentation, pose estimation, and motion prediction submodules; Machine learning operators include linear regression, random forest, and support vector machine submodules; Reinforcement learning operators include Q-learning, policy-based methods, and Actor-Critic submodules; Large language model agent operators include RAG, autogen, and conversational large model submodules; Mathematical operators include matrix operations and digital definition submodules; Tool operators include image acquisition, signal sending, timing, result conversion, and input and output sub-modules.

2. The automatic generation system of machine vision solutions based on a large language model according to claim 1, characterized in that: The workflow engine includes: application SDK, REST API service, and machine vision application software package.

3. The automatic generation system of machine vision solutions based on a large language model according to claim 1, characterized in that: The workflow representation is a structured text representation of a workflow, which is used to accurately describe the workflow using text and includes the following elements or objects: Global timeout: This is a required field and is used to describe the overall timeout for algorithm execution. Task List: This is a required field and is used to describe specific tasks.

4. The automatic generation system of machine vision solutions based on a large language model according to claim 3 is characterized in that: The objects in the task list array must contain the following fields: Task Name: This is a required field and is used to describe the task name. In a workflow, the task name cannot be repeated and is guaranteed to be auto-incremented by using a hyphen followed by a number. Operator dependency: This is a required field and can be an empty array. It is used to describe other tasks that this task depends on. Fill in the task names of other tasks. Operator Name: This is a required field and is used to describe the operator name for the specific execution of the task. The task name is unique in this session and the current generation algorithm, but multiple operator names can be used as needed. Operator configuration items: optional, used to describe the configuration of the task and inject configuration items into the operator; Asynchronous or not: It is optional and defaults to no. It is used to describe whether it is an asynchronous task. Operator execution condition: This is an optional item used to describe the operator execution condition. The condition is given in the form of a key-value pair and supports five condition judgments: >=, <=, >, <, =. The condition value supports four types: int, double, float, and string. Operator timeout: This is an optional parameter used to describe the operator timeout.

5. The automatic generation system of machine vision solutions based on a large language model according to claim 1, characterized in that: The missing operator queue is used to describe the operators and their descriptions that are still missing in the pre-implemented operator library in order to complete the task specified by the current user. It includes the following elements or objects: Operator name: used to describe the unique identifier of the operator in the pre-implemented operator library. The operator name reflects its basic function. Configuration items: An operator can have multiple configuration items. Each configuration item has a name, type, and initial value. The actual value of the configuration item is given an initial value by the workflow agent in the workflow representation and adjusted by the parameter agent. In the missing operator queue, the workflow agent is required to provide the name, type, and initial value of the configuration item to be added to the operator, and add a human language text explanation for each configuration item. Input: An operator can have multiple inputs. Each input has an integer index that increases from 0, a type, and a default value. The actual value of the input cannot be given by any agent. Instead, the workflow engine calculates it in real time based on the data of upstream and downstream operators during inference. In the absence of an operator queue, the workflow agent is required to provide the index, type, and default value of the new operator input and add a human language text explanation for each input. Output item: An operator can only have one output item, which has a type attribute. The actual value of the output item is calculated in real time by the workflow engine based on the data of upstream and downstream operators during reasoning. If there is a lack of an operator queue, the workflow agent is required to provide the type attribute of the new operator output item and add a human language text explanation.

6. The automatic generation system of machine vision solutions based on a large language model according to claim 1, characterized in that: The operator implementation code includes: Operator declaration: declare the operator by giving the operator name, input items, output items, and textual explanations of each item; Input declaration: A code snippet that declares and instantiates each input item in an operator from an upstream operator; Configuration item declaration: a code snippet that declares and instantiates each configuration item in the operator from the workflow representation; Result calculation: The code segment that calculates the result based on the actual values of input items and configuration items; Output item declaration and return: Return the result obtained by the result calculation code segment.

7. A method for automatically generating a machine vision solution based on a large language model, applied to the system for automatically generating a machine vision solution based on a large language model according to claim 1, characterized in that: The following steps are involved: The multi-agent module understands the input human language text based on the preheating prompt words and outputs the workflow representation and operator implementation code; The pre-implemented operator library stores manually written pre-implemented operators and operator implementation codes output by the multi-agent module, and selects operators from them to form workflows; The workflow engine parses the workflow representation output by the multi-agent module, loads the operators in the pre-implemented operator library, and runs the workflow. When it receives a machine vision application task trigger signal, it parses the signal and triggers the corresponding workflow to complete the real-time reasoning of the machine vision application task.

8. The method for automatically generating a machine vision solution based on a large language model according to claim 7, characterized in that: The multi-agent module performs the following steps: The workflow agent understands the input human language text based on the pre-heating prompt words, combines the operators in the pre-implemented operator library based on their specifications and functions, generates a workflow, and outputs a workflow representation that meets the user's needs and a queue of operators that are missing intermediate elements; The parameter agent understands and fine-tunes the input workflow representation based on the preheating prompt words, and outputs a workflow representation that contains the policy that meets the user's required parameter adjustments; The operator agent understands the basic information definition and explanatory description text of the missing operators in the input missing operator queue based on the preheating prompt words, defines the description and structure of the missing operators, and generates and outputs the operator implementation code.

Images