Large and small model collaborative robot autonomous assembly method and system

Abstract

The invention discloses a large and small model collaborative robot autonomous assembly method and system, and belongs to the field of advanced manufacturing technology intellectualization. The problems that traditional robot assembly depends on manual programming, adaptability is poor, and an algorithm cannot be optimized autonomously are solved. The method comprises the following steps: receiving an assembly project request submitted by a user in a natural language; the request is decoupled into a structured action primitive through an arrangement agent, and tasks are allocated; an algorithm design agent retrieves and adapts an algorithm from an open source library based on the action primitive, and completes training in a sandbox environment; the verification agent is responsible for automatic data acquisition and labeling, and carrying out overall performance test and iterative optimization on an integrated algorithm in a simulation environment; and finally generating an executable field control script. According to the method, the problems of algorithm autonomous design, simulation verification and closed-loop optimization when the robot faces a complex assembly task are solved, the development threshold and the deployment risk are reduced, and the intelligent level and the flexible adaptive capacity of an assembly system are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of advanced manufacturing technology and intelligent information technology, specifically relating to a robot autonomous assembly method and system for collaborative large and small models. Background Technology

[0002] With the deepening of the Industry 5.0 concept, manufacturing models are accelerating their transformation from large-scale mass production to flexible production with multiple varieties and small batches. The market demand for product personalization and customization is growing, requiring robotic assembly systems to have the ability to quickly respond to new tasks and dynamically adapt to environmental changes. However, existing robotic assembly technologies still have many bottlenecks: On the one hand, traditional robot control systems rely on predefined fixed programs and adopt a hard-coded mode. When faced with new assembly tasks or environmental fluctuations, professional engineers need to spend a lot of time refactoring the code and debugging parameters. The development cycle is usually several days or even weeks long, resulting in low response efficiency and difficulty in meeting the needs of agile manufacturing. On the other hand, although some technologies attempt to introduce large language models, they are only used as task planners, limited to calling existing algorithm library interfaces, and lack the ability to independently design and mutate new algorithms for specific assembly problems.

[0003] Meanwhile, data acquisition and annotation in industrial assembly scenarios face a dual dilemma: the industrial environment is complex and ever-changing, high-quality labeled data is scarce, and manual annotation is costly and inefficient, making it difficult to cold-start the algorithm training data. Furthermore, existing technologies lack effective simulation verification mechanisms; directly debugging algorithms on physical robots easily leads to risks such as collisions and hardware damage, resulting in extremely high on-site trial-and-error costs. Moreover, the vast amount of open-source algorithm resources has not been efficiently reused, with most solutions still tending to be developed from scratch, resulting in wasted R&D resources and difficulty in guaranteeing the advanced performance of the algorithms. These problems collectively lead to insufficient adaptability and robustness of existing robotic assembly systems in complex dynamic scenarios, making it difficult to meet the core requirements of Industry 5.0 flexible production. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a robot autonomous assembly method and system with large and small model collaboration to address the shortcomings of the prior art. This method and system solve the technical problem that robots cannot autonomously design, simulate and iteratively optimize new algorithms when facing complex assembly tasks, thereby enhancing the adaptability and robustness of robots in complex scenarios such as assembly and improving assembly efficiency.

[0005] The present invention adopts the following technical solution: A method for autonomous robot assembly with coordinated large and small models includes the following steps: S1. Receive the user's assembly project request, wherein the assembly project request includes the request subject and the assembly target; S2. Analyze the assembly engineering request obtained in step S1, decouple the assembly target into executable action primitives, allocate tasks according to the capabilities of the algorithm design agent and the verification agent, and obtain the algorithm design action primitives allocated to the algorithm design agent, the data acquisition task and the algorithm verification task allocated to the verification agent. S3. Analyze the algorithm design action primitives obtained in step S2, query the local knowledge base and remote algorithm open repositories, and filter to obtain suitable open source algorithms; if there is no training dataset suitable for the open source algorithm locally, delegate the data collection task to the verification agent and receive the labeled dataset fed back by the verification agent; write training and testing code based on the open source algorithm and labeled dataset, execute the training and testing process in the algorithm running sandbox and adjust the training hyperparameters until the verification index of the action primitive is met, and obtain the algorithm weights and corresponding weight addresses; S4. Receive the data acquisition task obtained in step S3, complete the acquisition and labeling of the dataset, obtain the labeled dataset and feed it back to the algorithm design agent; receive the algorithm weights and related input and output information, and combine them with the execution flow prompt corresponding to the action primitive obtained in step S2 to perform overall performance testing on the open source algorithm in the simulation environment and obtain the test results; S5. Receive the test results obtained in step S4, combine them with the algorithm logic corresponding to the algorithm weights obtained in step S3, generate a field execution script, control the robot to execute the field execution script, and obtain the assembly task completion result.

[0006] Preferably, in step S2, the decoupling of the action primitives and task allocation specifically include: S201. Perform task analysis on the assembly project request, clarify the core requirements of the requesting subject and assembly target, and break down the assembly target into hierarchical sub-tasks based on your own knowledge, clarify the algorithm / model requirements corresponding to each sub-task, and generate a task prompt. S202. Combine the task prompt with the assembly process request, and output an action primitive containing the target agent, input information, output information and verification indicators. The action primitive is returned in JSON format.

[0007] Preferably, the JSON-formatted action primitives include primitive action identifiers, target intelligent agents, task descriptions, input information, output information, and acceptance metrics.

[0008] Preferably, the hierarchical subtasks include at least one of target detection, pose estimation, and path planning.

[0009] Preferably, in step S3, the selection of suitable open-source algorithms specifically includes: The algorithm design class action primitives are analyzed, and search instructions are generated in combination with the engineering background. The MCP server is started, and the search_model tool is called to search for open source algorithms in the remote algorithm open repository with the search instructions as tags. At the same time, the get_model_card tool is called to obtain the documentation corresponding to the searched open source algorithms. Based on the documentation, open source algorithms suitable for industrial scenarios are obtained through analysis and screening.

[0010] Preferably, in step S3, when the data collection task is assigned to the verification agent, humans are simultaneously entrusted to collect data in the actual environment, while the verification agent collects data in a virtual environment. The labeled dataset is composed of data collected in the virtual environment and data collected in the actual environment.

[0011] Preferably, in step S3, the algorithm running sandbox is isolated from the server, and the training and testing processes are executed independently in the algorithm running sandbox.

[0012] Preferably, in step S4, the completion of dataset collection and annotation specifically includes: The SAM3 pre-trained model is called to generate a data acquisition script, which segments the target object and calculates the corresponding bounding box in a natural language manner. After checking and verifying the segmentation and calculation results, the labeled dataset is formed.

[0013] Preferably, in step S4, if the test result does not meet the preset standard, a failure reason is generated and fed back to the algorithm design agent. The algorithm design agent then performs algorithm iteration and update based on the failure reason. The iterated algorithm returns to step S4 to retest the overall performance.

[0014] Secondly, embodiments of the present invention provide a robot autonomous assembly system for collaborative large and small models, comprising: The orchestration module is used to receive the user's assembly engineering request, analyze the assembly engineering request and decouple the assembly target into executable action primitives, allocate tasks according to the capabilities of the design module and the verification module, generate the execution flow prompt corresponding to the action primitives, and generate the on-site execution script after receiving the test results and combining the algorithm logic. The design module is used to receive algorithm design class action primitives assigned by the orchestration module, query the local knowledge base and remote algorithm open repository and select suitable open source algorithms, delegate data collection tasks to the verification module, write training and testing code after receiving the labeled dataset, execute the training and testing process in the algorithm running sandbox and adjust the training hyperparameters until the verification indicators are met and output the algorithm weights and corresponding weight addresses. The verification module is used to receive data collection tasks assigned by the design module, complete the collection and labeling of the dataset and feed it back to the design module, receive the algorithm weights and related input and output information, and perform overall performance testing of the algorithm in the simulation environment in conjunction with the execution flow prompt and output the test results. The storage module is used to store local knowledge bases, open source algorithms retrieved from remote algorithm open repositories, labeled datasets, algorithm weights, training and testing code, and on-site execution scripts; The execution module is used to receive the field execution script generated by the orchestration module and control the robot to execute the field execution script to complete the assembly task.

[0015] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described robot autonomous assembly method for coordinated large and small models.

[0016] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described robot autonomous assembly method for coordinating large and small models.

[0017] Fifthly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described robot autonomous assembly method for coordinated large and small models.

[0018] In a sixth aspect, embodiments of the present invention provide an electronic device, including a computer program, wherein when the computer program is executed by the electronic device, it implements the steps of the above-described robot autonomous assembly method for coordinating large and small models.

[0019] Compared with the prior art, the present invention has at least the following beneficial effects: A multi-agent collaborative robot assembly method based on large and small models seamlessly integrates previously fragmented tasks such as task understanding, algorithm development, testing, and deployment. Through collaboration among agents and a closed-loop data flow, it forms an intelligent system capable of self-iterative optimization. Addressing the challenges of high programming barriers, heavy reliance on expert experience, long algorithm development cycles, and difficulty adapting to diverse industrial scenarios in traditional robot assembly, this method establishes a closed-loop automated workflow of "orchestration-design-verification" based on the multi-agent collaborative approach of large language models. End-to-end autonomy from natural language to on-site execution is achieved through a request-decoupling-search / mutation-simulation-deployment approach. Complex robot assembly tasks are transformed into executable and verifiable atomic action primitives, breaking down the barriers between algorithm design and practical deployment, significantly reducing the trial-and-error costs of assembly processes, and demonstrating a high level of intelligence and practical application value. Through the division of labor and collaboration among multiple agents, it effectively integrates and utilizes massive open-source algorithm resources, and iterative verification in a simulation environment ensures the robustness of the assembly strategy.

[0020] Furthermore, by breaking down the assembly target into subtasks and generating prompts containing the target agent, inputs, outputs, and verification metrics, a clear basis is provided for subsequent algorithm matching and task execution. The output of action primitives in JSON format ensures the consistency and compatibility of interactions between agents, avoiding ambiguity in information transmission. This hierarchical task planning effectively solves the context loss problem of long, complex tasks, making the goals, requirements, and executing entities of each subtask clear and explicit, significantly improving the system's ability to understand complex instructions and its execution accuracy.

[0021] Furthermore, the explicit definition of primitive action identifiers, target agents, and task descriptions ensures unique targeting and traceability in task transmission between agents. The inclusion of acceptance metrics provides quantitative standards for algorithm design and verification, avoiding the problem of vague compliance. The structured JSON format facilitates rapid parsing and processing by agents, improving the efficiency of task allocation and execution, while ensuring unbiased information transmission at each stage. This provides data interaction assurance for the efficient operation of the entire collaborative system, enabling multi-agent collaboration to have a unified information interaction standard.

[0022] Furthermore, by using these typical tasks as primitives, the system's capability boundaries are clearly defined, enabling it to directly cover the key links of perception, cognition, decision-making, and execution in the assembly process, thus ensuring the professionalism and effectiveness of the method for industrial assembly scenarios.

[0023] Furthermore, by calling the `search_model` and `get_model_card` tools through the MCP server and generating search instructions based on the engineering background, the system ensures that the selected open-source algorithms are suitable for industrial scenarios. This search-analysis-screening process avoids compatibility issues caused by blindly reusing open-source algorithms, while also saving the time and cost of developing algorithms from scratch. Leveraging the vast resources of remote algorithm open repositories allows the system to quickly acquire high-quality algorithms, and the analysis of documentation ensures the industrial applicability of the algorithms, significantly shortening the algorithm development cycle.

[0024] Furthermore, the study validates that the intelligent agent collects data in a virtual environment, while humans supplement it in the actual environment. The resulting labeled dataset combines scalability and realism. Virtual environment data collection enables large-scale generation and reduces the cost of manual data collection. The supplementation of data from the actual environment ensures the dataset's scene adaptability and avoids the problem of virtual data becoming disconnected from the real-world scenario. This hybrid dataset model improves the generalization ability of model training and effectively solves the challenge of cold start with industrial data.

[0025] Furthermore, the isolation design prevents potential anomalies during training from impacting the server's core system, reducing the risk of system crashes. Simultaneously, the independent sandbox environment ensures that the training and testing processes are unaffected by other tasks, guaranteeing the accuracy of hyperparameter tuning and algorithm verification. It also avoids training result deviations caused by external factors, providing a stable and reliable environment for algorithm training and ensuring the effectiveness of algorithm weights.

[0026] Furthermore, by calling the SAM3 pre-trained model to generate the acquisition script, the target objects are segmented using natural language and bounding boxes are calculated, significantly improving the efficiency of data annotation and avoiding the tediousness and errors of manual annotation. The built-in checking and verification process further ensures the accuracy of the labeled data and guarantees the quality of the dataset. This not only reduces labor costs but also improves the speed and accuracy of data generation, providing high-quality data support for rapid algorithm training and solving the problems of low efficiency, high cost, and large errors in traditional manual annotation.

[0027] Furthermore, when test results fail to meet standards, the reasons for the failure are fed back, prompting the algorithm design agent to iterate and update. After each iteration, the algorithm is retested, forming a closed loop of design-testing-optimization. This mechanism can promptly identify algorithm defects and make targeted improvements, preventing substandard algorithms from entering the field execution phase. Through multiple iterations, the algorithm's performance continuously approaches the verification indicators, ensuring the high accuracy and reliability of the final algorithm and effectively mitigating the risks of collisions and hardware damage during field assembly.

[0028] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0029] In summary, this invention constructs an autonomous algorithm design and simulation system for robot assembly based on a multi-agent collaborative architecture. Through in-depth analysis of natural language instructions and intelligent utilization of open-source algorithm resources, it transforms the originally high-threshold development process that relies on manual labor into a human-like autonomous design and verification process. Furthermore, by utilizing automated data generation and closed-loop verification in a virtual simulation environment, it solves the problems of difficult data acquisition and high deployment risks in industrial scenarios, significantly improving the flexibility and intelligence of robot assembly lines.

[0030] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0031] Figure 1 This is an overall flowchart of the present invention; Figure 2 This is an information flow diagram of the present invention; Figure 3 This is a flowchart illustrating an example of the present invention; Figure 4 This is a diagram showing the task analysis results of the orchestration agent of the present invention; Figure 5 A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 6 This is a block diagram of a chip provided according to an embodiment of the present invention.

[0032] Among them, 60. Computer equipment; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0035] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0036] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0037] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0038] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0039] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0040] The robot autonomous assembly system based on a large language model (LLM) constructed in this invention is centered on a multi-agent collaborative framework consisting of an Orchestrator Agent, an Algorithm Design Agent, and a Validation Agent. These three agents are all built upon a large language model (LLM) and are assigned specific roles, capabilities, and behavioral patterns through carefully designed prompts. They communicate and collaborate through structured data flows (primarily JSON-formatted action primitives and information) to jointly complete a closed-loop task from natural language instructions to physical execution.

[0041] The orchestration agent acts as the system's brain and commander-in-chief, responsible for interacting with users and receiving and deeply analyzing high-level assembly task requirements. Its core capability is to decompose and plan fuzzy, unstructured natural language descriptions into a series of precise, executable, and verifiable action primitives, and is responsible for task scheduling and result aggregation.

[0042] Algorithm design agent: As the creator of the system, it is responsible for autonomously finding, adapting, and even creating the required algorithm models based on the requirements of action primitives. It has the ability to access and utilize massive open-source algorithm knowledge bases (such as Hugging Face, GitHub), and can fine-tune and train algorithms according to specific scenarios.

[0043] Verification Agent: As the system's safety officer and quality inspector, it undertakes two key responsibilities: First, it automatically synthesizes high-quality labeled training data in a virtual simulation environment according to requirements; second, before deployment, it integrates and performs end-to-end testing and verification of all algorithms produced by the algorithm design agent in the simulation environment to ensure that its overall performance meets the requirements, forming a reinforcement learning closed loop of design-verification-feedback.

[0044] This invention provides a robot autonomous assembly method based on the collaboration of large and small models. It transforms the traditional offline assembly development mode, which relies on expert experience and manual programming, into an online autonomous design mode centered on multi-agent collaboration. By standardizing and decoupling natural language instructions, retrieving and mutating open-source algorithms, and automatically labeling synthetic data, the method enables the autonomous construction of assembly process algorithms. This transforms assembly requirements from fuzzy natural language to high-precision executable code and from general open-source models to industrial-specific strategies. It also establishes a closed-loop feedback mechanism between the virtual simulation environment and the physical assembly site. This reduces the threshold for algorithm development and the risk of trial and error on site while enabling the efficient reuse of massive open-source algorithm resources. Furthermore, it employs a verification agent to quickly generate the assembly control process and performs full-process simulation verification and iterative optimization.

[0045] Please see Figure 1This invention discloses a robot autonomous assembly method based on the coordinated use of large and small models, which is based on a carefully designed... prompt By defining different agent roles within a large model, and starting from user needs, the system works closely together with agents through orchestration, algorithm design, and validation. Based on a knowledge base and open-source algorithm libraries, it automatically searches for solutions to relevant problems. The detailed workflow of the system is as follows: Figure 2 As shown, the specific steps are as follows: S1. The overall process of the system is based on the user's specific engineering request. The engineering request needs to clearly define the subject of the request and the final goal, so that the orchestration agent can quickly analyze the user's intent. Users submit assembly engineering requests through a natural language interface. These requests must clearly define the requesting entity and the assembly objective. Upon receiving the request, the orchestration agent first performs semantic understanding, extracting key entities, actions, constraints, and the final goal.

[0046] S2. After the orchestration agent receives the user request from step S1, it first analyzes the request, clarifies all subjects and goals, decouples the goal into executable action primitives, and allocates tasks according to the capabilities of the algorithm-designed and validated agents. The information flow is as follows: Figure 2 As shown, the specific case process is as follows: Figure 3 As shown, the details are as follows: S201. First, the intelligent system analyzes the request task. After clarifying the request subject and goal, it combines its own knowledge to divide the user request goal into executable primitive modules. It then plans the steps and algorithms / models required to complete the task hierarchically and outputs them in prompt form. Figure 3 Taking the user assembly request in the example, the generated prompt is as follows: Figure 4 As shown; Task planning and Prompt generation: For each sub-skill, the orchestration agent further plans the algorithm or model type required for its implementation and generates a structured "task prompt". This prompt may be described as: "Step 1: Requires an object detection model, with camera images as input and a 2D bounding box of a red gear as output, with an acceptance metric of accuracy ≥ 98%".

[0047] S202. Input the prompt from step S201 and the user request instruction from step S1 together to the orchestration agent. The orchestration agent will output the action primitives to complete the step according to the capabilities of the algorithm design agent and the verification agent, and return them in JSON format. The returned content must include the target agent, input, output, and verification metric.

[0048] Taking the action primitive "object detection calculation design" as an example, its complete JSON action primitives at each level are as follows: [ { "Primitive Action": 1, "Intelligent Agent": "Algorithm Design Agent", Task Description: Object Detection Model Input: " ", Output: " <boundingbox>", "Acceptance Criteria": "<precision≥98%> " } ] Assembly tasks are complex and unstructured. By using an orchestration agent, the user's fuzzy natural language requirements are decoupled into standardized "action primitives," providing a standardized interface and structure for subsequent algorithm matching, code generation, and task acceptance. Through this hierarchical task planning, the macro-level assembly goal is broken down into specific sub-tasks such as target detection, pose estimation, and path planning. The input, output, and acceptance criteria for each sub-task are clearly defined, effectively solving the problems of context loss and logical confusion in long and complex tasks, and improving the system's ability to understand and execute complex instructions.

[0049] S3. The algorithm design intelligence receives all action primitives related to algorithm design from step S2. By querying the local knowledge base and remote algorithm open repository, it searches for algorithms that can implement the current action primitive. Based on direct reuse and algorithm mutation, it designs the optimal algorithm to complete the action of the primitive. The specific steps are as follows: S301. After receiving the action primitive from step S2, the algorithm design agent will analyze the action primitive, combine the input, output and acceptance criteria information of the action primitive with the engineering background, and output the best retrieval instruction. Taking object detection in action primitive 1 as an example, the analysis result is that "action primitive 1 relies on a highly robust and rapidly transferable object detection model, which needs to be plug-and-play in diverse industrial scenarios." For primitive action 2 (piston 6D pose estimation): the analysis results indicate that a high-precision 6D pose estimation algorithm is required, which supports the fusion calculation based on RGB-D images and CAD models, and is adapted to the characteristics of smooth piston surfaces and few textures. The search command is 6D pose estimation RGB-D image CAD fusion high precision industrial parts. For primitive action 3 (assembly path planning): the analysis result is that a collaborative robot obstacle avoidance path planning algorithm is needed, which satisfies joint motion constraints, generates a smooth trajectory, and balances assembly efficiency and safety. The search commands are collaborative robot, obstacle avoidance path planning, joint constraints, smooth trajectory, assembly scenario.

[0050] S302. After the task is defined, the algorithm design agent will automatically start the MCP server, call the search_model tool of HuggingfaceMCP, and retrieve the ten most downloaded object detection models with the label "object detection"; In addition, the get_model_card tool is used to search for the corresponding README document of the model, analyze and recommend the most suitable target detection model for industrial scenarios; Action 1: Search results: Obtain the top ten object detection models by download volume, including YOLOv8, Faster R-CNN, EfficientDet, etc. Use the get_model_card tool to obtain the README documents of each model, analyze their industrial scene adaptation cases, inference speed and accuracy indicators, and finally select YOLOv8 (inference speed 45ms / frame, industrial scene detection accuracy 97.8%) as the base algorithm; The search results for the primitive action 2 were: Colmap, PoseCNN, PVNet and other 6D pose estimation algorithms were found. Based on the analysis of the documentation, the PVNet algorithm supports the fusion of RGB-D images and CAD models. The pose estimation error of industrial parts is ≤0.008mm. It is suitable for parts with little texture. Therefore, PVNet was selected as the basic algorithm. Primitive Action 3 Search Results: Path planning algorithms such as RRT-Connect, A*, and Informed RRT were selected. Informed RRT The algorithm performs better in obstacle avoidance efficiency and trajectory smoothness, and supports input of robot joint constraint parameters, so it was selected as the base algorithm.

[0051] S303. Based on the recommended optimal algorithm and the README reference document, write the training and testing code; If there is no dataset for this scenario on the local machine, the data collection task is delegated to the verification agent and the human again in the form of action primitives, so that they can automatically collect and label data in the virtual environment and the real environment respectively. Training code for YOLOv8: The input image resolution is set to 1920×1080, the batch size is 16, the SGD optimizer is used, and the initial learning rate is 0.01; the test code includes functions such as accuracy calculation (mAP@0.5) and inference time statistics. The training code for PVNet integrates an RGB-D data reading interface, sets the CAD model matching threshold to 0.95, and performs 5000 training iterations. The test code focuses on verifying whether the pose estimation error meets the requirement of ≤0.01mm. For the code in Informed RRT *: Import the DH parameters and joint limit range (rotation angle of each joint ±180°) of the UR5 robot, set the path point interval to 0.01m, and the objective function is "shortest trajectory length + least execution time".

[0052] S304. After data collection is completed, the training and testing process is executed in an algorithm sandbox isolated from the server; and the training hyperparameters are continuously adjusted according to the training results to meet the detection index requirements of the action primitives. YOLOv8 training: The learning rate was kept at 0.01 for the first 100 iterations, decayed to 0.001 for 100-200 iterations, and decayed to 0.0001 for 200-300 iterations. After 300 iterations, the initial detection accuracy was 96.2%, which did not meet the acceptance metric of 98%. The batch size was adjusted to 32, and a Mosaic data augmentation strategy was added. After another 200 iterations, the detection accuracy improved to 98.5%, and the inference time was 42ms, meeting the acceptance metric. PVNet Training: After initial training, the pose estimation error was 0.012mm. By increasing the depth image weight coefficient (adjusted from 0.3 to 0.5) and optimizing the loss function, after 4000 iterations of training, the pose estimation error was reduced to 0.009mm, which met the requirements. Informed RRT *Debugging: The initially generated path contained 8 collision points. By adjusting the obstacle expansion coefficient (from 0.02m to 0.03m), the replanned path had no collisions, and the execution time was calculated to be 35 seconds, which meets the requirement of ≤40 seconds.

[0053] S305. After training is complete, save the model weights and return success and the corresponding weight address.

[0054] After all algorithm training and testing are successful, the algorithm design agent saves the YOLOv8 weight file (approximately 28MB), the PVNet weight file (approximately 45MB), and the Informed RRT * configuration file to the local storage path " / home / project / assembly_algorithms / weight / ", and outputs a success signal and the corresponding weight address to the orchestration agent.

[0055] To address the inefficiency of developing algorithms from scratch in industrial scenarios, the algorithm design agent introduces a Model Context Protocol (MCP), enabling seamless integration with external open-source knowledge bases such as Huggingface and GitHub. Through automatic querying, document reading, and filtering recommendations, the system prioritizes the reuse of existing high-quality algorithms / models and then modifies them to adapt to specific tasks. This "reuse + fine-tuning" mechanism avoids reinventing the wheel, significantly shortens the algorithm development cycle, and ensures the advanced nature of the technical solutions used.

[0056] S4. The verification agent mainly performs two types of tasks: data collection and delegation tasks from the algorithm design agent, and verification tasks for the algorithms / models designed and trained by the algorithm design agent. The specific steps are as follows: S401. Taking action primitive 1 as an example, the algorithm design agent, based on the target detection task, will delegate the task to a verification agent to collect the category and location information of objects in the environment, and specify a detailed data collection format; the detailed task delegation action primitive... json The format is as follows: [ { "primitive action": n, "Intelligent Agent": "Validation Agent", Task Description: Dataset collection for object dection Output format: " <yolo>", "quantity": " <200> " } ] After receiving the task assigned by the algorithm design agent, the verification agent will carefully analyze the task description and data collection format, automatically call the SAM3 pre-trained model, write scripts, segment the corresponding objects in natural language, calculate the corresponding bounding boxes, and hand them over to the verification agent for inspection. After completing the inspection and verification, the verification agent is then... json The format sends a message to the algorithm design agent indicating that the dataset has been successfully constructed. This message includes a success signal and the dataset storage address. The format is as follows: [ { Status: "success" "Save location": " / home / project / dataset / " } ] S402. After the algorithm-designed intelligent agent has completed the training and testing of all action primitive-related models, it will... json The format for sending all algorithm / model information to the validating agent is as follows: [ { "Object detection algorithm weight address": " / home / project / object_detection / weight"; "enter": , Output: <boundingbox>, }, { "Pose estimation algorithm weight address": " / home / project / pose_estimation / weight"; "enter": <depth> <mask> <cad> Output: <6D pose> }, ... ] After receiving all the algorithm information designed by the algorithm design agent, the verification agent takes the execution flow prompt generated by the orchestration agent in step S2 as input to the verification agent, instructing the verification agent to write a flow to orchestrate and call all the algorithms in order to test their overall performance in the simulation environment. To address the pain points of scarce training data and high costs of manual annotation, the validation agent integrates the advanced SAM3 model with a simulation environment. Image data is automatically acquired in the virtual environment, and SAM3 is used for high-precision automatic segmentation and bounding box calculation, automatically generating labeled training datasets. This not only achieves full automation of data acquisition and annotation but also significantly improves the model's generalization ability in real-world environments through pre-training on virtual data, solving the cold start problem in industrial data.

[0057] S403. After completing all tests, the verification agent will organize all test results. If the test passes, it will be executed on-site. If the test fails, the reason for the failure will be sent to the algorithm design agent for iterative update of the algorithm.

[0058] To address the pain points of scarce training data and high costs of manual annotation, the validation agent integrates the advanced SAM3 model with a simulation environment. Image data is automatically acquired in the virtual environment, and SAM3 is used for high-precision automatic segmentation and bounding box calculation, automatically generating a labeled training dataset. This method not only achieves full automation of data acquisition and annotation but also significantly improves the model's generalization ability in real-world environments through pre-training on virtual data, solving the cold start problem in industrial data.

[0059] S5. Upon receiving a signal indicating successful testing of the verification agent, the orchestration agent will write an on-site execution script based on the successful experience of the verification agent to complete the assembly task.

[0060] After receiving the test success signal and detailed data from 100 simulation tests from the verification agent, the orchestration agent combines the algorithm design with the optimal algorithm logic of the agent and writes the on-site execution script. The script is written in Python and includes a communication interface with the UR5 robot (based on TCP / IP protocol), an algorithm weight loading module, a real-time data processing module, and an exception handling module. Communication interface: Set the robot IP address to 192.168.1.100, the communication baud rate to 115200, and the data transmission delay to ≤10ms; Anomaly handling module: If target detection fails (no valid coordinates for 3 consecutive frames), trigger a reshoot; if the assembly force exceeds 50N, immediately stop the action and return to a safe position; Precision calibration module: Every 10 assembly tasks are performed, the vision system is automatically invoked to detect the fit gap. If the deviation exceeds 0.005mm, the pose estimation algorithm parameters are automatically adjusted.

[0061] The on-site execution script is sent to the control system of the UR5 robot, and the robot performs the assembly actions according to the script: First, the cylinder image is acquired through the vision system and the cylinder position is located by the YOLOv8 algorithm; then the camera posture is adjusted to acquire piston data and the 6D pose of the piston is output by the PVNet algorithm; next, the path planning trajectory is loaded and the end effector moves according to the trajectory to grasp the piston; finally, the piston and cylinder are assembled, and the data of the whole process is fed back to the system backend in real time.

[0062] Through on-site trial production verification, the method of this invention has achieved efficient and precise assembly of the 4G15 engine cylinder block and piston, with an average assembly cycle of 53 seconds per piece, a fit clearance of ≤0.02mm, and a one-time assembly qualification rate of 99.3%, which is a significant improvement over the traditional manual programming scheme (assembly cycle of 85 seconds per piece, qualification rate of 90%).

[0063] This invention establishes a rigorous design-simulation-feedback closed-loop verification mechanism. The verification agent performs end-to-end testing on the code generated by the algorithm design agent in a simulation environment. Only after all indicators (such as accuracy and success rate) meet the acceptance criteria is the algorithm allowed to be deployed to the field robot for execution. If the test fails, the system automatically feeds back the error information to the algorithm design agent for hyperparameter adjustment or code rewriting. This iterative optimization mechanism based on the simulation environment effectively avoids the collision risks and hardware damage that may result from direct debugging on the physical robot, ensuring the safety and reliability of field deployment.

[0064] By orchestrating the analysis and utilization of historical knowledge by intelligent agents, the system possesses the ability to continuously learn and accumulate knowledge. As the number of tasks executed increases, the system can accumulate more successful action primitives and assembly processes. When faced with similar new tasks, it can more quickly retrieve and reuse historical solutions, realizing the self-evolution and quality improvement of assembly process knowledge, and reducing the redundant consumption of computing resources.

[0065] Unlike traditional large-scale model applications that rely on predefined interfaces and static tool libraries for function calling, this invention goes beyond passively selecting existing functions and empowers the system with the ability to "create tools." Through the collaborative work of orchestrating agents, algorithm design agents, and verification agents, a fully automated closed-loop system is constructed, encompassing "intent decoupling - algorithm search and mutation - data synthesis and training - simulation verification - on-site execution." This invention not only enables the dynamic expansion and real-time generation of robot assembly skills but also solves the last-mile problem from algorithm design to physical deployment through an interactive feedback mechanism between simulation and reality.

[0066] In another embodiment of the present invention, a robot autonomous assembly system with coordinated large and small models is provided. This system can be used to implement the above-mentioned robot autonomous assembly method with coordinated large and small models. Specifically, the robot autonomous assembly system with coordinated large and small models includes an orchestration module, a design module, a verification module, a storage module, and an execution module.

[0067] The orchestration module is used to receive the user's assembly engineering request, analyze the assembly engineering request and decouple the assembly target into executable action primitives, allocate tasks according to the capabilities of the design module and the verification module, generate the execution flow prompt corresponding to the action primitives, and generate the on-site execution script after receiving the test results and combining the algorithm logic. The design module is used to receive algorithm design class action primitives assigned by the orchestration module, query the local knowledge base and remote algorithm open repository and select suitable open source algorithms, delegate data collection tasks to the verification module, write training and testing code after receiving the labeled dataset, execute the training and testing process in the algorithm running sandbox and adjust the training hyperparameters until the verification indicators are met and output the algorithm weights and corresponding weight addresses. The verification module is used to receive data collection tasks assigned by the design module, complete the collection and labeling of the dataset and feed it back to the design module, receive the algorithm weights and related input and output information, and perform overall performance testing of the algorithm in the simulation environment in conjunction with the execution flow prompt and output the test results. The storage module is used to store local knowledge bases, open source algorithms retrieved from remote algorithm open repositories, labeled datasets, algorithm weights, training and testing code, and on-site execution scripts; The execution module is used to receive the field execution script generated by the orchestration module and control the robot to execute the field execution script to complete the assembly task.

[0068] This invention provides a terminal device comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or function. The processor described in this embodiment can be used in the operation of a robot autonomous assembly method for collaborative large and small models, including: The system receives an assembly project request from a user, which includes a request subject and an assembly target. It analyzes the assembly project request, decoupling the assembly target into executable action primitives. Based on the capabilities of the algorithm design agent and the verification agent, tasks are allocated, resulting in algorithm design action primitives assigned to the algorithm design agent, data acquisition tasks assigned to the verification agent, and algorithm verification tasks. The system analyzes the algorithm design action primitives, queries the local knowledge base and remote algorithm open repositories, and selects suitable open-source algorithms. If no suitable training dataset for the open-source algorithm is available locally, a data acquisition task is delegated to the verification agent, and the system receives the labeled dataset returned by the verification agent. Based on the open-source algorithm and the labeled dataset... Training and testing code is written, and the training and testing process is executed in the algorithm running sandbox, adjusting the training hyperparameters until the verification indicators of the action primitives are met, obtaining the algorithm weights and their corresponding weight addresses; the data acquisition task is received, the dataset is collected and labeled, the labeled dataset is obtained, and fed back to the algorithm design agent; the algorithm weights and related input / output information are received, and combined with the execution prompt corresponding to the action primitives, the overall performance of the open-source algorithm is tested in a simulation environment, obtaining the test results; the test results are received, and combined with the algorithm logic corresponding to the algorithm weights, a field execution script is generated, and the robot is controlled to execute the field execution script to obtain the assembly task completion result.

[0069] Please see Figure 5 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the robot autonomous assembly method for large and small model collaboration in this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the robot autonomous assembly system for large and small model collaboration in this embodiment. To avoid repetition, these details are not elaborated here.

[0070] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 5 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0071] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0072] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or memory of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0073] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0074] Please see Figure 6 The terminal device is an electronic device 600, which is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0075] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0076] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0077] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0078] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0079] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem). This communication can be performed via input / output interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network, wide area network, and / or public network, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0080] Example 4 This invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). More specific examples of the computer-readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical fiber, portable compact disk read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0081] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0082] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0083] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the robot autonomous assembly method for large and small model collaboration in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps: The system receives an assembly project request from a user, which includes a request subject and an assembly target. It analyzes the assembly project request, decoupling the assembly target into executable action primitives. Based on the capabilities of the algorithm design agent and the verification agent, tasks are allocated, resulting in algorithm design action primitives assigned to the algorithm design agent, data acquisition tasks assigned to the verification agent, and algorithm verification tasks. The system analyzes the algorithm design action primitives, queries the local knowledge base and remote algorithm open repositories, and selects suitable open-source algorithms. If no suitable training dataset for the open-source algorithm is available locally, a data acquisition task is delegated to the verification agent, and the system receives the labeled dataset returned by the verification agent. Based on the open-source algorithm and the labeled dataset... Training and testing code is written, and the training and testing process is executed in the algorithm running sandbox, adjusting the training hyperparameters until the verification indicators of the action primitives are met, obtaining the algorithm weights and their corresponding weight addresses; the data acquisition task is received, the dataset is collected and labeled, the labeled dataset is obtained, and fed back to the algorithm design agent; the algorithm weights and related input / output information are received, and combined with the execution prompt corresponding to the action primitives, the overall performance of the open-source algorithm is tested in a simulation environment, obtaining the test results; the test results are received, and combined with the algorithm logic corresponding to the algorithm weights, a field execution script is generated, and the robot is controlled to execute the field execution script to obtain the assembly task completion result.

[0084] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0085] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0086] The engine block and piston assembly task at an automotive parts factory was selected as the test scenario. This task involves multiple sub-tasks such as target detection, pose estimation, and path planning. Traditional assembly methods require manual programming and debugging, which is poorly adaptable to batch and model changes. The method of this invention was compared with the traditional manual programming method, and the following data were obtained:

[0087] One hundred different types of assembly tasks (covering multiple scenarios such as electronic components and mechanical structures) were constructed in a virtual simulation environment to verify the universality of the method of this invention: Action primitive decoupling accuracy: 98.5%, ensuring that the subtask division is accurately matched with the capabilities of the intelligent agent.

[0088] Open source algorithm retrieval fit rate: 97% of the time, the optimal algorithm suitable for industrial scenarios was successfully selected.

[0089] Simulation verification confirms consistency with field execution: With a 99.2% success rate, algorithms that pass simulations can be directly applied in the field.

[0090] Algorithm iterative optimization efficiency: The average iteration time is reduced by 60%, rapidly improving algorithm performance.

[0091] Experimental data show that the method of this invention significantly shortens the development cycle of new tasks and model adaptation, reduces data annotation costs and on-site debugging risks, and significantly improves the assembly success rate by autonomously decoupling tasks, reusing open-source algorithms, and closed-loop simulation verification. It also eliminates reliance on manual labor, realizes full autonomy and flexible adaptation of the assembly process, effectively solves the limitations of traditional solutions in complex dynamic scenarios, and is suitable for flexible production needs of multiple varieties and small batches.

[0092] In summary, this invention presents a robot autonomous assembly method and system based on large and small model collaboration. It transforms the traditional robot assembly development model, which relies on manual programming and expert experience, into a fully automated closed-loop autonomous design model driven by multiple intelligent agents. Through three core technologies—natural language decoupling, intelligent retrieval and reuse of open-source algorithms, and virtual environment simulation verification and iterative optimization—this invention effectively solves the problem of robots being unable to autonomously design, verify, and optimize dedicated algorithms when facing complex and unstructured assembly tasks. This method significantly reduces the technical threshold and time cost of algorithm development, greatly reduces the risks of on-site debugging and hardware wear and tear through pre-simulation verification, and overcomes the bottleneck of data scarcity in industrial scenarios by utilizing automated data synthesis. Ultimately, this invention endows the robot assembly system with high adaptability and robustness, enabling it to quickly respond to changing flexible production demands and achieving a dual improvement in assembly efficiency and intelligence.

[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0094] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0095] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0096] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0099] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random-access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0100] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.< / cad> < / mask> < / depth> < / boundingbox> < / yolo> < / boundingbox>

Claims

1. A method for autonomous robot assembly using coordinated large and small models, characterized in that, Includes the following steps: S1. Receive the user's assembly project request, wherein the assembly project request includes the request subject and the assembly target; S2. Analyze the assembly engineering request obtained in step S1, decouple the assembly target into executable action primitives, allocate tasks according to the capabilities of the algorithm design agent and the verification agent, and obtain the algorithm design action primitives allocated to the algorithm design agent, the data acquisition task and the algorithm verification task allocated to the verification agent. S3. Analyze the algorithm design class action primitives obtained in step S2, query the local knowledge base and remote algorithm open repository, and select suitable open source algorithms. If there is no training dataset adapted to the open-source algorithm locally, a data collection task is delegated to the verification agent, and the labeled dataset fed back by the verification agent is received. Based on the open-source algorithm and labeled dataset, training and testing code was written. The training and testing process was executed in the algorithm running sandbox and the training hyperparameters were adjusted until the verification index of the action primitive was met, and the algorithm weights and corresponding weight addresses were obtained. S4. Verify that the intelligent agent receives the data acquisition task obtained in step S3, completes the acquisition and labeling of the dataset, obtains the labeled dataset, and feeds it back to the algorithm design intelligent agent; receive the algorithm weights and related input and output information, and combine them with the execution flow prompt corresponding to the action primitive obtained in step S2 to perform overall performance testing of the open source algorithm in the simulation environment and obtain the test results. S5. Receive the test results obtained in step S4, combine them with the algorithm logic corresponding to the algorithm weights obtained in step S3, generate a field execution script, control the robot to execute the field execution script, and obtain the assembly task completion result.

2. The robot autonomous assembly method based on the collaboration of large and small models according to claim 1, characterized in that, In step S2, the decoupling of the action primitives and task allocation specifically include: S201. Perform task analysis on the assembly project request, clarify the core requirements of the requesting subject and assembly target, and break down the assembly target into hierarchical sub-tasks based on your own knowledge, clarify the algorithm / model requirements corresponding to each sub-task, and generate a task prompt. S202. Combine the task prompt with the assembly process request, and output an action primitive containing the target agent, input information, output information and verification indicators. The action primitive is returned in JSON format.

3. The robot autonomous assembly method based on the collaboration of large and small models according to claim 2, characterized in that, The action primitives in JSON format include primitive action identifiers, target intelligent agents, task descriptions, input information, output information, and acceptance metrics.

4. The robot autonomous assembly method based on the collaboration of large and small models according to claim 2, characterized in that, The hierarchical subtasks include at least one of object detection, pose estimation, and path planning.

5. The robot autonomous assembly method based on the collaboration of large and small models according to claim 1, characterized in that, In step S3, the selection of suitable open-source algorithms specifically includes: The algorithm design class action primitives are analyzed, and search instructions are generated in combination with the engineering background. The MCP server is started, and the search_model tool is called to search for open source algorithms in the remote algorithm open repository with the search instructions as tags. At the same time, the get_model_card tool is called to obtain the documentation corresponding to the searched open source algorithms. Based on the documentation, open source algorithms suitable for industrial scenarios are obtained through analysis and screening.

6. The robot autonomous assembly method based on the collaboration of large and small models according to claim 1, characterized in that, In step S3, when the data collection task is assigned to the verification agent, humans are simultaneously entrusted to collect data in the actual environment, while the verification agent collects data in the virtual environment. The labeled dataset is composed of data collected in the virtual environment and data collected in the actual environment.

7. The robot autonomous assembly method based on the collaboration of large and small models according to claim 1, characterized in that, In step S3, the algorithm running sandbox is isolated from the server, and the training and testing processes are executed independently in the algorithm running sandbox.

8. The robot autonomous assembly method based on the collaboration of large and small models according to claim 1, characterized in that, Step S4, specifically the completion of dataset collection and annotation, includes: The SAM3 pre-trained model is called to generate a data acquisition script, which segments the target object and calculates the corresponding bounding box in a natural language manner. After checking and verifying the segmentation and calculation results, the labeled dataset is formed.

9. The robot autonomous assembly method based on the collaboration of large and small models according to claim 1, characterized in that, In step S4, if the test result does not meet the preset standard, a failure reason is generated and fed back to the algorithm design agent. The algorithm design agent then iterates and updates the algorithm based on the failure reason. The iterated algorithm returns to step S4 to retest the overall performance.

10. A robot autonomous assembly system for collaborative large and small models, characterized in that, include: The orchestration module is used to receive the user's assembly engineering request, analyze the assembly engineering request and decouple the assembly target into executable action primitives, allocate tasks according to the capabilities of the design module and the verification module, generate the execution flow prompt corresponding to the action primitives, and generate the on-site execution script after receiving the test results and combining the algorithm logic. The design module is used to receive algorithm design class action primitives assigned by the orchestration module, query the local knowledge base and remote algorithm open repository and select suitable open source algorithms, delegate data collection tasks to the verification module, write training and testing code after receiving the labeled dataset, execute the training and testing process in the algorithm running sandbox and adjust the training hyperparameters until the verification indicators are met and output the algorithm weights and corresponding weight addresses. The verification module is used to receive data collection tasks assigned by the design module, complete the collection and labeling of the dataset and feed it back to the design module, receive the algorithm weights and related input and output information, and perform overall performance testing of the algorithm in the simulation environment in conjunction with the execution flow prompt and output the test results. The storage module is used to store local knowledge bases, open source algorithms retrieved from remote algorithm open repositories, labeled datasets, algorithm weights, training and testing code, and on-site execution scripts; The execution module is used to receive the field execution script generated by the orchestration module and control the robot to execute the field execution script to complete the assembly task.

Images