Intelligent mechanical arm control system scheme based on multi-model cooperation

Abstract

The invention provides an intelligent mechanical arm control system scheme based on multi-model cooperation, and belongs to the field of automation and intelligent control. The system is divided into four parts of man-machine interaction, detection and perception, task decoupling planning and program generation, and robot execution operation. Man-machine interaction is carried out, a user inputs an instruction in a text or voice format, and instruction processing is carried out through voice recognition and a natural language translation model; detecting and sensing, and identifying and positioning a target captured by the camera by using a multi-modal large model; task decoupling planning and program generation: analyzing the user instruction and the perception information into an executable program by applying a task planning model and a program translation model; and checking and correcting the program by using the program checking model, and driving the mechanical arm to complete a task. According to the system scheme, variable environments can be dealt with, complex language instructions can be understood, intelligent control over the mechanical arm is achieved, and the multi-task execution capacity and the intelligent level of the mechanical arm are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of robotics, and more particularly to a multi-model collaborative intelligent robotic arm control system solution, which belongs to the field of automation and intelligent control. The application scope of the present invention includes intelligent operation of robotic arms and autonomous decision-making and control in complex environments, and is particularly suitable for the field of intelligent robot control that requires multifunctional robotic arm operation. Background Art

[0002] In recent years, with the rapid development of Industry 4.0, intelligent robotic arms, as an important part of robotics technology, have been widely used in industrial manufacturing, logistics sorting, medical assistance, and service robots. The intelligent control capability of robotic arms directly determines their efficiency and reliability in different application scenarios. In existing technologies, the control of robotic arms usually relies on the following mainstream methods:

[0003] 1. Control method based on trajectory planning: Trajectory planning technology is the basis of robot arm control. Traditional methods mostly use preset trajectories and implement task execution through servo control. However, fixed trajectories cannot adapt to task requirements in dynamic environments, such as dynamic object grasping or path temporary obstacle avoidance, resulting in insufficient flexibility and adaptability.

[0004] 2. Real-time control method based on visual sensor feedback: With the development of computer vision technology, visual feedback control has gradually become one of the mainstream technologies. This method uses visual sensors to obtain environmental information, generates real-time control instructions through image processing and calculation, and combines traditional trajectory planning and other algorithms to control the robot. However, the real-time and robustness of visual feedback technology are limited by computing resources and algorithm complexity, and there are significant limitations when dealing with scenes such as occlusion and light changes.

[0005] Although the above technologies have made significant progress in their respective fields, there are still the following technical bottlenecks in practical applications:

[0006] 1. Insufficient environmental adaptability: Existing robotic arm control systems are obviously insufficient in dealing with unstructured environments and complex dynamic scenarios. For example, the mission scenarios of service robots are often highly diverse and uncertain, such as various operating supplies, complex and changing service scenarios, etc., which puts extremely high demands on the environmental perception and motion planning of the robotic arm.

[0007] 2. Limited executable tasks: Currently, mainstream robotic arm control systems are controllers based on fixed logic programming. In particular, robotic arms on industrial production lines can only execute a pre-compiled program to achieve a repetitive trajectory and operation process. Some service robots or robotic arms on the market simply increase the complexity of the program and predefine more operation processes. This is far from enough to achieve the level of embodied intelligence. They cannot complete tasks beyond the predefined ones and do not have generalization capabilities.

[0008] 3. Intelligence needs to be improved: Especially in terms of human-computer interaction, the robot's understanding of human commands is still based on predefined instructions and predefined task libraries. In other words, the robot does not understand human commands, but simply matches the received instructions with the predefined operation library. In order to respond to a variety of human language commands, it is necessary to continuously enrich the predefined content, and such a system has limited generalization ability for human commands.

[0009] In order to solve the above problems, it is proposed to introduce a collaborative mechanism of multiple models, use the multimodal model to understand the characteristics of image information, improve the system's adaptability to complex environments; use the universal language model's ability to understand human language to complete the task planning of the robotic arm, significantly improve the types of tasks that the robotic arm system can perform and its generalization ability, and at the same time improve the intelligence level of the system. However, there is currently a lack of a complete intelligent robotic arm control system solution with multi-model collaborative control.

[0010] Based on this, the present invention proposes a multi-model collaborative intelligent robotic arm control system solution. This solution integrates multiple technologies such as trajectory planning, visual feedback, and natural language processing with a large model solution as the core to form a multi-model collaborative robotic arm control system, providing a comprehensive and intelligent solution for the robotic arm control system to solve complex tasks. Summary of the invention

[0011] The technical problem to be solved by the present invention is to provide an intelligent robotic arm control system solution based on multi-model collaboration in response to the above-mentioned deficiencies in the prior art, which can cope with complex and diverse service environments, understand the complex and obscure natural language commands issued by users, and operate the robotic arm to meet user needs.

[0012] In order to solve the above technical problems, the technical solution adopted by the present invention is: modular design system, dividing the system into four parts: human-computer interaction instruction reception, detection and perception, task decoupling planning and program generation, and execution operation. And introduce large model technology in each module, and propose corresponding technical implementation plans. The system solution specifically refers to dividing the system framework design into the following 4 steps:

[0013] Step 1: Collect the natural language of user voice and text input, process the user input with a large model solution, and form a specific text format in English;

[0014] Step 2: Based on the large model solution, the visual detection and target positioning solution processes the images collected by the camera sensor, identifies the target operation object of the positioning system, and returns its corresponding posture information; based on the large model solution, the environmental information understanding and perception converts the image returned by the camera into a text describing the environmental information;

[0015] Step 3: Task decoupling and planning based on the large model solution: According to the English text information of the user language instruction from step 1 and the target posture information returned from step 2, task decoupling and planning are performed; based on the large model solution, the planned tasks are converted into executable machine language codes so that they can be directly used for the control and execution of the robot arm;

[0016] Step 4: Based on the large model solution, review, correct and optimize the variable execution program designed in step 3, and then compile and run it to control the robot arm to perform a series of operations that are accurate and meet user needs.

[0017] First, in step 1, the natural language of user voice and text input is collected, and the user input is processed by the big model solution to form a specific English text format. The specific steps are: (1) Apply the pre-trained natural language translation model to uniformly translate the text of any language input by the user into English text; (2) For the user's voice input, use the pre-trained speech-to-text model as the core of speech recognition, collect the voice instructions issued by the user, convert them into text through the speech-to-text model, and then uniformly translate them into English text by the language translation model; (3) Input the obtained English text into the pre-trained format conversion big model, which has the ability of semantic parsing, can deeply deconstruct the text structure, identify and parse the core predicate-object structure in the instruction, and then convert it into a highly structured task expression form, such as "Get-apple" and "Put-plate", so as to optimize the parsing ability of the instruction interpretation and task planning big model in the subsequent step 3, thereby improving the robot arm's accurate understanding of user intentions and the intelligent level of task execution.

[0018] In step 2, the visual detection and target positioning solution based on the large model solution has the following specific steps: the depth camera is used as the perception sensor, and the pre-trained recognition and positioning multimodal large model is applied to the RGB image returned by the sensor for target recognition. The model has the ability to identify the target category and target pixel coordinates in the RGB image, and can determine whether the target in the user command exists:

[0019] (1) If the target exists, the pixel coordinates of the target on the image are output. The coordinates of the target in the camera coordinate system are calculated by combining the depth image and camera parameters returned by the camera. The coordinates of the target in the robot coordinate system are solved by combining the camera and the robot arm kinematics and output in the following format:

[0020]

[0021] The complete process of identifying the existence of the target object and obtaining the target coordinates is encapsulated as a perception API, and its calling format is "detect('xxx')", which can directly return whether xxx exists and its coordinates in the robot arm coordinate system.

[0022] In addition, in step 2, the understanding and perception of environmental information based on the large model solution is as follows: the pre-trained multimodal large model of environmental description is used to understand the RGB image returned by the camera. The model has the following capabilities:

[0023] Object recognition: identifying the types of objects in the environment and their locations;

[0024] Spatial relational reasoning: understanding the spatial relationships between objects, such as "the apple is on the table" and "the plate is near the apple";

[0025] Environment description generation: Generate readable environment description text based on the detection results to support subsequent task planning, such as "There is an apple on the table. There is a plate around the apple.";

[0026] The purpose of describing environmental information is to provide an objective basis for subsequent task decoupling and planning, so as to avoid subsequent task planning that does not conform to the objective environment or errors, causing the robotic arm to perform incorrect actions. For example, if the planned task requires "grabbing a plate", but the plate is not detected in the environmental information, an error is returned to prevent the robotic arm from performing invalid actions; if the target object is obscured or invisible, a request can be made to adjust the camera angle and perform additional perception operations.

[0027] In step 3, the task decoupling and planning based on the large model solution is carried out. The specific steps are as follows: the English description text of the working environment information returned from step 2 and the English text of the user requirement instructions are synchronously integrated into the task information text to be parsed:

[0028] {

[0029] "Environment description":"There is an apple on the table. There is aplate around the apple.",

[0030] "User instruction":"Get—apple.Put—plate."

[0031] }

[0032] Based on the above information, task parsing and structural processing are performed: the pre-trained instruction interpretation and task planning model understands and infers the user's intention in the task information text, reasoning and planning the tasks and arranges them in a logical order, enriches the details of complex and obscure tasks, and splits them into n simple execution tasks; for example, for the task of "placing an apple on a plate", the model needs to parse the operation action (grasping, placing), the object (apple, plate) and the execution order, and then generate a structured task sequence:

[0033] Object detection: Identify the world coordinates of a given object in a scene, “1. Get the spatial coordinates of the apple.”

[0034] Object grabbing: Perform grabbing actions based on the detection results, "2.Picking apples."

[0035] Target positioning: Determine the location information of the target to be placed, "3. Get the spatial coordinates of the plate."

[0036] Object placement: Complete the placement operation, "4.Put it on the plate."

[0037] The task items obtained by the instruction interpretation and planning model are then passed on to the trained program translation model for task-by-task program translation.

[0038] In step 3, based on the big model solution, the planned task is converted into executable machine language code so that it can be directly used for the control and execution of the robot arm. The specific steps are as follows: First, the system receives the list-based task description generated by the task planning module. The description is based on user instructions, environmental perception data and task objectives, including but not limited to operations such as grabbing, placing, moving, and rotating. Since the natural language task description cannot be directly used for robot control, the system needs to rely on the pre-trained translation big model to convert the task text into a compilable and runnable control program: In order to ensure that the generated program is compatible with the specific robot arm control system, the translation big model needs to combine the following two core APIs for code conversion:

[0039] Perception API (Step 2: Get the world coordinates of the target), which is used to call sensors or computer vision models to obtain the coordinates of the target in the environment, such as detect('apple') returns the location of the apple;

[0040] Robot arm control API (program interface for compiling and running in step 3), used to call robot arm motion control instructions, such as get(pos_apple) to perform grabbing operations;

[0041] The translation model converts structured tasks into executable program text based on this:

[0042] pos_apple = detect('apple') # Get the location of the apple

[0043] get(pos_apple)#Grab the apple

[0044] pos_plate = detect('plate') # Get the position of the plate

[0045] put(pos_plate)#Place the apple on the plate

[0046] This outputs machine language that conforms to the robot arm control logic and can be executed by a computer.

[0047] In step 4, based on the large model solution, the variable execution program designed in step 3 is reviewed, corrected, and optimized, and then compiled and run to control the robot arm to perform a series of operations that are accurate and meet user needs. The specific steps are as follows:

[0048] First, in each system execution cycle, the robot arm control program text generated by step 3 will be dynamically adjusted to adapt to environmental variables, task changes and execution strategy optimization. Due to the changes in the context and environmental perception results of different tasks, program logic may deviate and even cause potential risk of task execution failure. Therefore, this step uses a pre-trained multi-level code review and optimization model to comprehensively review the control program and correct and enhance it in multiple dimensions, including:

[0049] (1) Syntax and logic consistency check: The control program is analyzed at the lexical, grammatical and semantic levels to ensure that the code conforms to the syntax specifications of the target programming language and avoids common errors such as undefined variables, type mismatch, loop exceptions, and array out-of-bounds errors. In addition, the model further checks the logical consistency between instructions to ensure the context of program execution. For example, the "grab an object" operation cannot be executed when the "detection target does not exist" condition.

[0050] (2) Constraint and safety verification: Safety checks are performed on the control program, including but not limited to the robot arm's kinematic constraints, joint angle limits, collision detection, and torque limits, to ensure that the robot arm will not cause equipment damage or safety accidents due to trajectory calculation errors, redundant movements, or exceeding physical limits during operation. At the same time, the system performs context dependency analysis on each executed instruction to ensure that the preconditions of the task are met to avoid abnormal execution, such as the robot arm failing to complete the "grasping" task before "releasing the object";

[0051] (3) Task optimization and resource scheduling: Intelligently optimize the task sequence of the control program to maximize execution efficiency and minimize resource consumption; for example, reduce the idle motion of the robot arm through dynamic reordering, optimize path planning to reduce execution time, and adjust the grasping angle to improve operation stability; in addition, the system can adjust the execution plan according to task priority and real-time feedback information to adapt to environmental changes or temporary needs of users;

[0052] (4) Redundant code detection and program simplification: Redundant instructions, unnecessary loops, and repeatedly called APIs or variables that may exist in the control program are trimmed to improve code execution efficiency. For example, if the "detect object" operation has been executed in the previous frame and returned valid coordinates, there is no need to call the detection API again in the subsequent frame, thereby reducing computing overhead and improving overall response speed.

[0053] The control program that has undergone the above-mentioned multi-level optimization and verification will be compiled and deployed to the robotic arm controller to achieve precise control of the robotic arm, ensure that it completes the operation tasks according to the user's intentions, improve the stability, intelligence level and safety of execution, and thus meet the refined needs in complex application scenarios.

[0054] Compared with the existing technology, the intelligent robotic arm control system designed in this scheme improves the robotic arm's ability to cope with complex environments and its ability to understand complex and obscure natural language instructions, thereby realizing complex and intelligent robotic arm control and improving the robotic arm's multi-tasking execution capability and intelligence level. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A structural diagram of an intelligent robotic arm control system solution based on multi-model collaboration provided by an embodiment of the present invention;

[0056] Figure 2 A structural diagram of a modular technical solution provided by an embodiment of the present invention;

[0057] Figure 3 A flow chart of a human-computer interaction module provided by an embodiment of the present invention;

[0058] Figure 4 A structural diagram of a detection and perception module provided in an embodiment of the present invention;

[0059] Figure 5 A flowchart of a task decoupling planning and program generation module provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0060] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Special note: The system solution of the present invention has no special requirements for the selection of models, mechanical arms, cameras and other hardware models. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0061] like Figure 1 As shown, this embodiment provides an intelligent robotic arm control system solution based on multi-model collaboration, which is described in detail as follows.

[0062] The main function of the intelligent robot system of this embodiment is to control the movement of the machine according to the demand instructions put forward by the user and the environmental information of the robot's current environment, so as to meet the current needs of the user. The system hardware carrier built in this embodiment consists of a six-axis robot arm and its end effector gripper, a computer, a keyboard, a microphone, a depth camera, a network cable and a switch.

[0063] The user sends instructions to the robot system through the keyboard or microphone. After receiving the instructions, the system will identify, plan and make decisions on the task, and process and perceive the images returned by the visual sensor depth camera. It will collect information from multiple dimensions and issue operation instructions to the robotic arm actuator to control the robotic arm to complete movement and target operations, thereby meeting user needs.

[0064] The modular system design solution in this case is as follows: Figure 2 As shown in the figure, the system is divided into: human-computer interaction module, detection and perception module, task decoupling planning and program generation module, and execution operation module. A large model is introduced in the first three modules, and corresponding technical solutions are proposed.

[0065] In the human-computer interaction module, this embodiment uses the open source speech-to-text model Whisper-small as the speech-to-text model. For the language translation model, the open source large language model Gemma-2B is selected to complete the unified conversion of languages ​​into English text format. Both models are implemented in a local offline deployment manner. In the detection and perception module, this embodiment uses the open source multimodal large model Paligemma2-3b-pt-224 as the technical core of the detection module, and is implemented in a local offline deployment model. In the task decoupling planning and program generation module, it is deployed online by calling the Gemini1.5-flash API. The execution operation module is composed of the robot arm and its end effector gripper motion control program API.

[0066] This embodiment designs the system according to a modular solution, and the specific implementation details of each module are introduced below.

[0067] For the human-computer interaction module, its processing flow is as follows: Figure 3 As shown in the figure, the user inputs user commands to the robot control system through the keyboard or microphone, which can be input in text or voice format. After receiving the user input information, the human-computer interaction module first determines the information format: when the user inputs non-text, the voice file is passed to the Whisper-small model for recognition and transcription into text format; if the user input command is in text format, no transcription is performed. The obtained text format command is handed over to the Gemma-2B model fine-tuned by Prompt for text translation. For some common languages, the Gemma-2B pre-trained model already has the ability to translate them into English. The module outputs the obtained English text format command and passes it to the task decoupling planning and program generation module for subsequent processing.

[0068] For the detection and perception module, its processing flow is as follows: Figure 4As shown, the system can obtain images of the robot's physical environment through the sensor depth camera. The images include RGB color images and depth images containing distance information. The RGB image returned by the sensor is passed to the multimodal large model Paligemma2-3b-pt-224 for target recognition, and the pixel coordinate values ​​of the object on the image are determined. Combined with the depth image and camera parameters returned by the camera, the coordinate values ​​of the target in the camera coordinate system are calculated. The camera is fixed to the end of the robotic arm to form a camera-robotic arm system with "eyes in the hand", and hand-eye calibration is performed. Combined with the calibrated hand-eye matrix and the end position of the robotic arm, the coordinates of the target in the world coordinate system are solved according to the kinematics of the robotic arm, and the complete calculation process for obtaining the target world coordinates is encapsulated as a perception module API.

[0069] In addition, the detection and perception module also includes the application of the multimodal large model Palignemma2-3b-pt-224 to describe the environmental information. The RGB image captured by the camera is passed into the Palignemma2-3b-pt-224 multimodal large model. The model makes a text description of the robot's operating environment based on the image. Finally, the detection and perception module will return a text description of the environmental information.

[0070] Regardless of the target world coordinate system or the text description of the environmental information obtained by the detection and perception module, they will be passed to the task decoupling planning and program generation module for subsequent processing. Figure 5 As shown in the figure, the task decoupling planning and program generation module receives the English instruction text output from the human interaction module and the environmental information text from the perception module, and passes the integrated text information to the decoupling and planning model. The model is deployed in the form of fine-tuning online calling Gemini1.5-flash API and fine-tuning with a certain format of prompt, so that it can be based on general knowledge and ability. Figure 5 The output is in the form shown in , that is, the planned n task set texts are output.

[0071] The n planned task set texts are further passed into the machine language translation model, which is also deployed in the form of fine-tuning the online call to the Gemini1.5-flash API, and fine-tuned with the perception API of the target world coordinates obtained in the perception module and the robotic arm and its end effector gripper motion control API pre-packaged in the execution module. The purpose is to make the executable program generated by the machine language translation model conform to the predefined program API format and be compiled and run by the execution module. After the n task set texts are fine-tuned, the translation model will obtain the executable program text.

[0072] The operation execution module further receives the executable program text generated by the task decoupling planning and program generation module, and passes it to the pre-trained error correction model for program correction. The model is also deployed in the form of fine-tuning online calling Gemini1.5-flash API, outputs a more standardized and correct program text, and writes the program text into a pre-created python script file. After executing the python script, the computer will communicate with the robot arm and its end effector gripper to control the robot to move, target operation and other actions, and finally complete the user's instructions to meet user needs.

[0073] In this way, this embodiment completes a system execution cycle, and the system will continue to record the next instruction from the user and enter the next system execution cycle, and so on and so forth to provide services to the user.

[0074] The intelligent robotic arm control system solution of this embodiment can not only adapt to the complex and changeable service environment, but also understand the user's complex natural language instructions, realize efficient and intelligent robotic arm operation, and significantly improve the multi-tasking execution capability and intelligence level of the robotic arm.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope defined by the claims of the present invention. In addition, the present invention is supported by the National Natural Science Foundation of China (62273085).

Claims

1. A multi-model collaborative intelligent robotic arm control system solution, characterized by: Apply large model solutions in multiple aspects of robot control system design: The system solution specifically refers to dividing the system framework design into the following four steps: Step 1: Collect the natural language of user voice and text input, process the user input with a large model solution, and form a specific text format in English; Step 2: Based on the large model solution, the visual detection and target positioning solution processes the images collected by the camera sensor, identifies the target operation object of the positioning system, and returns its corresponding posture information; based on the large model solution, the environmental information understanding and perception converts the image returned by the camera into a text describing the environmental information; Step 3: Task decoupling and planning based on the large model solution: According to the English text information of the user language instruction from step 1 and the target posture information returned from step 2, task decoupling and planning are performed; based on the large model solution, the planned tasks are converted into executable machine language codes so that they can be directly used for the control and execution of the robot arm; Step 4: Based on the large model solution, review, correct and optimize the variable execution program designed in step 3, and then compile and run it to control the robot arm to perform a series of operations that are accurate and meet user needs.

2. The intelligent robotic arm control system solution based on multi-model collaboration according to claim 1 is characterized in that: In the step 1, the natural language of the user's voice and text input is collected, and the user input is processed by a large model solution to form a specific English text format. The specific steps are: (1) Applying a pre-trained natural language translation model, all texts in any language input by the user are uniformly translated into English text; (2) For the user's voice input, a pre-trained speech-to-text model is used as the core of speech recognition, and the voice instructions issued by the user are collected, converted into text by the speech-to-text model, and then uniformly translated into English text by the language translation model; (3) The obtained English text is input into a pre-trained format conversion large model, which has semantic parsing capabilities, can deeply deconstruct the text structure, identify and parse the core predicate-object structure in the instruction, and then convert it into a highly structured task expression form, such as: "Get—apple" "Put—plate", so as to optimize the parsing capabilities of the instruction interpretation and task planning large model in the subsequent step 3, thereby improving the robot arm's accurate understanding of the user's intention and the intelligent level of task execution.

3. The intelligent robotic arm control system solution based on multi-model collaboration according to claim 1 is characterized in that: In step 2, the visual detection and target positioning solution based on the large model solution has the following specific steps: using the depth camera as a perception sensor, applying a pre-trained recognition and positioning multimodal large model to the RGB image returned by the sensor for target recognition. The model has the ability to recognize the target category and target pixel coordinates in the RGB image, and can determine whether the target in the user command exists: (1) If the target exists, the pixel coordinates of the target on the image are output. The coordinates of the target in the camera coordinate system are calculated by combining the depth image and camera parameters returned by the camera. The coordinates of the target in the robot coordinate system are solved by combining the camera and the robot arm kinematics and output in the following format: { "status":"found", "position":[0.35,0.12,0.75]#Coordinates in the world coordinate system (X, Y, Z) } (2) If the target does not exist, output: { "status":"not_found", "position":None } The complete process of identifying the existence of the target object and obtaining the target coordinates is encapsulated as a perception API, and its calling format is "detect('xxx')", which can directly return whether xxx exists and its coordinates in the robot arm coordinate system.

4. The intelligent robotic arm control system solution based on multi-model collaboration according to claim 1 is characterized in that: In step 2, the understanding and perception of environmental information based on the large model solution includes the following specific steps: applying a pre-trained large multimodal model of environmental description to understand the RGB image returned by the camera. The model has the following capabilities: Object recognition: identifying the types of objects in the environment and their locations; Spatial relational reasoning: understanding the spatial relationships between objects, such as "the apple is on the table" and "the plate is near the apple"; Environment description generation: Generate readable environment description text based on the detection results to support subsequent task planning, such as "There is an apple on the table. There is a plate around the apple."; The purpose of describing environmental information is to provide an objective basis for subsequent task decoupling and planning, so as to avoid subsequent task planning that does not conform to the objective environment or errors, causing the robotic arm to perform incorrect actions. For example, if the planned task requires "grabbing a plate", but the plate is not detected in the environmental information, an error is returned to prevent the robotic arm from performing invalid actions; if the target object is blocked or invisible, a request can be made to adjust the camera angle and perform additional perception operations.

5. The intelligent robotic arm control system solution based on multi-model collaboration according to claim 1 is characterized in that: In step 3, the task decoupling and planning based on the large model solution is specifically performed as follows: the English description text of the working environment information returned in step 2 and the English text of the user requirement instructions are synchronously integrated into the task information text to be parsed: { "Environment description":"There is an apple on the table. There is aplate around the apple.","User instruction":"Get—apple.Put—plate." } Based on the above information, task parsing and structural processing are performed: the pre-trained instruction interpretation and task planning model understands and infers the user's intention in the task information text, reasoning and planning the tasks and arranges them in a logical order, enriches the details of complex and obscure tasks, and splits them into n simple execution tasks; for example, for the task of "placing an apple on a plate", the model needs to parse the operation action (grasping, placing), the object (apple, plate) and the execution order, and then generate a structured task sequence: Object detection: Identify the world coordinates of a specified object in the scene, "1. Get the spatial coordinates of the apple." Object grasping: Perform grasping actions based on the detection results, "2. Picking apples." Target positioning: Determine the location information of the target to be placed, "3. Get the spatial coordinates of the plate." Object placement: Complete the placement operation, "4.Put it on the plate." The task items obtained by the instruction interpretation and planning model are then passed on to the trained program translation model for task-by-task program translation.

6. The intelligent robotic arm control system solution based on multi-model collaboration according to claim 1 is characterized in that: In step 3, based on the big model solution, the planned task is converted into executable machine language code so that it can be directly used for the control and execution of the robot arm. The specific steps are as follows: First, the system receives the list-based task description generated by the task planning module. The description is based on user instructions, environmental perception data and task objectives, including but not limited to operations such as grabbing, placing, moving, and rotating. Since the natural language task description cannot be directly used for robot control, the system needs to rely on the pre-trained translation big model to convert the task text into a compilable and runnable control program: In order to ensure that the generated program is compatible with the specific robot arm control system, the translation big model needs to be combined with the following two core APIs for code conversion: Perception API (Step 2: Get the world coordinates of the target), which is used to call sensors or computer vision models to obtain the coordinates of the target in the environment, such as detect('apple') returns the location of the apple; Robot arm control API (program interface for compiling and running in step 3), used to call robot arm motion control instructions, such as get(pos_apple) to perform grabbing operations; The translation model converts structured tasks into executable program text based on this: pos_apple = detect('apple') # Get the location of the apple get(pos_apple)#Grab the apple pos_plate = detect('plate') # Get the position of the plate put(pos_plate)#Place the apple on the plate This outputs machine language that conforms to the robot arm control logic and can be executed by a computer.

7. The intelligent robotic arm control system solution based on multi-model collaboration according to claim 1 is characterized in that: In step 4, based on the large model solution, the variable execution program designed in step 3 is reviewed, corrected and optimized, and then compiled and run to control the robot arm to perform a series of operations that are accurate and meet user needs. The specific steps are as follows: First, in each system execution cycle, the robot arm control program text generated by step 3 will be dynamically adjusted to adapt to environmental variables, task changes and execution strategy optimization. Due to the changes in the context and environmental perception results of different tasks, program logic may deviate and even cause potential risk of task execution failure. Therefore, this step uses a pre-trained multi-level code review and optimization model to comprehensively review the control program and correct and enhance it in multiple dimensions, including: (1) Syntax and logic consistency check: The control program is analyzed at the lexical, grammatical and semantic levels to ensure that the code complies with the syntax specifications of the target programming language and avoid common errors such as undefined variables, type mismatch, loop exceptions, and array out-of-bounds errors. In addition, the model further checks the logical consistency between instructions to ensure the context of program execution. For example, the "grab an object" operation cannot be executed when the "detection target does not exist" condition. (2) Constraint and safety verification: Safety checks are performed on the control program, including but not limited to the robot arm's kinematic constraints, joint angle limits, collision detection, and torque limits, to ensure that the robot arm will not be damaged or cause safety accidents due to trajectory calculation errors, redundant movements, or exceeding physical limits during operation. At the same time, the system performs context dependency analysis on each executed instruction to ensure that the preconditions of the task are met and to avoid abnormal execution, such as the robot arm failing to complete the "grasp" task before "releasing the object"; (3) Task optimization and resource scheduling: Intelligently optimize the task sequence of the control program to maximize execution efficiency and minimize resource consumption; for example, reduce the idle motion of the robot arm through dynamic reordering, optimize path planning to reduce execution time, and adjust the grasping angle to improve operation stability; in addition, the system can adjust the execution plan according to task priority and real-time feedback information to adapt to environmental changes or temporary needs of users; (4) Redundant code detection and program simplification: Redundant instructions, unnecessary loops, and repeatedly called APIs or variables that may exist in the control program are trimmed to improve code execution efficiency. For example, if the "detect object" operation has been executed in the previous frame and returned valid coordinates, there is no need to call the detection API again in the subsequent frame, thereby reducing computing overhead and improving overall response speed. The control program that has undergone the above-mentioned multi-level optimization and verification will be compiled and deployed to the robotic arm controller to achieve precise control of the robotic arm, ensure that it completes the operation tasks according to the user's intentions, improve the stability, intelligence level and safety of execution, and thus meet the refined needs in complex application scenarios.

Images