A human-robot collaborative robot arm control method based on a multi-modal large model

Abstract

The application provides a man-machine cooperation mechanical arm control method based on a multi-modal large model, visual information of an assembly scene, text instruction information and prompt information are combined and input into the multi-modal large model for understanding and task decomposition, operation instructions and mechanical arm control codes with different function check bits are generated. The pixel area and coordinates of the target object are obtained by combining the multi-modal large model or the part knowledge base retrieval with the DINO-X model, the mapping relationship between the mechanical arm end effector coordinate system and the vision system coordinate system is converted into the pose coordinates of the mechanical arm to generate the mechanical arm control instruction, the mechanical arm is controlled to move by combining the operation instruction with different function check bits and the mechanical arm control code, the multi-modal large model is combined with the voice instruction and the visual information to accurately and efficiently generate the mechanical arm control instruction, and the problem of poor generalization and low precision caused by the illusion of the multi-modal large model is overcome.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, specifically to a human-machine collaborative robotic arm control method based on a multimodal large model. Background Technology

[0002] With the widespread application of multimodal large models in fields such as natural language processing and computer vision, a series of advanced models have emerged. These models can not only process information from a single modality but also integrate inputs from multiple modalities, such as speech, images, and text. In human-robot collaborative assembly, combining multimodal large models can greatly improve the robot's adaptability and operational efficiency in complex environments. For example, the "Robot Manipulation Method Based on Visual Language Large Model" disclosed in Chinese Invention Patent (Application No.: CN202410784930.4).

[0003] However, using existing multimodal large models in actual industrial robot assembly still faces the following problems:

[0004] I. In complex assembly environments, how to accurately combine voice commands, visual information and robot control systems to ensure effective information integration and complete the understanding of complex commands containing multimodal information is an urgent problem to be solved.

[0005] Second, when the robot performs target inspection on unconventional parts, the model may incorrectly identify the position or appearance of an object, leading to incorrect assembly actions.

[0006] In summary, the application of multimodal large models in robots still suffers from limitations in understanding and processing diverse and complex information, as well as issues such as illusion, poor generalization ability, and low success rate when encountering new scenarios or instructions. Summary of the Invention

[0007] The purpose of this invention is to design a human-machine collaborative robotic arm control method based on a multimodal large model, overcome the problems of poor generalization and low accuracy caused by the illusion of multimodal large models, and achieve precise integration of voice commands, visual information and robot control system in complex assembly environments, ensuring effective information integration and completing robotic arm control under complex command understanding containing multimodal information.

[0008] The present invention provides a human-machine collaborative robotic arm control method based on a multimodal large model, comprising the following steps:

[0009] It receives voice commands and descriptions from users regarding assembly tasks, performs voice feature recognition on the voice commands, and converts them into text commands.

[0010] By combining visual information, text instruction information, and prompt information from the assembly scene into a multimodal large model for understanding and task decomposition, operation instructions and robotic arm control codes with different functional verification bits are generated.

[0011] Based on the component knowledge base containing stored component image information, corresponding name information, and manually annotated component design information such as bounding boxes and four-corner pixel coordinates in the images, it is determined whether component design information exists in the visual information and text instruction information of the assembly scene. If not, it is determined that the multimodal large model can recognize regular objects. The bounding box coordinates of the target object are identified through the multimodal large model, and after the bounding box coordinates are expanded, the pixel region is extracted and the outline is identified, the edge information is corrected, and the pixel coordinates of the target object are recalculated.

[0012] If it exists, it is identified as an unconventional object. Based on the knowledge base of stored parts, such as parts category, shape features and standard size, the image and labeled pixel coordinate information of the target part are extracted. Combined with the visual information of the assembly scene, the image is input into the DINO-X model to obtain the pixel area and coordinates of the target object.

[0013] The pixel regions and coordinates of the target object form the preliminary three-dimensional information of the target position, determine the mapping relationship between the coordinate system of the end effector of the robotic arm and the coordinate system of the vision system, and convert it into the pose coordinates of the robotic arm and combine it into the control commands of the robotic arm generated by the multimodal large model.

[0014] The movement of the robotic arm is controlled by combining the robotic arm control instructions with operation instructions and robotic arm control codes that have different functional verification bits.

[0015] Preferably, the assembly scene visual information includes assembly scene RGB information and depth information.

[0016] Preferably, the prompt information includes instruction requirements for different assembly tasks in a format that conforms to a dictionary structure, context information with voice, visual information, task parameters and environmental background information, corresponding robotic arm control code generated by the instruction requirements, and task output format with check bit rules.

[0017] Preferably, the instructions for the different assembly tasks include: voice control to return to the initial position, voice control to move to a specified pose, voice control to move the end effector a specified distance, voice control to move a specified joint a specified angle, voice control to grasp a specified object, voice control to place in a specified position, and voice control to deliver a specified object.

[0018] Preferably, the operation instructions with different functional verification bits include resetting the robotic arm, moving the end effector to a specified pose, moving the end effector a specified distance, moving a specified joint to a specified angle, grasping a specified object, placing it in a specified position, and delivering a specified object.

[0019] This invention proposes a human-machine collaborative robotic arm control method based on a multimodal large model. By combining visual information from the assembly scene, text instructions, and prompt information input into the multimodal large model for understanding and task decomposition, operational instructions and robotic arm control codes with different functional verification bits are generated. The pixel region and coordinates of the target object are obtained by searching the multimodal large model or a component knowledge base and combining it with the DINO-X model. Based on the mapping relationship between the coordinate system of the robotic arm's end effector and the coordinate system of the vision system, the coordinates are converted into the pose coordinates of the robotic arm to generate robotic arm control instructions. The robotic arm movement is controlled by combining the operational instructions and robotic arm control codes with different functional verification bits. This method effectively integrates voice instructions and visual information through the multimodal large model to generate robotic arm control instructions efficiently and accurately, while overcoming the poor generalization and low accuracy problems caused by the illusion problem of multimodal large models. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the human-machine collaborative assembly robotic arm control method based on a multimodal large model provided in an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of the target object pixel coordinate recognition structure for the human-machine collaborative assembly robotic arm control method based on a multimodal large model provided in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of a human-machine collaborative assembly scenario provided in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the UR3 robotic arm model provided in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0025] like Figure 1 As shown, this embodiment of the invention provides a human-machine collaborative robotic arm control method based on a multimodal large model, including the following steps:

[0026] The system receives user voice commands and converts them into text commands. Users input voice commands via microphone, including operation instructions and descriptions related to assembly tasks. The system processes the acquired voice signals using deep learning-based speech recognition algorithms (such as end-to-end speech recognition models or hybrid models based on HMM-DNN), extracting speech features and analyzing and decoding the speech information. The decoded text information undergoes preprocessing in a subsequent Natural Language Processing (NLP) module, including denoising, word segmentation, and semantic analysis to obtain the processed text commands.

[0027] Combining visual information from the assembly scene, textual instructions, and prompt information with a specified structure, the data is input into a multimodal large model for further understanding and task decomposition. This generates operation instructions and robotic arm control codes with different functional verification bits, providing data support for the robot's assembly tasks. The visual information of the assembly scene is the RGB and depth information of the assembly scene acquired through the Kinect v2 camera. The prompt information with a specified structure includes instruction requirements for different assembly tasks and their corresponding generated control codes, and the format conforms to a dictionary structure. The prompt information contains seven assembly control tasks, as detailed below:

[0028] 1. Voice control to return to the initial position;

[0029] 2. Use voice control to move to the designated position;

[0030] 3. Voice control moves the end effector a specified distance;

[0031] 4. Voice control allows for the movement of a specified joint by a specified angle;

[0032] 5. Voice control for grasping specified objects;

[0033] 6. Use voice control to place the device in the designated location;

[0034] 7. Voice control for delivering specified objects.

[0035] Each specified prompt contains a description of the instruction that clearly defines the specific assembly task objective, contextual information including voice, visual information, task parameters, and environmental background information, the task output format specifying the robotic arm control code structure to be generated by the model, and check bit rules.

[0036] like Figure 2As shown, based on the detection of whether component information from the constructed knowledge base exists in the text instruction information, the design employs multimodal large model recognition combined with contour detection and correction, and knowledge base retrieval combined with the DINO-X model to identify and obtain the pixel coordinates of the target object. The component information in the knowledge base includes component image information, corresponding name information, and manually labeled top-left (x-x) pixel coordinates of the four corners of the bounding box of the component in the image. min y min Top-Right(x) max y min Bottom-Right(x) max y max Bottom-Right(x) min y mam ).

[0037] Multimodal large-scale model recognition combined with contour detection and correction can be divided into text instruction and visual information fusion and contour detection and correction. The former inputs user-provided text instructions and visual information from the assembly scene into the multimodal large-scale model. The large model utilizes its visual language processing capabilities to initially generate the bounding box coordinates of the target object. The latter, based on the initial bounding box coordinates generated by the large model, extracts the pixel regions of the target object and applies edge detection algorithms (such as Canny or Sobel operators) to these regions to further accurately identify the object's contour. Edge information is corrected through dilation operations and morphological restoration methods to ensure the contour closely matches the actual boundary of the object. Finally, the pixel coordinates of the target object are recalculated based on the corrected contour, outputting more accurate coordinate values. In other words, contour detection extracts edge information from the pixel regions inside the bounding box within the expanded range of the large model, resulting in detected edges that are closer to the actual boundary of the object. To improve the robustness of contour detection, deep learning-based edge detection models (such as HED or DexiNed) can be used to further enhance the ability to capture object boundaries in complex scenes.

[0038] Retrieving knowledge base information using the DINO-X model can be divided into knowledge base information retrieval and DINO-X model application. The former constructs a knowledge base containing component categories, shape features, and standard dimensions. It detects whether text instructions contain component-related information (such as "screw," "nut," or "gear"). If so, it extracts the image of the target component and its labeled pixel coordinates from the knowledge base through keyword retrieval. The latter simultaneously inputs the scene's visual information, the extracted image of the target component, and its labeled pixel coordinates into the DINO-X model. Leveraging its strong modeling capabilities for object features and spatial relationships, it identifies objects in the scene. Based on the retrieved knowledge base features, it matches the target object features output by the DINO-X model to locate the most suitable object region in the scene. Using a high-quality segmentation mask generated by the DINO-X model, it further extracts the pixel region and coordinates of the target object. In other words, the DINO-X model's inputs for object recognition and pixel coordinate localization are the retrieved image information, the specified object's pixel coordinates in the image, and the image information of the current assembly scene obtained from the Kinect v2 camera.

[0039] The target position pixel coordinates are converted into 6-bit pose coordinates for the robotic arm using a hand-eye calibration method. A multimodal large model is then used to generate control commands for the robotic arm. Specifically, the pixel coordinates of the target object are obtained. And the depth value d, forming preliminary three-dimensional information of the target position. Using the hand-eye calibration method, the mapping relationship between the coordinate system of the robotic arm end effector and the coordinate system of the vision system is determined. The hand-eye calibration uses the rotation matrix R and the translation vector T to represent the transformation relationship between the two coordinate systems: P 机械臂 =R·P 视觉 +T, P 视觉 =(x,y,z) T For pixel coordinates Based on the calibrated values ​​and the 3D coordinates obtained from the depth value d, the 6-bit pose coordinates of the target object are calculated: P 机械臂 = (x, y, z, θ) x θ y θ z This information is then used as the parameter in the control commands for the robotic arm generated by the multimodal large model.

[0040] The teach pendant executes assembly functions such as assisted movement, grasping, placement, and delivery based on operation instructions with different functional verification bits and robotic arm control code. The method for executing interactive functions such as assisted movement, grasping, placement, and delivery involves determining the verification bit information I through a Python client and transmitting a 7-bit data stream (I, x, y, z, θ) to the teach pendant server. x θ y θ z The teach pendant server controls the movement of the robotic arm based on the position information and pose information.

[0041] The check bit I corresponds to the type of assembly task, where:

[0042] I = 1: The robotic arm resets.

[0043] I=2: The end effector moves to the specified pose.

[0044] I=3: The end effector moves a specified distance.

[0045] I=4: Specifies the angle by which the joint moves.

[0046] I=5: Grab the specified object.

[0047] I=6: Place in the specified location.

[0048] I=7: Submit the specified object.

[0049] A human-machine collaborative assembly robotic arm control method based on a multimodal large model, such as a human-machine collaborative scenario. Figure 3 As shown, it includes a computer 10, a UR3 robotic arm 20, a reducer 30, a shaft part 40, a flange 50, a bearing 60, a gear 70, and an imaging device, a Kinect v2 camera 80. The UR3 robotic arm 20 is as follows... Figure 4 As shown.

[0050] This invention transforms the robot control programming problem into a text generation problem, constructing a human-robot collaborative assembly robot control command generation model based on a multimodal large model. It precisely integrates voice commands, visual information, and the robot control system, ensuring effective information integration and enabling the robotic arm to autonomously perceive and control itself in a dynamic assembly environment while understanding complex instructions containing multimodal information. Compared to traditional manual programming, this method effectively improves the intelligence and interactivity of the human-robot collaborative assembly process, while its precise and rapid generation of control commands significantly increases assembly efficiency. To improve the versatility and specificity of object recognition during the grasping process, for the recognition accuracy problem of common objects (screwdrivers, toolboxes, etc.), contour detection correction is performed on the recognition results of the large model; for unconventional parts (flanges, sleeves, etc.), a knowledge base for unconventional parts is constructed and retrieved, combined with the DINO-X model, overcoming the problems of poor generalization and low recognition accuracy caused by the illusion of multimodal large models.

[0051] The above embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Although the invention has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications, or equivalent substitutions of the technical solutions of the invention do not depart from the spirit and scope of the invention and should be covered within the scope of the claims of the invention.

Claims

1. A human-machine collaborative robotic arm control method based on a multimodal large model, characterized in that, Includes the following steps: It receives voice commands and descriptions from users regarding assembly tasks, performs voice feature recognition on the voice commands, and converts them into text commands. By combining visual information, text instruction information, and prompt information from the assembly scene into a multimodal large model for understanding and task decomposition, operation instructions and robotic arm control codes with different functional verification bits are generated. Based on a component knowledge base containing stored component image information, corresponding name information, and manually annotated component design information such as bounding boxes and four-corner pixel coordinates in the images, the system determines whether component design information exists in the visual information and text instruction information of the assembly scene. If not, it is identified as a regular object that can be recognized by the multimodal large model. The system then identifies the bounding box coordinates of the target object through the multimodal large model, and after expanding the bounding box coordinates, it extracts and identifies the outline of the pixel region, corrects the edge information, and recalculates the pixel coordinates of the target object. If it exists, it is identified as an unconventional object. Based on the knowledge base storing information related to parts such as part categories, shape features and standard dimensions, the image and labeled pixel coordinate information of the target part are extracted. Combined with the visual information of the assembly scene, the image is input into the DINO-X model to obtain the pixel area and coordinates of the target object. The pixel region and coordinates of the target object form the preliminary three-dimensional information of the target position, determine the mapping relationship between the coordinate system of the robotic arm end effector and the coordinate system of the vision system, and convert it into the pose coordinates of the robotic arm to generate robotic arm control commands. The movement of the robotic arm is controlled by combining the robotic arm control instructions with operation instructions and robotic arm control codes that have different functional verification bits.

2. The human-machine collaborative robotic arm control method based on a multimodal large model as described in claim 1, characterized in that, The assembly scene visual information includes assembly scene RGB information and depth information.

3. The human-machine collaborative robotic arm control method based on a multimodal large model as described in claim 1, characterized in that, The prompt information includes instruction requirements for different assembly tasks in a format that conforms to a dictionary structure, context information with voice, visual information, task parameters and environmental background information, corresponding robotic arm control code generated by the instruction requirements, and task output format with check bit rules.

4. The human-machine collaborative robotic arm control method based on a multimodal large model as described in claim 3, characterized in that, The instructions for the different assembly tasks include: voice control to return to the initial position, voice control to move to a specified pose, voice control to move the end effector a specified distance, voice control to move a specified joint a specified angle, voice control to grasp a specified object, voice control to place in a specified position, and voice control to deliver a specified object.

5. The human-machine collaborative robotic arm control method based on a multimodal large model as described in claim 1, characterized in that, The operation instructions with different functional verification bits include: robotic arm reset, end effector moves to a specified pose, end effector moves a specified distance, specified joint moves at a specified angle, grabs a specified object, places it in a specified position, and delivers a specified object.

Images