Man-machine cooperation mechanical arm control method based on multi-modal large model

Abstract

According to the man-machine cooperation mechanical arm control method based on the multi-mode large model, assembly scene visual information, text instruction information and prompt information are combined and input into the multi-mode large model for understanding and task decomposition, and operation instructions with different function check bits and mechanical arm control codes are generated. A pixel area and coordinates of a target object are obtained by combining retrieval of a multi-modal large model or a part knowledge base with a DINO-X model and are converted into pose coordinates of a mechanical arm according to a mapping relation between a coordinate system of an end effector of the mechanical arm and a coordinate system of a visual system, and a mechanical arm control instruction is generated. The mechanical arm is controlled to move by combining operation instructions with different functional check bits and mechanical arm control codes, so that the problems of poor generalization and low precision caused by the illusion problem of a multi-mode large model are solved while the mechanical arm control instructions are accurately and efficiently generated by combining the multi-mode large model with voice instructions and visual information reasoning decisions.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing technology, and in particular to a human-machine collaborative robot arm control method based on a multimodal large model. Background Art

[0002] With the widespread application of large multimodal models in fields such as natural language processing and computer vision, a series of advanced models have emerged that can not only process information from a single modality, but also integrate input from multiple modalities, such as voice, images, and text. In the field of human-robot collaborative assembly, the combination of large multimodal models can greatly improve the adaptability and operational efficiency of robots in complex environments. For example, the "Robot manipulation method based on a large visual language model" disclosed in the Chinese invention patent (application number: CN202410784930.4)

[0003] However, the use of existing multimodal large models in actual industrial robot assembly still faces the following problems:

[0004] First, in complex assembly environments, how to accurately combine voice commands, visual information, and robotic control systems to ensure effective information integration and complete the understanding of complex commands containing multimodal information is an urgent problem to be solved;

[0005] Second, when the robot detects unconventional parts, the model may incorrectly identify the location or appearance of an object, resulting in incorrect assembly actions.

[0006] In summary, when large multimodal models are applied to robots, their ability to understand and process diverse and complex information is limited, and when they are used in new scenarios or new instructions, there are still problems such as hallucinations, poor generalization ability, and low success rate. Summary of the Invention

[0007] The purpose of the technical solution of the present invention is to design a human-machine collaborative robot arm control method based on a multimodal large model, overcome the problems of poor generalization and low precision caused by the hallucination problem of the multimodal large model, and accurately combine voice commands, visual information and robot control systems in complex assembly environments to ensure effective integration of information and complete robot arm control under the understanding of complex commands containing multimodal information.

[0008] The technical solution of the present invention provides a method for controlling a human-machine collaborative robot arm based on a multimodal large model, comprising the following steps:

[0009] Receive user voice instructions and descriptions related to assembly tasks, perform voice feature recognition on the voice instructions, and convert them into text instructions;

[0010] Combine assembly scene visual information, text instruction information, and prompt information into a multimodal large model for understanding and task decomposition, generating operation instructions and robotic arm control codes with different function check bits.

[0011] Based on the component knowledge base that stores component image information, corresponding name information, and manually annotated component bounding boxes in the images, four-corner pixel coordinates, and other component design information, it is determined whether the assembly scene visual information and text instruction information contain component design information. If not, it is determined that the multimodal large model can recognize conventional objects. The bounding box coordinates of the target object are identified through the multimodal large model, and after the bounding box coordinates are expanded, pixel area extraction and recognition contour are performed, edge information is corrected, and the pixel coordinates of the target object are recalculated;

[0012] If an object exists, it is identified as an unconventional object. The target part image and annotated pixel coordinate information are extracted based on the part knowledge base that stores part categories, shape features, and standard sizes. This information is combined with the assembly scene visual information and input into the DINO-X model to obtain the pixel area and coordinates of the target object.

[0013] The pixel area and coordinates of the target object are used to form preliminary three-dimensional information of the target position, and the mapping relationship between the robot arm end effector coordinate system and the vision system coordinate system is determined. The information is converted into the robot arm's pose coordinates and incorporated into the robot arm control instructions generated by the multimodal large model.

[0014] The movement of the robotic arm is controlled according to the robotic arm control instructions combined with the operation instructions with different function check bits and the robotic arm control code.

[0015] Preferably, the assembly scene visual information includes assembly scene RGB information and depth information.

[0016] Preferably, the prompt information includes instruction requirements for different assembly tasks in a format that meets the dictionary structure, context information with voice, visual information, task parameters and environmental background information, corresponding robotic arm control code generated by the instruction requirements, and task output format of check bit rules.

[0017] Preferably, the instruction requirements for the different assembly tasks include: voice control to return to the initial position, voice control to move to a specified posture, voice control to move the end effector a specified distance, voice control to move a specified joint at a specified angle, voice control to grasp a specified object, voice control to place it at a specified position, and voice control to deliver a specified object.

[0018] Preferably, the operation instructions with different function check bits include resetting the robot arm, moving the end effector to a specified posture, moving the end effector to a specified distance, moving a specified joint to a specified angle, grabbing a specified object, placing it at a specified location, and delivering a specified object.

[0019] The technical solution of the present invention proposes a human-machine collaborative robot arm control method based on a multimodal large model. By combining assembly scene visual information, text instruction information and prompt information into the multimodal large model for understanding and task decomposition, operation instructions and robot arm control codes with different function check bits are generated. The pixel area and coordinates of the target object are obtained by searching the multimodal large model or the parts knowledge base combined with the DINO-X model. According to the mapping relationship between the robot arm end effector coordinate system and the visual system coordinate system, the pose coordinates are converted into the robot arm to generate the robot arm control instructions. The operation instructions with different function check bits and the robot arm control codes are combined to control the movement of the robot arm. The multimodal large model effectively integrates voice instructions and visual information to efficiently and accurately generate robot arm control instructions, while overcoming the problems of poor generalization and low precision caused by the hallucination problem of the multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic flow chart of a method for controlling a human-machine collaborative assembly robot arm based on a multimodal large model provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of the target object pixel coordinate recognition structure of the multimodal large model-based human-machine collaborative assembly robot control method provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a human-machine collaborative assembly scenario provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of the UR3 robotic arm model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0025] like Figure 1 As shown, an embodiment of the present invention provides a method for controlling a human-machine collaborative robot arm based on a multimodal large model, comprising the following steps:

[0026] Receive user voice commands and convert them into text. Users input voice commands through a microphone, including instructions and descriptions related to assembly tasks. The collected voice signals are processed using a deep learning-based speech recognition algorithm (such as an end-to-end speech recognition model or a hybrid model based on HMM-DNN), extracting speech features and analyzing and decoding the voice information. The decoded text information is then preprocessed by the subsequent natural language processing (NLP) module, including denoising, word segmentation, and semantic analysis, to produce the processed text command information.

[0027] The assembly scene visual information, text instruction information, and prompt information of a specified structure are combined and input into the multimodal large model for further understanding and task decomposition. Operation instructions and robotic arm control codes with different functional check bits are generated to provide data support for the robot assembly task. Among them, the assembly scene visual information is the RGB and depth information of the assembly scene information obtained by the Kinect v2 camera. The prompt information of the specified structure includes the instruction requirements for different assembly tasks and the corresponding control codes generated by them, and the format meets the dictionary structure. The prompt information contains 7 types of assembly control tasks, as follows:

[0028] 1. Voice control returns to its original position;

[0029] 2. Voice control to move to the specified position;

[0030] 3. Voice control of the end effector to move a specified distance;

[0031] 4. Voice control to move a specified joint to a specified angle;

[0032] 5. Voice control to grab the specified object;

[0033] 6. Voice control is placed at the designated location;

[0034] 7. Voice control to deliver the specified object.

[0035] The prompt information of each specified structure contains an instruction description that clearly defines the specific assembly task objectives, context information with voice, visual information, task parameters and environmental background information, the robot arm control code structure that the specified model needs to generate, and the task output format of the check bit rules.

[0036] like Figure 2As shown in the figure, based on the detection of whether the text instruction information contains the component information in the knowledge base, the pixel coordinates of the target object are identified and obtained through multimodal large model recognition combined with contour detection and correction, and by retrieving the knowledge base information combined with the DINO-X model. Among them, the component information in the knowledge base includes the component image information, the corresponding name information, and the top-left (x-axis) pixel coordinates of the four corners of the bounding box of the manually annotated component in the image. min ,y min ), Top-Right(x max ,y min ), Bottom-Right(x max ,y max ), Bottom-Right(x min ,y mam ).

[0037] Multimodal large-scale model recognition combined with contour detection and correction can be divided into text-based command and visual information fusion and contour detection and correction. The former inputs user-provided text-based command information and visual information of the assembly scene into the multimodal large-scale model. The large-scale model leverages its visual language processing capabilities to initially generate bounding box coordinates for the target object. The latter extracts pixel regions of the target object based on the initial bounding box coordinates generated by the large-scale model. Edge detection algorithms (such as the Canny or Sobel operators) are applied to these regions to further accurately identify the object's contour. Dilation and morphological inpainting methods are then used to correct the edge information, ensuring that the contour conforms to the actual object boundary. Finally, the pixel coordinates of the target object are recalculated based on the corrected contour, outputting more accurate coordinate values. Contour detection involves extracting edge information from the pixel regions within the bounding box within the expanded range of the large-scale model. This results in detected edges that are closer to the actual object boundary. To improve the robustness of contour detection, deep learning-based edge detection models (such as HED or DexiNed) can be used to further enhance the ability to capture object boundaries in complex scenes.

[0038] Retrieving knowledge base information combined with the DINO-X model can be divided into knowledge base information retrieval and DINO-X model application. The former constructs a knowledge base containing part categories, shape features, and standard dimensions. It then detects whether text instructions contain part-related information (such as "screw," "nut," or "gear"). If so, it uses keyword retrieval to extract images of the target parts and their annotated pixel coordinates from the knowledge base. The latter simultaneously inputs the scene's visual information, along with the extracted images and annotated pixel coordinates of the target parts, into the DINO-X model. Leveraging its strong modeling capabilities for object features and spatial relationships, it identifies objects in the scene. Based on the retrieved knowledge base features, it matches the target object features output by the DINO-X model, localizing the most qualified object regions in the scene. Using the high-quality segmentation masks generated by the DINO-X model, it further extracts the target object's pixel regions and coordinates. Specifically, the DINO-X model's input for object recognition and pixel coordinate localization consists of the retrieved image information, the pixel coordinates of the specified object in the image, and the image information of the current assembly scene captured by the Kinect v2 camera.

[0039] The pixel coordinates of the target position are converted into the 6-bit pose coordinates of the robot arm through the hand-eye calibration method. The multimodal large model generates the robot arm control instructions. Specifically, the pixel coordinates of the target object are obtained. The initial three-dimensional information of the target position is formed by the hand-eye calibration method, which determines the mapping relationship between the robot end effector coordinate system and the visual system coordinate system. The hand-eye calibration method uses the rotation matrix R and the translation vector T to represent the transformation relationship between the two coordinate systems: P 机械臂 =R·P 视觉 +T,P 视觉 =(x,y,z) T By pixel coordinates The three-dimensional coordinate points calculated by the depth value d are used to calculate the 6-position coordinates of the target object based on the calibration: P 机械臂 =(x, y, z, θ x ,θ y ,θ z ) and uses it as a parameter in the robotic arm control instructions generated by the multimodal large model.

[0040] The teach pendant performs assembly functions such as assisted movement, grabbing, placement, and delivery based on the operation instructions with different function check bits and the robot arm control code. The interactive functions such as assisted movement, grabbing, placement, and delivery are performed by using the Python client to determine the check bit information I and transmit a 7-bit data stream (I, x, y, z, θ) to the teach pendant server. x ,θ y ,θ z ), the teaching pendant server controls the movement of the robotic arm based on the position verification information and the posture information.

[0041] The check digit I corresponds to the type of assembly task, where:

[0042] I=1: The robot arm resets.

[0043] I=2: The end effector moves to the specified pose.

[0044] I=3: The end effector moves the specified distance.

[0045] I=4: The specified joint moves at a specified angle.

[0046] I=5: Grab the specified object.

[0047] I=6: Place at the specified location.

[0048] I=7: Submit the specified object.

[0049] A human-machine collaborative assembly robot control method based on a multimodal large model for human-machine collaborative scenarios such as Figure 3 As shown, the system includes a computer 10, a UR3 robot arm 20, a reducer 30, a shaft part 40, a flange 50, a bearing shaft 60, a gear 70 and an imaging device Kinect v2 camera 80, wherein the UR3 robot arm 20 is as shown in FIG. Figure 4 shown.

[0050] The embodiment of the present invention converts the robot control program design problem into a text generation problem, constructs a human-machine collaborative assembly robot control instruction generation model based on a multimodal large model, accurately combines voice instructions, visual information and the robot control system, ensures the effective integration of information, and completes the autonomous perception and control of the robot arm in a dynamic assembly environment under the understanding of complex instructions containing multimodal information. Compared with the traditional manual programming method, it effectively improves the intelligence and interactivity of the human-machine collaborative assembly process. At the same time, its accurate and fast generation of control instructions greatly improves the assembly efficiency. In order to improve the versatility and specificity of object recognition during the grasping process, for the recognition accuracy problem of conventional objects (screwdrivers, toolboxes, etc.), contour detection correction is performed on the large model recognition results; for unconventional parts (flanges, sleeves, etc.), a knowledge base is constructed for unconventional parts for retrieval and combined with the DINO-X model, thereby overcoming the problems of poor generalization and low recognition accuracy caused by the hallucination problem of the multimodal large model.

[0051] The above embodiments are intended to illustrate the present invention only and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that various combinations, modifications, or equivalent substitutions of the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and should be encompassed by the scope of the claims of the present invention.

Claims

1. A human-machine collaborative robot arm control method based on a multimodal large model, characterized in that: The following steps are involved: Receive user voice instructions and descriptions related to assembly tasks, perform voice feature recognition on the voice instructions, and convert them into text instructions; Combine assembly scene visual information, text instruction information, and prompt information into a multimodal large model for understanding and task decomposition, generating operation instructions and robotic arm control codes with different function check bits. Based on the component knowledge base that stores component image information, corresponding name information, and manually annotated component bounding boxes in the image, four-corner pixel coordinates and other component design information, it is determined whether the assembly scene visual information and text instruction information contain component design information. If not, it is considered a regular object that can be recognized by the multimodal large model. The bounding box coordinates of the target object are identified by the multimodal large model, and after the bounding box coordinates are expanded, the pixel area is extracted to identify the contour, the edge information is corrected, and the pixel coordinates of the target object are recalculated; If an object exists, it is identified as an unconventional object. The target part image and annotated pixel coordinate information are extracted from the knowledge base that stores part category, shape characteristics, standard size and other related information. The image and annotated pixel coordinate information are combined with the assembly scene visual information and input into the DINO-X model to obtain the pixel area and coordinates of the target object. The pixel area and coordinates of the target object are used to form preliminary three-dimensional information of the target position, and the mapping relationship between the robot arm end effector coordinate system and the vision system coordinate system is determined. The information is converted into the robot arm's pose coordinates to generate the robot arm control instructions; The movement of the robotic arm is controlled according to the robotic arm control instructions combined with the operation instructions with different function check bits and the robotic arm control code.

2. A human-machine collaborative robot arm control method based on a multimodal large model according to claim 1, characterized in that: The assembly scene visual information includes assembly scene RGB information and depth information.

3. The method for controlling a human-machine collaborative robot arm based on a multimodal large model according to claim 1, wherein: The prompt information includes instruction requirements for different assembly tasks in a format that meets the dictionary structure, context information with voice, visual information, task parameters and environmental background information, the corresponding robotic arm control code generated by the instruction requirements, and the task output format of the check bit rule.

4. The method for controlling a human-machine collaborative robot arm based on a multimodal large model according to claim 3, wherein: The command requirements for the different assembly tasks include: voice control to return to the initial position, voice control to move to a specified posture, voice control to move the end effector a specified distance, voice control to move a specified joint to a specified angle, voice control to grasp a specified object, voice control to place it at a specified location, and voice control to deliver a specified object.

5. The method for controlling a human-machine collaborative robot arm based on a multimodal large model according to claim 1, wherein: The operation instructions with different function check bits include resetting the robot arm, moving the end effector to a specified posture, moving the end effector a specified distance, moving a specified joint to a specified angle, grabbing a specified object, placing it at a specified location, and delivering a specified object.

Images