A control method and device for a multimodal intelligent manipulator

Abstract

The present invention discloses a control method and device for a multimodal intelligent manipulator, which relates to the fields of manipulators and artificial intelligence technology. The method comprises the following steps: capturing left and right images of a target object using binocular vision technology; outputting the category and boundary coordinates of the target object through a YOLO model; extracting the center coordinates of the target object in the left and right images; matching objects in the left and right images that have the same category and similar coordinates as the target object; calculating the disparity and depth information between the left and right images to combine the three-dimensional spatial coordinates of the target object; smoothing the historical coordinates of each target object and outputting a target detection result of the YOLO model; inputting the target detection result into a task decision model and converting it into a structured instruction; and executing, calculating, and adjusting the grasping force according to the structured instruction. The present invention adopts the control method and device for the multimodal intelligent manipulator described above, and enables the manipulator to autonomously identify targets and optimize execution paths through deep learning and intelligent control algorithms.

Description

Technical Field

[0001] The present invention relates to the field of manipulators and artificial intelligence technologies, and in particular to a control method and device for a multimodal intelligent manipulator. Background Art

[0002] At present, the fields of intelligent manufacturing, smart medical care, service robots, agricultural automation, etc. have increasingly higher requirements for the intelligence and flexibility of robots. Traditional robots mostly rely on a single control mode, such as preset trajectory control or simple visual feedback control, which is difficult to meet the dynamic needs in complex environments. Multimodal intelligent robots integrate multiple perception methods, enabling robots to understand the external environment more accurately, adapt to different tasks, and interact with humans more naturally.

[0003] However, most robots on the market are still mainly based on single-modal control and fail to fully utilize multi-modal fusion technology for information fusion, resulting in a lot of room for improvement in their flexibility, intelligence and human-computer interaction capabilities. Current research trends show that robots that integrate multiple perception and control technologies have become an important development direction in the field of intelligent robots. However, in the face of complex environments, good adaptability and human-computer collaboration efficiency still need to be improved. Summary of the Invention

[0004] The purpose of the present invention is to provide a control method and device for a multimodal intelligent manipulator. Through deep learning and intelligent control algorithms, the manipulator can autonomously identify targets, optimize execution paths, and adapt to environmental changes. It can integrate multiple modal information to achieve more efficient, more accurate, and more natural intelligent operations. It is suitable for multiple application scenarios such as industrial manufacturing, medical assistance, agricultural automation, and home services, and provides more personalized and intelligent operation methods according to user needs.

[0005] To achieve the above object, the present invention provides a control method for a multimodal intelligent manipulator, comprising the following steps:

[0006] S1, based on binocular vision technology, two cameras are used to capture the left and right images of the target object in the same scene;

[0007] S2. Identify and locate the target object through the YOLO model, and output the category and bounding box coordinates of the target object;

[0008] S3, extract the center point coordinates of the target object in the left and right images based on S1 and S2;

[0009] S4, through S2 and S3, match the categories of the target objects in the left and right images. y Objects with similar transverse coordinates;

[0010] S5, calculating the disparity between the left and right images and the depth information of the object to estimate the three-dimensional space coordinates of the target object;

[0011] S6, record the historical coordinates of S5, smooth the historical coordinates of each target object to calculate the actual three-dimensional space coordinates and combine them with the detection label and bounding box coordinates to output the target detection result of the YOLO model;

[0012] S7, input the target detection results of S6 into the task decision model to convert the task information into structured instructions;

[0013] S8, the manipulator performs the grasping action according to the structured instructions of S7 and calculates and adjusts the grasping force.

[0014] Preferably, in S4, the categories and y-axis horizontal coordinates of the objects in the left and right images are compared to determine whether they are the same object. If they are the same object, the difference in the y-axis horizontal coordinates of the objects in the left and right images is calculated, and the object with the smallest difference is selected for matching; if a matching object cannot be found, it is regarded as an unmatched object. At this time, the camera position needs to be adjusted based on the two-dimensional coordinates of the object and start again from S1.

[0015] Preferably, the calculation process of the parallax between the left and right images and the depth information of the object in S5 is as follows:

[0016] Parallax Calculation formula:

[0017] ; (1)

[0018] in, and are the horizontal pixel coordinates of the target objects in the left and right images respectively;

[0019] depth Z Calculation formula:

[0020] ; (2)

[0021] in, B is the baseline of the camera in millimeters, is the focal length of the camera in millimeters; 100 is the unit conversion factor.

[0022] Preferably, the formula for calculating the actual three-dimensional space coordinates in S6 is as follows:

[0023] ; (3)

[0024] ; (4)

[0025] in, and is the center point coordinate of the target object detected in the image, is the pixel size in mm / pixel, ( ) is the actual three-dimensional space coordinate of the transformation.

[0026] Preferably, the specific content of the task decision model in S7 is as follows:

[0027] The S71 and GPT-4o models use the Transformer structure to encode the input natural language task instructions, extract contextual semantic information, and combine pre-trained knowledge to understand the task. The visual encoder VisionEncoder in the model extracts features from the input image, generates a high-dimensional visual feature vector, and annotates key targets such as object detection and semantic segmentation. The shared latent space is used to align text and visual features to ensure that the information of the two modalities is interconnected.

[0028] S72, based on the knowledge base pre-trained by the GPT-4o model, combined with Prompt-based Learning, parses user instructions, identifies core task objectives and constraints, and uses dependency analysis and entity recognition (NER) methods to break down executable task information into structured instructions for the robot to execute.

[0029] Preferably, the specific process of calculating and adjusting the gripping force of the manipulator in S8 is as follows:

[0030] S81. Preliminarily determine the degree of opening and closing of the gripper based on the geometric information of the target object obtained by the binocular vision technology in S1. Then, dynamically estimate the mass of the target object by combining the information fed back by the three-dimensional tactile sensor installed on the manipulator. The specific process is as follows:

[0031] In a stationary state, after the gripper lifts the target object, the 3D tactile sensor can sense:

[0032] ; (5)

[0033] in, is the tangential force of the gripper, is the mass of the object, is accelerated by gravity;

[0034] Estimated quality for:

[0035] ; (6)

[0036] S82. The process and formula for calculating the gripping force are as follows:

[0037] ; (7)

[0038] in, is the normal force applied by each side of the gripper, is the friction coefficient between the gripper and the target object;

[0039] S83. Determine whether the gripper has slipped out of the gripper based on whether the tangential force exceeds the friction limit. The specific determination formula is as follows:

[0040] ; (8)

[0041] in, is the current normal force, is the current tangential force, For the gripper Tangential force in the axial direction, For the gripper Tangential force in the axial direction;

[0042] S84. If the result of S83 is slipping, adjust the gripping force according to the estimated mass in S71. The adjustment formula is as follows:

[0043] ; (9)

[0044] in, is the safety factor, and its value range is .

[0045] The present invention provides a control device for a multimodal intelligent manipulator, comprising a base, to which a large arm fixing seat, a large arm, an intermediate arm, a small arm and an end effector are sequentially connected above the base, a clamp interface is provided on the end effector, and a power supply interface, a bus interface, a safety switch interface and a switching output interface are respectively provided on the side of the base.

[0046] Preferably, the maximum horizontal rotation radius of the manipulator device is set to 661.5 mm, the vertical movement range is set to 1122 mm, and the interference radius of the base is set to 120 mm.

[0047] Therefore, the present invention adopts a control method and device for a multimodal intelligent manipulator as described above, which has the following beneficial effects compared with the prior art:

[0048] 1. The control method of the multimodal intelligent manipulator in this application utilizes multimodal fusion technology to fuse information, thereby greatly improving its flexibility, intelligence, and human-computer interaction capabilities. It can fuse multimodal information to achieve more efficient, more accurate, and more natural intelligent operation, and is suitable for multiple application scenarios such as industrial manufacturing, medical assistance, agricultural automation, and home services.

[0049] 2. The end effector of the manipulator device in this application can be equipped with a variety of end tools (such as grippers, suction cups, etc.) to realize specific operating functions. The overall mechanical structure is compact and the joints are flexible. It is suitable for desktop work scenes or loaded onto mobile platforms (such as AGV or humanoid manipulators). It has high stability and easy operation. The modular design of the manipulator facilitates maintenance and expansion, providing users with highly customized application possibilities.

[0050] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flowchart of a control method for a multi-modal intelligent manipulator of the present invention;

[0052] Figure 2 This is an overall structural diagram of a control device for a multi-modal intelligent manipulator of the present invention;

[0053] Reference numerals

[0054] 1. Base; 2. Upper arm fixing seat; 3. Upper arm; 4. Intermediate arm; 5. Lower arm; 6. End effector; 7. Fixture interface; 8. Power interface; 9. Bus interface; 10. Safety switch interface; 11. Switch output interface. DETAILED DESCRIPTION

[0055] In the description of the present invention, it should be noted that the terms "upper", "lower", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the inventive product is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they should not be understood as limiting the present invention.

[0056] Example

[0057] like Figure 1 As shown, a control method of a multimodal intelligent manipulator of the present invention comprises the following steps:

[0058] S1, based on binocular vision technology, two cameras are used to capture the left and right images of the target object in the same scene;

[0059] S2. Identify and locate the target object through the YOLO model, and output the category and bounding box coordinates of the target object;

[0060] S3, extract the center point coordinates of the target object in the left and right images based on S1 and S2;

[0061] S4. Match objects in the left and right images that have the same category and similar y-axis horizontal coordinates as the target object through S2 and S3. By comparing the categories and y-axis horizontal coordinates of the objects in the left and right images, determine whether they are the same object. If they are the same object, calculate the difference in the y-axis horizontal coordinates of the objects in the left and right images, and select the object with the smallest difference for matching. If no matching object can be found, it is considered an unmatched object. At this time, the camera position needs to be adjusted based on the two-dimensional coordinates of the object and restart from S1.

[0062] S5. Estimate the three-dimensional spatial coordinates of the target object using the parallax between the left and right images and the depth information of the object. The calculation process of the parallax between the left and right images and the depth information of the object is as follows:

[0063] Parallax Calculation formula:

[0064] ; (1)

[0065] in, and are the horizontal pixel coordinates of the target objects in the left and right images respectively;

[0066] depth Z Calculation formula:

[0067] ; (2)

[0068] in, B is the baseline of the camera in millimeters, is the focal length of the camera in millimeters; 100 is the unit conversion factor;

[0069] S6. Record the historical coordinates of S5, smooth the historical coordinates of each target object to calculate the actual three-dimensional space coordinates, and combine them with the detection label and bounding box coordinates to output the target detection result of the YOLO model; the formula for calculating the actual three-dimensional space coordinates is as follows:

[0070] ; (3)

[0071] ; (4)

[0072] in, and is the center point coordinate of the target object detected in the image, is the pixel size in mm / pixel, ( ) is the actual three-dimensional space coordinate of the transformation;

[0073] S7. Input the target detection results of S6 into the task decision model to convert the task information into structured instructions. The specific content of the task decision model is as follows:

[0074] The S71 and GPT-4o models use the Transformer structure to encode the input natural language task instructions, extract contextual semantic information, and combine pre-trained knowledge to understand the task. The visual encoder VisionEncoder in the model extracts features from the input image, generates a high-dimensional visual feature vector, and annotates key targets such as object detection and semantic segmentation. The shared latent space is used to align text and visual features to ensure that the information of the two modalities is interconnected.

[0075] S72, based on the knowledge base pre-trained by the GPT-4o model, combined with prompt-based learning, parses user instructions, identifies core task objectives and constraints, and uses dependency analysis and entity recognition (NER) methods to break down executable task information into structured instructions for the robot to execute;

[0076] S8, the manipulator performs the grasping action and calculates and adjusts the grasping force according to the structured instruction of S6;

[0077] The specific process of the manipulator calculating the grasping force is as follows:

[0078] S81. Preliminarily determine the degree of opening and closing of the gripper based on the geometric information of the target object obtained by the binocular vision technology in S1. Then, dynamically estimate the mass of the target object by combining the information fed back by the three-dimensional tactile sensor installed on the manipulator. The specific process is as follows:

[0079] In a stationary state, after the gripper lifts the target object, the 3D tactile sensor can sense:

[0080] ; (5)

[0081] in, is the tangential force of the gripper, is the mass of the object, is accelerated by gravity;

[0082] Estimated quality for:

[0083] ; (6)

[0084] S82. The process and formula for calculating the gripping force are as follows:

[0085] ; (7)

[0086] in, is the normal force applied by each side of the gripper, is the friction coefficient between the gripper and the target object;

[0087] S83. Determine whether the gripper has slipped out of the gripper based on whether the tangential force exceeds the friction limit. The specific determination formula is as follows:

[0088] ; (8)

[0089] in, is the current normal force, is the current tangential force, For the gripper Tangential force in the axial direction, For the gripper Tangential force in the axial direction;

[0090] S84. If the result of S83 is slipping, adjust the gripping force according to the estimated mass in S71. The adjustment formula is as follows:

[0091] ; (9)

[0092] in, is the safety factor, and its value range is .

[0093] like Figure 2 As shown, the present invention also provides a control device for a multimodal intelligent manipulator, including a base 1, a large arm fixing seat 2, a large arm 3, an intermediate arm 4, a small arm 5 and an end effector 6 are sequentially connected to the top of the base 1, a clamp interface 7 is provided on the end effector 6, and a power interface 8, a bus interface 9, a safety switch interface 10 and a switching output interface 11 are respectively provided on the side of the base 1. The base 1 not only provides support and stabilization functions for the manipulator, but also includes multiple interfaces, such as a power interface 8, a bus interface 9, a safety switch interface 10 and a switching output interface 11, so as to realize power supply, signal transmission and external device control. The power interface 8 is connected to a power adapter, and the safety switch interface 9 is connected to a power adapter. The safety interface 10 is connected to the short-circuit interface plug, and the user can also lead out the wiring to an external switch. When the safety switch interface 10 is short-circuited to GND, the manipulator can move normally. If the safety switch is open to GND, the manipulator immediately stops moving and waits for the signal to be short-circuited to the ground before continuing to move. The manipulator's arm 3 and arm 5 are connected through multiple joints to form a flexible motion structure. Among them, the arm 3 has an integrated controller embedded in it, which provides the manipulator with efficient motion control and signal processing capabilities. The arm 5 is connected to the arm 3 through the intermediate arm 4, and is connected to the fixture interface 7 of the end effector 6. It can be installed with a variety of end tools (such as grippers, suction cups, etc.) to realize specific operating functions.

[0094] The maximum horizontal rotation radius of the manipulator device is set to 661.5 mm, the vertical motion range is set to 1122 mm, and the interference radius of base 1 is set to 120 mm. The manipulator has excellent motion performance and high-precision control capabilities, as shown in Table 1.

[0095] Table 1 Mechanical data details of the manipulator device

[0096] ;

[0097] During the specific implementation process, the operations of controlling the robot through software are as follows;

[0098] 1. Robot reset calibration

[0099] Open the robot control software, select the serial port number to which the robot is connected, and click Open Serial Port. Drag the "Robot Calibration" action from the Action List to the Selected Actions list. Click the "Execute Action" button, and the robot will automatically begin calibration. After starting automatic calibration, each axis will rotate in the predetermined direction, stopping when each axis reaches the corresponding positioning switch. When all axes reach the positioning point, the robot rotates to the fixed starting posture.

[0100] After the robot is reset or calibrated, it enters the waiting state for motion commands. If it receives a motion command from the host computer, the robot immediately calculates and interpolates from its current position to the target point. If there is no motion command, the robot remains in its current position. Simultaneously, the robot sends real-time status information, including positioning coordinates and axis angles, to the host computer.

[0101] 2. Basic process motion control

[0102] After opening the serial port using the software, the robot's current 3D coordinates (X, Y, Z) and the angles of each axis (a1-a6) will be displayed at the top of the software interface. Users can control the robot by changing the 3D coordinates or the angles of each axis. The 3D coordinates are in millimeters (mm), and the angles are in degrees. Drag the "Change Coordinates" function from the Action list to the Selected Actions list. Enter the desired coordinates (X, Y, Z) in the function's input box, separating the 3D coordinates with commas. Then, click the "Execute Action" button at the bottom of the software, and the robot will automatically move to the corresponding angle.

[0103] 3. Angle setting of the robot end effector

[0104] When using a robot arm to grasp objects, sometimes you need to grasp them from different angles. For example, you can grasp spheres such as small balls directly from above, while grasping cylinders such as bottles requires grasping them at a slightly tilted angle to the horizontal. Move the "Change Gripper Terminal Angle" function to the front before using the "Change Coordinates" function.

[0105] The angle in the "Change Gripper End Angle" function represents the angle between the robot's end effector and the ground. If set to 0 degrees, the robot will grab the object from above, and if set to 90 degrees, the robot will grab the object from the horizontal direction.

[0106] 4. Robot single-axis angle setting

[0107] The "Change Single Axis Angle" function changes the angle of a specific axis while keeping the other angles unchanged. Drag the "Change Single Axis Angle" function from the Action list to the Selected Actions list, enter the desired axis and angle, and click the "Execute Action" button at the bottom of the software. The robot arm will then change its angle accordingly.

[0108] 5. Call the camera

[0109] The VB-1 vision platform is equipped with a dedicated visual camera and algorithm for target detection and target 3D coordinate detection for the robot. The camera call steps are as follows:

[0110] (a) Drag the "Open Camera for Object Detection" function from the Action list to the Selected Actions list;

[0111] (b) Enter the name of the object you want to detect in the action list. Click to execute the action. After a while, the camera's detection results will pop up.

[0112] 6. Large model + visual control robot

[0113] The software supports using a binocular camera to provide visual information for the robot, and using a large model to assist the user in manipulating the robot. The hand follows the instruction by dragging the "Visual Large Language Model Understanding Instruction Function Use" function from the action list to the selected action list, and then inputting "Robot follows hand movement" to execute the action.

[0114] After a short wait, the left and right camera images will pop up. The left camera will display the detected hand box, the detection confidence, and the hand's 3D coordinates. At this point, press the "a" key on the keyboard to start the robot arm following the hand's movements. Press the "q" key to stop the robot arm. This algorithm detects 3D coordinates by exploiting the parallax between the two cameras. Therefore, the hand must be visible in both cameras simultaneously to obtain 3D coordinates; otherwise, only 2D coordinates will be obtained.

[0115] 7. Semantic Understanding of Large Language Models

[0116] Drag the "Visual Large Language Model Understanding Instruction Function Use" function from the action list to the selected action list, enter the user's command, such as "Pour water from the bottle into the cup", and then execute the action.

[0117] After a short wait, the left and right camera images will pop up, detecting the desired object in the image, such as a bottle or cup. At this point, press the "a" key on the keyboard, and the robot will start to execute the user's instructions. Press the "q" key on the keyboard, and the robot will stop moving.

[0118] Therefore, the present invention adopts a control method and device for a multimodal intelligent manipulator as described above. Through deep learning and intelligent control algorithms, the manipulator can autonomously identify targets, optimize execution paths, and adapt to environmental changes. It can integrate multiple modal information to achieve more efficient, more accurate, and more natural intelligent operations. It is suitable for multiple application scenarios such as industrial manufacturing, medical assistance, agricultural automation, and home services, and provides more personalized and intelligent operation methods according to user needs.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A control method for a multimodal intelligent manipulator, characterized in that: The following steps are involved: S1, based on binocular vision technology, two cameras are used to capture the left and right images of the target object in the same scene; S2. Identify and locate the target object through the YOLO model, and output the category and bounding box coordinates of the target object; S3, extract the center point coordinates of the target object in the left and right images based on S1 and S2; S4, through S2 and S3, match the categories of the target objects in the left and right images. y The object with the closest axis transverse coordinate; S5, calculating the disparity between the left and right images and the depth information of the object to combine the three-dimensional space coordinates of the target object; S6, record the historical coordinates of S5, smooth the historical coordinates of each target object to calculate the actual three-dimensional space coordinates and combine them with the detection label and bounding box coordinates to output the target detection result of the YOLO model; S7, input the target detection results of S6 into the task decision model to convert the task information into structured instructions; The S71 and GPT-4o models use the Transformer structure to encode input natural language task instructions, extract contextual semantic information, and combine pre-trained knowledge to understand the task. The Vision Encoder in the model extracts features from the input image, generates a high-dimensional visual feature vector, and annotates key targets. The shared latent space is used to align text and visual features to ensure that the information of the two modalities is interconnected. S72, based on the knowledge base pre-trained by the GPT-4o model, combined with prompt-based learning, parses user instructions, identifies core task objectives and constraints, and uses dependency analysis and entity recognition (NER) methods to break down executable task information into structured instructions for the robot to execute; S8, the manipulator performs the grasping action and calculates and adjusts the grasping force according to the structured instructions of S7; S81. Preliminarily determine the degree of opening and closing of the gripper based on the geometric information of the target object obtained by the binocular vision technology in S1. Then, dynamically estimate the mass of the target object by combining the information fed back by the three-dimensional tactile sensor installed on the manipulator. The specific process is as follows: In a stationary state, after the gripper lifts the target object, the 3D tactile sensor can sense: ； （5） in, is the tangential force of the gripper, m is the mass of the object, is accelerated by gravity; Estimated quality for: ；（6） S82. The process and formula for calculating the gripping force are as follows: ；（7） in, is the normal force applied by both sides of the gripper, μ is the friction coefficient between the gripper and the target object; S83. Determine whether the gripper has slipped out of the gripper based on whether the tangential force exceeds the friction limit. The specific determination formula is as follows: ；（8） in, is the current normal force, is the current tangential force, For the gripper x Tangential force in the axial direction, For the gripper y Tangential force in the axial direction; S84. If the result of S83 is slipping, adjust the gripping force according to the estimated mass in S71. The adjustment formula is as follows: ；（9） in, is the safety factor, and its value range is =1.2~1.

5.

2. The control method of a multimodal intelligent manipulator according to claim 1, characterized in that: In S4, the categories of objects in the left and right images are compared to determine whether they are the same object. If they are the same object, the object's y The object with the smallest difference in horizontal coordinates is selected for matching; if no matching object is found, it is considered an unmatched object. At this time, the camera position needs to be adjusted and restarted from S1 based on the two-dimensional coordinates of the object.

3. The control method of a multimodal intelligent manipulator according to claim 2, characterized in that: The calculation process of the disparity between the left and right images and the depth information of the object in S5 is as follows: Parallax Calculation formula: ；（1） in, and are the horizontal pixel coordinates of the target objects in the left and right images respectively; depth Z Calculation formula: ；（2） in, B is the baseline of the camera in millimeters, f is the focal length of the camera in millimeters; 100 is the unit conversion factor.

4. The control method of a multimodal intelligent manipulator according to claim 3, characterized in that: The formula for calculating the actual three-dimensional space coordinates described in S6 is as follows: ；（3） ； （4） in, and is the center point coordinate of the target object detected in the image, pixel_size is the pixel size in mm / pixel, ( ) is the actual three-dimensional space coordinate of the transformation.

Images