A method for intelligent grasping manipulator based on monocular depth estimation and multimodal positioning

Abstract

The present invention is a method for intelligent grasping manipulator based on monocular depth estimation and multimodal positioning. By combining a monocular camera and a multimodal algorithm model, the three-dimensional position positioning and grasping tasks of the target object are realized. The system does not require a preset environment and has strong versatility and robustness. It is characterized in that the method includes the following steps: 1) parsing of voice instructions; 2) customization of the algorithm model; 3) scene understanding and target positioning; 4) conversion of 2D pixel coordinates of the target object to actual coordinates; 5) estimation of the depth coordinates of the target object; 6) actual compensation of multiple parameters; 7) motion control of the manipulator. The present invention realizes the three-dimensional position positioning and grasping tasks of the target object by combining a monocular camera and a multimodal algorithm model. The system does not require a preset environment and has strong versatility and robustness.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for an intelligent grasping mechanical arm based on monocular depth estimation and multimodal positioning. Background Art

[0002] At present, robotic arm grasping technology has been widely used in industrial production, logistics distribution, laboratory automation and other fields. The relevant technologies mainly include the following:

[0003] 1. Robotic arm technology based on fixed grasping points

[0004] Most robotic arms rely on preset gripping points and fixed operation paths. For example, in an assembly line environment, the gripping points of the robotic arm are manually calibrated in advance or dynamically acquired by multiple sensors (such as RGB-D cameras or lidar). The main features of this type of technology are:

[0005] 1) Fixed: The grasping target and position must be defined in advance or provided dynamically by external devices.

[0006] 2) Reliance on multiple sensors: Devices such as binocular cameras, depth cameras or lidar are used to reconstruct three-dimensional information of spatial position.

[0007] 3) Application limitations: It is difficult to cope with the complex grasping requirements of multiple objects in unknown scenes or dynamic environments.

[0008] 2. Object detection and grasping based on deep learning

[0009] Using target detection algorithms (such as the YOLO series, Faster R-CNN, etc.), some technologies can automatically detect and capture targets in images and estimate their 3D positions based on sensor data. The disadvantages of this type of technology include:

[0010] 1) Single task: The target detection algorithm can only identify the objects contained in the training set and cannot understand and identify the scene.

[0011] 2) Hardware dependence: requires binocular cameras, lidar or sensors, which increases costs.

[0012] 3. Voice control and multimodal collaboration

[0013] Some research has attempted to combine speech recognition with robotic arm control, but its application scope is relatively limited:

[0014] 1) The simplicity of voice control: most of them are simple grasping commands (such as "grab object A"), lacking the ability to understand complex scenes and semantic parsing.

[0015] 2) Lack of scene perception: Unable to combine image semantic information and command understanding tasks required.

[0016] In the existing technology, although researchers have realized a variety of methods for robotic arms to grasp targets, the following problems and defects are common:

[0017] High hardware cost: Reliance on multi-sensor devices or high-performance hardware increases system complexity and deployment costs.

[0018] Limited operating environment: Most technologies cannot adapt to grasping tasks in unpredictable environments.

[0019] Insufficient multimodal collaboration: The collaboration between voice, image, and control is weak, resulting in poor system interactivity.

[0020] To this end, a method of intelligent grasping robotic arm based on monocular depth estimation and multimodal positioning is designed to overcome the above problems. Summary of the invention

[0021] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a low-cost, versatile and intelligent grasping robot arm method based on monocular depth estimation and multimodal positioning suitable for complex dynamic scenes. The present invention does not rely on preset targets or environments, and realizes the three-dimensional position positioning and grasping tasks of target objects by combining monocular cameras and multimodal algorithm models. The system does not require a preset environment and has strong versatility and robustness.

[0022] The present invention is implemented through the following technical solutions: a method for an intelligent grasping manipulator based on monocular depth estimation and multimodal positioning, which realizes the three-dimensional position positioning and grasping tasks of a target object by combining a monocular camera and a multimodal algorithm model. The system does not require a preset environment and has strong versatility and robustness. The method comprises the following steps:

[0023] 1) Parsing of voice commands: Users input voice commands through a microphone, which are transcribed into text by a speech recognition model trained with an open source or custom dataset. At the same time, a monocular camera with a fixed top-down angle captures scene images, and the images and text are input into a large multimodal model. In parallel, the images are input into a monocular depth estimation algorithm.

[0024] 2) Customization of algorithm models: In general scenarios, open source multimodal models and monocular depth estimation algorithms can be used, and algorithm models can also be fine-tuned for specific scenarios;

[0025] 3) Scene understanding and target positioning: The multimodal large model combines the target object and relative position relationship in the scene image understanding instruction to locate and output the pixel coordinates of the detection frame of the target object; the monocular depth estimation algorithm predicts the depth value of the scene image, and the average depth value of the corresponding target can be obtained based on the detection frame coordinates of the located target object;

[0026] 4) Conversion of the target object's 2D pixel coordinates to actual coordinates: The system calculates the target object's actual coordinates (x, y) in the plane by combining the previously calibrated pixel coordinates with the actual coordinates and the pixel coordinates of the actual scene;

[0027] 5) Estimation of the depth coordinates of the target object: The system uses the depth values ​​of the pixels within the target object detection frame area to calculate the average depth of its surface, combined with the calibration mapping relationship between the camera and the actual space depth. The calculation principle of the depth mapping relationship is the same as the calculation method of the pixel coordinate mapping, so as to obtain the actual depth coordinates (z) of the target object;

[0028] 6) Multi-parameter actual compensation: According to the model parameters of the monocular camera, the edge of the camera's image viewing angle needs to be compensated for distortion. The specific camera needs to be analyzed to ensure the mapping accuracy of the three-dimensional coordinates. The size of the actual robotic arm grasping device needs to be considered, and the suction pump, mechanical claw, etc. need to be compensated in a targeted manner to ensure the reliability of grasping objects.

[0029] 7) Robotic arm motion control: The system integrates (x, y, z) coordinates to form a three-dimensional grasping point instruction. Through the API interface provided by the Raspberry Pi, the three-dimensional coordinates of the grasping point are input into the robotic arm to control the robotic arm to reach the specified position and perform the grasping or placing operation.

[0030] Preferably, the conversion in step 4) includes: calculating a ratio of the mapping relationship between two sets of known pixel coordinates and actual coordinates; using the pixel reference point and the actual reference point as a reference, and converting the pixel coordinates of the target object into plane actual coordinates through the ratio.

[0031] Preferably, the specific calculation process is as follows:

[0032] Step 1: Calculate pixel distance and actual distance:

[0033]

[0034] Step 2: Find the scaling factor:

[0035]

[0036] Step 3: Calculate the actual coordinates based on the input pixel coordinates:

[0037] X mc =X a_ref +(Y im -Y p_ref )·ratio

[0038] Y mc =Y a_ref +(X im -X p_ref)·ratio.

[0039] The beneficial effects of the present invention are as follows:

[0040] Compared with the existing mechanical arm grasping, the present invention has the following advantages:

[0041] 1. Realize 3D positioning at low cost: The present invention only uses an ordinary monocular camera and a monocular depth estimation algorithm to calculate the 3D position of the target through the mapping relationship between pixel coordinates and actual coordinates and depth. It does not require high-cost equipment such as binocular cameras or lidars, thus greatly reducing hardware costs.

[0042] 2. Multimodal collaboration improves task complexity: Using multimodal models, the system can understand complex semantic instructions, locate the target object based on image information, and achieve intelligent matching of the target and the position. For example, it can handle instructions that include relative position relationships such as "put A to the left of B".

[0043] 3. Strong versatility and adaptability: The system does not require preset grasping targets or scenes and is suitable for grasping tasks in unknown environments, which significantly improves the versatility and robustness of the system.

[0044] 4. Simplified deployment and operation: Through the collaboration of low-cost devices such as Raspberry Pi and servers, lightweight deployment is supported. Users can complete complex tasks through voice commands. The operation is simple and the usage threshold is lowered.

[0045] 5. Improve efficiency and accuracy: The present invention combines the monocular depth estimation algorithm and mapping relationship to quickly and accurately calculate the three-dimensional spatial position of the target, thereby improving the efficiency and accuracy of the robotic arm grasping.

[0046] 6. Energy saving and environmental protection: The system achieves efficient computing and low-power operation through the combination of software and hardware, which is suitable for the energy-saving needs of embedded systems. At the same time, it reduces the use of environmental sensors and is more environmentally friendly.

[0047] In summary, the present invention not only realizes technical innovation, but also takes into account cost, versatility and user-friendliness, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the overall structure of the present invention. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to more clearly understand the objectives, technical solutions and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0050] In the description of the present invention, it is necessary to understand that the orientation or position relationship indicated by the terms such as "up", "down", "left", "right", "inside", "outside", "horizontal" and "vertical" is based on the orientation or position relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention, rather than indicating or implying that the device or original referred to must have a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0051] The present invention will be described in detail below with reference to the accompanying drawings: Figure 1 As shown, a method for an intelligent grasping robot arm based on monocular depth estimation and multimodal positioning realizes the three-dimensional position positioning and grasping tasks of the target object by combining a monocular camera and a multimodal algorithm model. The system does not require a preset environment and has strong versatility and robustness. The method includes the following steps:

[0052] 1) Parsing of voice commands: Users input voice commands through a microphone, which are transcribed into text by a speech recognition model trained with an open source or custom dataset. At the same time, a monocular camera with a fixed top-down angle captures scene images, and the images and text are input into a large multimodal model. In parallel, the images are input into a monocular depth estimation algorithm.

[0053] 2) Customization of algorithm models: In general scenarios, open source multimodal models and monocular depth estimation algorithms can be used, and algorithm models can also be fine-tuned for specific scenarios;

[0054] 3) Scene understanding and target positioning: The multimodal large model combines the target object and relative position relationship in the scene image understanding instruction to locate and output the pixel coordinates of the detection frame of the target object; the monocular depth estimation algorithm predicts the depth value of the scene image, and the average depth value of the corresponding target can be obtained based on the detection frame coordinates of the located target object;

[0055] 4) Conversion of the target object's 2D pixel coordinates to actual coordinates: The system calculates the target object's actual coordinates (x, y) in the plane by combining the previously calibrated pixel coordinates with the actual coordinates and the pixel coordinates of the actual scene;

[0056] 5) Estimation of the depth coordinates of the target object: The system uses the depth values ​​of the pixels within the target object detection frame area to calculate the average depth of its surface, combined with the calibration mapping relationship between the camera and the actual space depth. The calculation principle of the depth mapping relationship is the same as the calculation method of the pixel coordinate mapping, so as to obtain the actual depth coordinates (z) of the target object;

[0057] 6) Multi-parameter actual compensation: According to the model parameters of the monocular camera, the edge of the camera's image viewing angle needs to be compensated for distortion. The specific camera needs to be analyzed to ensure the mapping accuracy of the three-dimensional coordinates. The size of the actual robotic arm grasping device needs to be considered, and the suction pump, mechanical claw, etc. need to be compensated in a targeted manner to ensure the reliability of grasping objects.

[0058] 7) Robotic arm motion control: The system integrates (x, y, z) coordinates to form a three-dimensional grasping point instruction. Through the API interface provided by the Raspberry Pi, the three-dimensional coordinates of the grasping point are input into the robotic arm to control the robotic arm to reach the specified position and perform the grasping or placing operation.

[0059] The conversion in step 4) includes: calculating the ratio of the mapping relationship between two sets of known pixel coordinates and actual coordinates (the ratio of pixel distance to actual distance); using the pixel reference point and the actual reference point as a reference, and converting the pixel coordinates of the target object into the plane actual coordinates through the ratio.

[0060] The specific calculation process is as follows:

[0061] Step 1: Calculate pixel distance and actual distance:

[0062]

[0063] Step 2: Find the scaling factor:

[0064]

[0065] Step 3: Calculate the actual coordinates based on the input pixel coordinates:

[0066] X mc =X a_ref +(Y im -Y p_ref )·ratio

[0067] Y mc =Y a_ref +(X im -X p_ref )·ratio.

[0068] The characteristics of the present invention are as follows:

[0069] 1. Achieve cost-effective 3D location awareness

[0070] The present invention uses a monocular camera to acquire scene images, combines a monocular depth estimation algorithm and the mapping relationship between pixel coordinates and actual coordinates, and accurately calculates the three-dimensional position (x, y, z) of the target object, avoiding the use of a binocular camera or other high-cost depth sensors.

[0071] 2. Implementing multimodal semantic instructions and scene understanding

[0072] By parsing user voice commands through a large multimodal model and combining them with scene images, the target object and its relative position can be accurately located (such as "put object A to the left of object B"), meeting the execution requirements of complex semantic commands.

[0073] 3. Realize intelligent control of the robotic arm

[0074] Based on the calculated 3D coordinates of the target, the motion trajectory and task execution of the robotic arm are controlled through the API interface provided by the Raspberry Pi, supporting precise grasping and accurate placement of target objects.

[0075] 4. Enhance the versatility and robustness of the system

[0076] The present invention does not rely on preset targets or environments, can adapt to dynamic and changing scenarios, supports the positioning and operation of multiple targets, and has high flexibility and robustness.

[0077] Example 1

[0078] The specific steps of the intelligent grasping robot arm in life scenarios include:

[0079] 1. Voice command analysis: Voice recognition uses the Whisper model trained with a custom data set. Commands are input through the microphone, such as "Please bring the coffee powder to the second cup from the left to the right." You can also input commands that require deeper understanding, such as "What should I do if the coffee in the first cup on the left is too bitter?" The system transcribes the speech into text through the voice recognition model, combines it with the image taken by the monocular camera, and inputs it into the multimodal large model and monocular depth estimation algorithm.

[0080] 2. Scene image understanding and positioning: The multimodal large model uses Qwen2-VL fine-tuned with custom life scene data to identify objects in the scene. Deep-level instructions need to be given corresponding solutions, such as "the coffee in the first cup on the left is too bitter" and needs "sugar", so the output detection frame is the area near the first cup on the left and objects related to "sugar" in the scene, such as a sugar jar, so as to obtain the pixel coordinates (px1, py1) and (px2, py2) of the target object or object. Use indoor data to train the monocular depth estimation algorithm Depth Anything, generate a depth map of the scene, and extract the average depth values ​​(z1) and (z2) in the target object detection frame area.

[0081] 3. Conversion from pixel coordinates to actual coordinates: The system uses the mapping relationship between calibrated pixels and actual coordinates to convert the detected pixel coordinates (px1, py1) of the target object into actual coordinates (x1, y1, z1) and (x2, y2, z2).

[0082] 4. Robotic arm motion control: The system generates the grabbing and placing instructions for the robot arm. The instructions include the grabbing point (x1, y1, z1) and the placing point (x2, y2, z2).

[0083] 5. Task completion verification: The system uses the camera to capture the final scene image and verify whether the target object 1 is correctly placed in the target object 2 area, for example, whether the sugar bowl is placed near the first cup on the left. If the position deviation exceeds the set threshold, it will automatically adjust and re-execute the placement operation.

[0084] Anyone familiar with the technology may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for intelligent grasping manipulator based on monocular depth estimation and multimodal positioning, which realizes the three-dimensional position positioning and grasping tasks of the target object by combining a monocular camera and a multimodal algorithm model. The system does not require a preset environment and has strong versatility and robustness. It is characterized by: The method comprises the following steps: 1) Parsing of voice commands: Users input voice commands through a microphone, which are transcribed into text by a speech recognition model trained with an open source or custom dataset. At the same time, a monocular camera with a fixed top-down angle captures scene images, and the images and text are input into a large multimodal model. In parallel, the images are input into a monocular depth estimation algorithm. 2) Customization of algorithm models: In general scenarios, open source multimodal models and monocular depth estimation algorithms can be used, and algorithm models can also be fine-tuned for specific scenarios; 3) Scene understanding and target positioning: The multimodal large model combines the target object and relative position relationship in the scene image understanding instruction to locate and output the pixel coordinates of the detection frame of the target object; the monocular depth estimation algorithm predicts the depth value of the scene image, and the average depth value of the corresponding target can be obtained based on the detection frame coordinates of the located target object; 4) Conversion of the target object's 2D pixel coordinates to actual coordinates: The system calculates the target object's actual coordinates (x, y) in the plane by combining the previously calibrated pixel coordinates with the actual coordinates and the pixel coordinates of the actual scene; 5) Estimation of the depth coordinates of the target object: The system uses the depth values ​​of the pixels within the target object detection frame area to calculate the average depth of its surface, combined with the calibration mapping relationship between the camera and the actual space depth. The calculation principle of the depth mapping relationship is the same as the calculation method of the pixel coordinate mapping, so as to obtain the actual depth coordinates (z) of the target object; 6) Multi-parameter actual compensation: According to the model parameters of the monocular camera, the edge of the camera's image viewing angle needs to be compensated for distortion. The specific camera needs to be analyzed to ensure the mapping accuracy of the three-dimensional coordinates. The size of the actual robotic arm grasping device needs to be considered, and the suction pump and mechanical claw need to be compensated in a targeted manner to ensure the reliability of grasping objects. 7) Robotic arm motion control: The system integrates (x, y, z) coordinates to form a three-dimensional grasping point instruction. Through the API interface provided by the Raspberry Pi, the three-dimensional coordinates of the grasping point are input into the robotic arm to control the robotic arm to reach the specified position and perform the grasping or placing operation.

2. The method of intelligent grasping robot arm based on monocular depth estimation and multimodal positioning according to claim 1, characterized in that: The conversion in step 4) includes: calculating a ratio of the mapping relationship between two sets of known pixel coordinates and actual coordinates; using the pixel reference point and the actual reference point as a reference, and converting the pixel coordinates of the target object into the plane actual coordinates through the ratio.

3. The method of intelligent grasping mechanical arm based on monocular depth estimation and multimodal positioning according to claim 2, characterized in that: The specific calculation process is as follows: Step 1: Calculate pixel distance and actual distance: Step 2: Find the scaling factor: Step 3: Calculate the actual coordinates based on the input pixel coordinates: X mc =X a_ref +(And im -AND p_ref )·ratio AND mc =And a_ref +(X im -X p_ref )·ratio.

Images