Mechanical arm visual grabbing method, device and equipment and storage medium

Abstract

The invention discloses a mechanical arm visual grabbing method, device and equipment and a storage medium, and relates to the technical field of intelligent control, and the mechanical arm visual grabbing method comprises the steps that a color image and a depth map are collected; performing target detection on the color image to obtain two-dimensional position information of a target object; according to the two-dimensional position information and the depth map, three-dimensional coordinates of the target object in the mechanical arm operation space are determined; and based on the three-dimensional coordinates, the mechanical arm is controlled to execute the grabbing action. The grabbing stability of the mechanical arm can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent control, in particular to a mechanical arm vision grasping method, device, equipment and storage medium. BACKGROUND

[0002] In the field of industrial automation, vision-guided mechanical arm grasping technology has become an important research direction. However, the traditional two-dimensional vision information lacks depth perception, resulting in low stability of mechanical arm grasping. Therefore, how to improve the stability of mechanical arm grasping is still a problem to be solved.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a mechanical arm vision grasping method, device, equipment and storage medium, aiming at solving the technical problem of improving the stability of mechanical arm grasping.

[0005] To achieve the above purpose, the present application provides a mechanical arm vision grasping method, which comprises: collecting a color image and a depth map; detecting a target object in the color image to obtain two-dimensional position information of the target object; determining three-dimensional coordinates of the target object in a mechanical arm operation space according to the two-dimensional position information and the depth map; controlling the mechanical arm to perform a grasping action based on the three-dimensional coordinates.

[0006] In an embodiment, the step of detecting a target object in the color image to obtain two-dimensional position information of the target object comprises: inputting the color image into a lightweight target detection network; outputting the boundary box coordinates and the center point pixel coordinates of the target object by forward reasoning of the lightweight target detection network.

[0007] In an embodiment, the backbone network of the lightweight target detection network adopts a multi-branch parallel convolution structure, and the neck network of the lightweight target detection network integrates a feature enhancement module based on differential convolution.

[0008] In an embodiment, the step of determining three-dimensional coordinates of the target object in a mechanical arm operation space according to the two-dimensional position information and the depth map comprises: extracting the depth value of the corresponding point from the depth map according to the pixel coordinates in the two-dimensional position information; Project the pixel coordinates and the depth values based on an intrinsic matrix of a camera to obtain three-dimensional coordinates in a camera coordinate system; Convert the three-dimensional coordinates in the camera coordinate system to three-dimensional coordinates in a robot base coordinate system through a hand-eye transformation matrix.

[0009] In an embodiment, the step of controlling the robot to perform a grasping action based on the three-dimensional coordinates comprises: Calculating target angles of joints of the robot according to the three-dimensional coordinates; Generating a motion trajectory based on the target angles; Controlling the robot to move along the trajectory and drive an end effector to complete grasping.

[0010] In an embodiment, before the step of performing target detection on the color image to obtain two-dimensional position information of a target object, the method further comprises: Performing distortion correction processing on the color image; Performing brightness enhancement and contrast adjustment on the corrected image.

[0011] In an embodiment, after the step of controlling the robot to perform a grasping action, the method further comprises: Real-time monitoring of a grasping state of the robot through a vision sensor; If it is detected that grasping fails, re-performing target detection and grasping.

[0012] In addition, to achieve the above-mentioned purposes, the present application further provides a robot vision grasping device, which comprises: An acquisition module configured to acquire a color image and a depth map; A processing module configured to perform target detection on the color image to obtain two-dimensional position information of a target object; A determination module configured to determine three-dimensional coordinates of the target object in a robot operating space according to the two-dimensional position information and the depth map; A control module configured to control a robot to perform a grasping action based on the three-dimensional coordinates.

[0013] In addition, to achieve the above-mentioned purposes, the present application further provides a robot vision grasping device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot vision grasping method as described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the mechanical arm visual grasping method.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the mechanical arm visual grasping method.

[0016] The present application provides a mechanical arm visual grasping method, and the present application collects a color image and a depth map; target detection is performed on the color image to obtain two-dimensional position information of a target object; three-dimensional coordinates of the target object in a mechanical arm operation space are determined according to the two-dimensional position information and the depth map; and a mechanical arm is controlled to perform a grasping action based on the three-dimensional coordinates. The present application collects a color image and a depth map; target detection is performed on the color image to obtain two-dimensional position information of a target object; three-dimensional coordinates of the target object in a mechanical arm operation space are determined according to the two-dimensional position information and the depth map; and a mechanical arm is controlled to perform a grasping action based on the three-dimensional coordinates. The present application fuses the target detection result of the color image and the ranging information of the depth map to obtain accurate three-dimensional coordinates of the target object, and uses the coordinates to drive the mechanical arm to complete a closed-loop grasping action, effectively reduces the grasping deviation caused by inaccurate positioning, and improves the stability of the mechanical arm grasping. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are incorporated into and form part of the specification, illustrate an embodiment consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0019] Figure 1 A flowchart is provided for the mechanical arm visual grasping method embodiment one of the present application; Figure 2 A mechanical arm visual grasping structure diagram is provided for the mechanical arm visual grasping method embodiment one of the present application; Figure 3 A flowchart is provided for the mechanical arm visual grasping method embodiment two of the present application; Figure 4 A module structure diagram of the mechanical arm visual grasping device of the present application embodiment is provided. Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the robotic arm vision grasping method in this embodiment of the application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] This application acquires color images and depth maps; performs target detection on the color images to obtain two-dimensional position information of the target object; determines the three-dimensional coordinates of the target object in the robotic arm's operating space based on the two-dimensional position information and the depth map; and controls the robotic arm to perform a grasping action based on the three-dimensional coordinates.

[0024] In the field of industrial automation, vision-guided robotic arm grasping technology has become an important research direction. However, the lack of depth perception in traditional two-dimensional vision information leads to low stability in robotic arm grasping. Therefore, how to improve the stability of robotic arm grasping remains a problem to be solved.

[0025] This application obtains the precise three-dimensional coordinates of the target object by fusing the target detection results of the color image with the ranging information of the depth map, and uses these coordinates to drive the robotic arm to complete the closed-loop grasping action, which effectively reduces the grasping deviation caused by inaccurate positioning, thereby improving the stability of the robotic arm's grasping.

[0026] Based on this, embodiments of this application provide a robotic arm vision grasping method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the robotic arm vision grasping method of this application.

[0027] In this embodiment, the robotic arm vision grasping method includes steps S10 to S40: Step S10: Acquire color images and depth maps; It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or a robotic arm vision grasping device capable of performing the above functions. The following description uses a robotic arm vision grasping device as an example to illustrate this embodiment and the subsequent embodiments.

[0028] It should be noted that depth imaging devices based on structured light principles can acquire color images and depth maps of the work area. The color image is an RGB image, and the depth map is a special type of two-dimensional image. Each pixel value represents the vertical distance from a corresponding point on the surface of an object in the scene to the camera's imaging plane. (See reference...) Figure 2 , Figure 2 This is a schematic diagram of the robotic arm's vision-grabbing structure. Specifically, the depth imaging device can be fixedly mounted on a bracket within the robotic arm's working range. Synchronous acquisition of RGB images and depth maps is achieved via a USB 3.0 high-speed interface. The image acquisition frame rate is no less than 30 frames per second, and the preferred resolution is 640×480 pixels to ensure the stability of detail resolution and time synchronization.

[0029] It should be noted that after acquiring the RGB image and depth map, the timestamps and frame numbers of the two need to be compared to ensure that the image data is completely consistent in space and time.

[0030] Step S20: Perform target detection on the color image to obtain the two-dimensional position information of the target object; It should be noted that object detection networks can be used to detect objects in color images. These networks can identify target objects (such as specific workpieces) in the image and output their two-dimensional position information.

[0031] In one feasible approach, the step of obtaining two-dimensional position information of a target object by performing target detection on the color image includes: inputting the color image into a lightweight target detection network; performing forward inference through the lightweight target detection network to output the bounding box coordinates and center point pixel coordinates of the target object. The backbone network of the lightweight target detection network adopts a multi-branch parallel convolutional structure, and the neck network of the lightweight target detection network integrates a feature enhancement module based on differential convolution.

[0032] It's important to note that before detection, the color image is normalized, for example, by scaling pixel values ​​to the 0-1 range. This processed data is then used as an input tensor and fed into a lightweight object detection network model. The network model performs forward propagation computation. Its output layer provides one or more detection results, each containing: the object category, confidence score, and the pixel coordinates of the top-left and bottom-right corners of the bounding box. Based on the output results, the pixel coordinates of the bounding box's center point can be further calculated. The object detection network supports parallel detection of multiple objects; therefore, the detection boxes can be sorted according to their confidence scores, and the object with the highest confidence score can be selected as the priority target. A non-maximum suppression algorithm is used to remove background interference boxes, avoiding duplicate detections and false positives.

[0033] It should be noted that the lightweight object detection network used in this embodiment is a lightweight YOLO model. The lightweight YOLO model is a model optimized for lightweighting based on the YOLO structure. Specifically, a multi-branch parallel convolutional network replaces the YOLOv8 backbone network, effectively reducing computational cost. The neck network uses a feature enhancement structure based on differential convolution, which can improve the ability to fuse complex texture features. Furthermore, the Normalized Wasserstein Distance - Efficient Intersection over Union Loss Function (NWD-EIoU) is used instead of the traditional Complete Intersection over Union (CIoU) loss function to improve model convergence speed and localization accuracy.

[0034] In one feasible approach, before the step of performing target detection on the color image to obtain the two-dimensional position information of the target object, the method further includes: performing distortion correction processing on the color image; and enhancing the brightness and adjusting the contrast of the corrected image.

[0035] It should be noted that distortion coefficients (radial and tangential distortion) obtained from camera calibration are used to perform distortion correction transformation on the original acquired color image, correcting image distortion caused by lens optical characteristics and ensuring the accuracy of image geometric relationships. Histogram equalization or adaptive gamma correction algorithms are then used to process the corrected image. The aim is to compensate for uneven or insufficient lighting in the scene, enhance the overall visual effect and feature contrast of the image, thereby improving the stability of the object detection network under different lighting conditions.

[0036] Step S30: Determine the three-dimensional coordinates of the target object in the robotic arm's operating space based on the two-dimensional position information and the depth map; It should be noted that, based on the two-dimensional position information and the depth map, the distance information of a point can be obtained by querying the pixel value corresponding to the center point coordinates in the depth map. Then, using the camera intrinsic parameters and the hand-eye calibration matrix, coordinate transformation is performed to obtain the three-dimensional spatial coordinates in the robotic arm's base coordinate system.

[0037] Step S40: Based on the three-dimensional coordinates, control the robotic arm to perform a grasping action.

[0038] It should be noted that the calculated 3D coordinates serve as the target gripping position for the robotic arm's end effector (gripper). The motion control module, based on the robotic arm's inverse kinematics model, calculates the sequence of joint angles required to drive each joint to this position and plans a smooth, collision-free motion trajectory. Finally, it sends commands to the robotic arm controller to drive it along the planned trajectory, completing the gripping process.

[0039] In one feasible approach, the step of controlling the robotic arm to perform a grasping action based on the three-dimensional coordinates includes: calculating the target angles of each joint of the robotic arm according to the three-dimensional coordinates; generating a motion trajectory based on the target angles; controlling the robotic arm to move along the trajectory and driving the end effector to complete the grasping action.

[0040] It should be noted that, based on the target pose that the robotic arm's end effector needs to reach, and in conjunction with the Denavit-Hartenberg (DH) parameters, kinematic algorithms are used to calculate the target pose of each joint of the robotic arm. This generates a grasping trajectory command, driving the end effector to complete a precise and stable grasping motion. DH parameters refer to a standardized method for describing the geometry of robot links, used to establish the robot's kinematic model. DH parameters include link length, link torsion angle, joint distance, and joint rotation angle. Kinematic algorithms include forward kinematics and inverse kinematics. Forward kinematics is used to determine the position and pose of the end effector in space when the angles (or displacements) of all joints are known. Inverse kinematics, given the desired target position and pose of the end effector, solves for the required angles (or displacements) of each joint.

[0041] In one feasible approach, after the step of controlling the robotic arm to perform the grasping action, the method further includes: monitoring the grasping status of the robotic arm in real time through a vision sensor; if a grasping failure is detected, re-executing the target detection and grasping.

[0042] It should be noted that after the gripper closes and lifts the object, the depth camera is triggered again to capture the scene after the grasp. Simple image analysis (such as checking if abrupt changes in depth values ​​in the gripper area have disappeared, or performing a quick target detection) determines whether the gripper successfully grasped the object. If the grasp is deemed unsuccessful, the robotic arm is moved to a safe observation position, and then the process of acquiring images, detecting the target, calculating coordinates, and performing the grasp again is restarted. This closed-loop feedback mechanism gives the system fault tolerance and automatic retry capabilities, improving the success rate and automation level of the entire workflow.

[0043] This embodiment acquires color images and depth maps; performs target detection on the color images to obtain the two-dimensional position information of the target object; determines the three-dimensional coordinates of the target object in the robotic arm's operating space based on the two-dimensional position information and the depth map; and controls the robotic arm to perform a grasping action based on the three-dimensional coordinates. This embodiment obtains precise three-dimensional coordinates of the target object by fusing the target detection results of the color image with the ranging information of the depth map, and uses these coordinates to drive the robotic arm to complete a closed-loop grasping action, effectively reducing grasping deviations caused by inaccurate positioning, thereby improving the stability of the robotic arm's grasping action.

[0044] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step S30 also includes steps S301 to S303: Step S301: Extract the depth value of the corresponding point from the depth map based on the pixel coordinates in the two-dimensional position information; It should be noted that the pixel value at the corresponding position in the depth image can be read based on the center pixel coordinates in the two-dimensional location information. This pixel value is the depth value.

[0045] Step S302: Based on the camera intrinsic parameter matrix, back-project the pixel coordinates and the depth value to obtain the three-dimensional coordinates in the camera coordinate system; It should be noted that the camera intrinsic parameter matrix is ​​a pre-set coordinate transformation matrix. To effectively connect the visually detected target position with the robotic arm's execution control, a mapping relationship between the visual coordinate system and the robotic arm's base coordinate system needs to be established beforehand. First, intrinsic parameters, including focal length f, are determined through camera calibration, and then the pixel coordinates (u, v) and depth values ​​are converted into three-dimensional coordinates in the camera coordinate system using the pinhole imaging formula. The camera calibration formula is as follows:

[0046] in, , f is the camera focal length, and dx and dy represent the pixel size in the x and y directions, respectively. u and v are the coordinate values ​​in the pixel coordinate system. R and t are components of the extrinsic parameter matrix. R is a 3×3 rotation matrix describing the rotation transformation from the world coordinate system to the camera coordinate system, and t is a 3×1 translation vector describing the translation transformation from the world coordinate system to the camera coordinate system.

[0047] Step S303: Transform the three-dimensional coordinates in the camera coordinate system to the three-dimensional coordinates in the robot arm base coordinate system using the hand-eye transformation matrix.

[0048] It should be noted that a rigid transformation matrix needs to be obtained through hand-eye calibration experiments to establish a precise mapping relationship between the camera coordinate system and the robot arm end effector coordinate system, and then further transformed to the robot arm base coordinate system. For calibration using the eye-in-hand mode, the transformation matrix between the robot arm base coordinate system and the camera coordinate system needs to be solved by acquiring and fitting multiple known attitude points. The coordinate transformation formula is as follows:

[0049] Where Base is the coordinate system of the robotic arm base, Wrist is the coordinate system of the wrist joint, Board is the coordinate system of the calibration plate, and Cam is the coordinate system of the camera. For the eye-in-hand calibration method, the positional relationship between Base and Cam needs to be determined. .

[0050] By using the hand-eye transformation matrix, the 3D coordinates in the camera coordinate system can be transformed into the 3D coordinates in the robot arm base coordinate system.

[0051] This embodiment extracts the depth value of the corresponding point from the depth image based on the pixel coordinates in the two-dimensional position information; based on the camera intrinsic parameter matrix, it performs back-projection calculation on the pixel coordinates and the depth value to obtain the three-dimensional coordinates in the camera coordinate system; and uses a hand-eye transformation matrix to transform the three-dimensional coordinates in the camera coordinate system to the three-dimensional coordinates in the robot arm's base coordinate system. This embodiment achieves a precise and direct mapping from image pixels to the three-dimensional coordinates of the robot arm's operating space through the hand-eye transformation matrix. This step effectively avoids the accumulation of coordinate transformation errors in traditional methods, providing a reliable spatial reference for the subsequent high-precision grasping of the robot arm.

[0052] This application also provides a robotic arm vision grasping device, please refer to... Figure 4 The robotic arm vision grasping device includes: Acquisition module 10 is used to acquire color images and depth maps; Processing module 20 is used to perform target detection on the color image to obtain the two-dimensional position information of the target object; The determining module 30 is used to determine the three-dimensional coordinates of the target object in the robotic arm's operating space based on the two-dimensional position information and the depth map; The control module 40 is used to control the robotic arm to perform grasping actions based on the three-dimensional coordinates.

[0053] This embodiment acquires color images and depth maps; performs target detection on the color images to obtain the two-dimensional position information of the target object; determines the three-dimensional coordinates of the target object in the robotic arm's operating space based on the two-dimensional position information and the depth map; and controls the robotic arm to perform a grasping action based on the three-dimensional coordinates. This embodiment obtains precise three-dimensional coordinates of the target object by fusing the target detection results of the color image with the ranging information of the depth map, and uses these coordinates to drive the robotic arm to complete a closed-loop grasping action, effectively reducing grasping deviations caused by inaccurate positioning, thereby improving the stability of the robotic arm's grasping action.

[0054] In one embodiment, the processing module 20 is further configured to input the color image into a lightweight object detection network; perform forward inference through the lightweight object detection network to output the bounding box coordinates and center point pixel coordinates of the target object; the backbone network of the lightweight object detection network adopts a multi-branch parallel convolution structure, and the neck network of the lightweight object detection network integrates a feature enhancement module based on differential convolution.

[0055] In one embodiment, the determining module 30 is further configured to extract the depth value of the corresponding point from the depth map based on the pixel coordinates in the two-dimensional position information; perform back projection calculation on the pixel coordinates and the depth value based on the camera intrinsic parameter matrix to obtain the three-dimensional coordinates in the camera coordinate system; and transform the three-dimensional coordinates in the camera coordinate system to the three-dimensional coordinates in the robot arm base coordinate system through the hand-eye transformation matrix.

[0056] In one embodiment, the control module 40 is further configured to calculate the target angles of each joint of the robotic arm based on the three-dimensional coordinates; generate a motion trajectory based on the target angles; control the robotic arm to move along the trajectory and drive the end effector to complete the grasping.

[0057] In one embodiment, the processing module 20 is further configured to perform distortion correction processing on the color image; and to enhance the brightness and adjust the contrast of the corrected image.

[0058] In one embodiment, the control module 40 is also used to monitor the grasping status of the robotic arm in real time via a vision sensor; if a grasping failure is detected, the target detection and grasping are re-executed.

[0059] The robotic arm vision grasping device provided in this application, employing the robotic arm vision grasping method in the above embodiments, can solve the technical problem of improving the stability of robotic arm grasping. Compared with the prior art, the beneficial effects of the robotic arm vision grasping device provided in this application are the same as the beneficial effects of the robotic arm vision grasping method provided in the above embodiments, and other technical features in the robotic arm vision grasping device are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0060] This application provides a robotic arm vision grasping device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the robotic arm vision grasping method in the first embodiment described above.

[0061] The following is for reference. Figure 5 The diagram illustrates a structural schematic of a robotic arm vision grasping device suitable for implementing embodiments of this application. The robotic arm vision grasping device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The robotic arm vision grasping device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0062] like Figure 5As shown, the robotic arm vision grasping device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the robotic arm vision grasping device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the robotic arm vision grasping device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show robotic arm vision grasping devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0063] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0064] The robotic arm vision grasping device provided in this application, employing the robotic arm vision grasping method in the above embodiments, can solve the technical problem of improving the stability of robotic arm grasping. Compared with the prior art, the beneficial effects of the robotic arm vision grasping device provided in this application are the same as the beneficial effects of the robotic arm vision grasping method provided in the above embodiments, and other technical features in this robotic arm vision grasping device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0065] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0066] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0067] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the robotic arm vision grasping method in the above embodiments.

[0068] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0069] The aforementioned computer-readable storage medium may be included in the robotic arm vision grasping device; or it may exist independently and not assembled into the robotic arm vision grasping device.

[0070] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the robotic arm vision grasping device, cause the robotic arm vision grasping device to: acquire color images and depth maps; perform target detection on the color images to obtain two-dimensional position information of the target object; determine the three-dimensional coordinates of the target object in the robotic arm's operating space based on the two-dimensional position information and the depth map; and control the robotic arm to perform grasping actions based on the three-dimensional coordinates.

[0071] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0072] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0073] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0074] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described robotic arm vision grasping method, thereby solving the technical problem of improving the stability of robotic arm grasping. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the robotic arm vision grasping method provided in the above embodiments, and will not be repeated here.

[0075] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the robotic arm vision grasping method described above.

[0076] The computer program product provided in this application can solve the technical problem of improving the stability of robotic arm grasping. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the robotic arm vision grasping method provided in the above embodiments, and will not be repeated here.

[0077] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A robotic arm vision grasping method, characterized in that, The method includes: Acquire color images and depth maps; Target detection is performed on the color image to obtain the two-dimensional position information of the target object; Based on the two-dimensional position information and the depth map, the three-dimensional coordinates of the target object in the robotic arm's operating space are determined; Based on the three-dimensional coordinates, the robotic arm is controlled to perform a grasping action.

2. The method as described in claim 1, characterized in that, The step of performing target detection on the color image to obtain the two-dimensional position information of the target object includes: The color image is input into a lightweight object detection network; The lightweight object detection network performs forward inference and outputs the bounding box coordinates and center point pixel coordinates of the target object.

3. The method as described in claim 2, characterized in that, The backbone of the lightweight object detection network adopts a multi-branch parallel convolutional structure, and the neck network of the lightweight object detection network integrates a feature enhancement module based on differential convolution.

4. The method as described in claim 1, characterized in that, The step of determining the three-dimensional coordinates of the target object in the robotic arm's operating space based on the two-dimensional position information and the depth map includes: Based on the pixel coordinates in the two-dimensional location information, the depth value of the corresponding point is extracted from the depth map; Based on the camera intrinsic parameter matrix, the pixel coordinates and the depth value are back-projected to obtain the three-dimensional coordinates in the camera coordinate system. The three-dimensional coordinates in the camera coordinate system are transformed to the three-dimensional coordinates in the robot arm base coordinate system by using a hand-eye transformation matrix.

5. The method as described in claim 1, characterized in that, The steps of controlling the robotic arm to perform the grasping action based on the three-dimensional coordinates include: Calculate the target angles of each joint of the robotic arm based on the three-dimensional coordinates; Generate a motion trajectory based on the target angle; The robotic arm is controlled to move along the trajectory and the end effector is driven to complete the grasping.

6. The method as described in claim 1, characterized in that, Before the step of performing target detection on the color image to obtain the two-dimensional position information of the target object, the method further includes: The color image is subjected to distortion correction processing; The corrected image is then enhanced in brightness and its contrast is adjusted.

7. The method as described in claim 1, characterized in that, After the step of controlling the robotic arm to perform the grasping action, the method further includes: The robotic arm's grasping status is monitored in real time using visual sensors; If a crawling failure is detected, target detection and crawling will be re-executed.

8. A robotic arm vision grasping device, characterized in that, The device includes: The acquisition module is used to acquire color images and depth maps; The processing module is used to perform target detection on the color image to obtain the two-dimensional position information of the target object; The determination module is used to determine the three-dimensional coordinates of the target object in the robotic arm's operating space based on the two-dimensional position information and the depth map; The control module is used to control the robotic arm to perform grasping actions based on the three-dimensional coordinates.

9. A robotic arm vision grasping device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robotic arm vision grasping method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the robotic arm vision grasping method as described in any one of claims 1 to 7.

Images