Target grabbing point data generation method, device and equipment and humanoid robot

Abstract

The embodiment of the invention provides a target grabbing point data generation method, device and equipment and a humanoid robot, and belongs to the technical field of target detection. The method comprises the following steps: acquiring a color image and a depth image of a target object shot by shooting equipment in real time; performing alignment correction processing according to the color image and the depth image to obtain a pseudo-color depth map; inputting the pseudo-color depth map into a pre-training deep learning segmentation model for detection segmentation processing to obtain a target segmentation mask map of the target object; performing centroid extraction processing according to the target segmentation mask pattern, and determining centroid coordinates of a target object; and performing depth correction processing according to the target object mass center coordinate, the pseudo-color depth map and a preset internal reference matrix to obtain a target three-dimensional coordinate, and determining the target three-dimensional coordinate as target grabbing point data. The method is used for achieving the effect of improving the accuracy of the captured point data obtained in a complex dynamic scene.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, specifically to the field of target detection technology, and in particular to a method, device, equipment and humanoid robot for generating target grasping point data. Background Art

[0002] With the development of robotics and computer vision, target detection and grasping point coordinate generation based on depth cameras have been widely used in industrial automation, service robots, and drones. Among them, the robot's grasping function of the target object is one of the necessary functions of the robot. The key to grasping the target object is to obtain the data of the target grasping point.

[0003] In the prior art, a method for determining grasping point data is to use a depth image taken by a depth camera in combination with a target segmentation technology to detect specific position data of a target object as grasping point data.

[0004] The inventors have discovered that the prior art has at least the following technical problems: in complex dynamic scenes, such as scenes where lighting conditions change or parts of the target object are obscured, there is a problem of low accuracy of the detected grasping point data. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment and humanoid robot for generating target grasping point data, so as to achieve the effect of improving the accuracy of grasping point data obtained in complex dynamic scenes.

[0006] In a first aspect, an embodiment of the present application provides a method for generating target grab point data, comprising:

[0007] Acquire the color image and depth image of the target object taken by the shooting device in real time;

[0008] Performing alignment correction processing on the color image and the depth image to obtain a pseudo-color depth map;

[0009] Inputting the pseudo-color depth map into a pre-trained deep learning segmentation model for detection and segmentation processing to obtain a target segmentation mask map of the target object;

[0010] Performing centroid extraction processing according to the target segmentation mask image to determine the centroid coordinates of the target object;

[0011] Depth correction processing is performed according to the target object's centroid coordinates, the pseudo-color depth map and a preset internal parameter matrix to obtain the target three-dimensional coordinates, and the target three-dimensional coordinates are determined as the target grabbing point data.

[0012] In a possible implementation, the aligning and correcting the color image and the depth image to obtain a pseudo-color depth map includes: aligning the color image and the depth image through a preset depth camera technology library to obtain data to be corrected, wherein the data to be corrected is color information and depth information at the same pixel position; and performing correction processing based on the data to be corrected and a preset depth scale factor to obtain a pseudo-color depth map.

[0013] In a possible implementation, the data to be corrected includes the original depth of each pixel; accordingly, the calculation formula used when performing the correction process according to the data to be corrected and the preset depth scale factor is:

[0014]

[0015] In the formula, is the corrected depth value of the pixel point (x, y), Z is the original depth of the pixel point (x, y), and S is the preset depth calibration factor.

[0016] In a possible implementation, the centroid extraction processing is performed according to the target segmentation mask image to obtain the centroid coordinates of the target object, including: determining the area of ​​the mask region, the first-order geometric distance about the x-axis in the mask image, and the first-order geometric distance about the y-axis in the mask image according to the target segmentation mask image; determining the x-axis coordinate value of the centroid according to the mask region area and the first-order geometric distance about the x-axis in the mask image; determining the y-axis coordinate value of the centroid according to the mask region area and the first-order geometric distance about the y-axis in the mask image; and determining the centroid coordinates of the target object according to the x-axis coordinate value of the centroid and the y-axis coordinate value of the centroid.

[0017] In a possible implementation, the depth correction processing is performed according to the target object's center of mass coordinates, the pseudo-color depth map and a preset internal parameter matrix to obtain the target three-dimensional coordinates, including: determining the target center of mass depth value according to the target object's center of mass coordinates and the pseudo-color depth map; performing depth correction processing according to the target object's center of mass coordinates, the target center of mass depth value and a preset internal parameter matrix to obtain the target three-dimensional coordinates.

[0018] In a possible implementation, the calculation formula used when performing depth correction processing according to the target object centroid coordinates, the target centroid depth value and a preset internal parameter matrix to obtain the target three-dimensional coordinates is:

[0019]

[0020] In the formula, is the three-dimensional coordinate of the target, is the target centroid depth value, K is the preset internal parameter matrix, cX is the x-axis coordinate value in the target object centroid coordinates, and cY is the y-axis coordinate value in the target object centroid coordinates.

[0021] In a possible implementation, it also includes: performing coordinate conversion processing on the target grasping point data to obtain the robot grasping coordinates in the robot coordinate system; generating a target grasping path according to the robot grasping coordinates; generating a grasping instruction according to the target grasping path; synthesizing the robot grasping coordinates into the real-time image captured by the shooting device according to the grasping instruction and controlling the actuator to complete the grasping of the target object according to the target grasping path.

[0022] In a second aspect, an embodiment of the present application further provides a target grab point data generating device, comprising:

[0023] An image acquisition module acquires the color image and depth image of the target object captured by the shooting device in real time;

[0024] An image alignment module, used to perform alignment correction processing according to the color image and the depth image to obtain a pseudo-color depth map;

[0025] A detection and segmentation module is used to input the pseudo-color depth map into a pre-trained deep learning segmentation model for detection and segmentation processing to obtain a target segmentation mask map of the target object;

[0026] A calculation module, used for performing centroid extraction processing according to the target segmentation mask image to determine the centroid coordinates of the target object;

[0027] The depth correction module is used to perform depth correction processing according to the target object's centroid coordinates, the pseudo-color depth map and a preset internal parameter matrix to obtain the target three-dimensional coordinates, and determine the target three-dimensional coordinates as the target grabbing point data.

[0028] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0029] The memory stores computer-executable instructions;

[0030] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0031] In a fourth aspect, an embodiment of the present application provides a humanoid robot, comprising a robot body, a photographing device disposed on the robot body, and an electronic device as described in the third aspect above disposed in the robot body.

[0032] The embodiments of the present application provide a method, device, equipment and humanoid robot for generating target grabbing point data, which aligns and corrects the color image and depth image captured by the depth camera to obtain a pseudo-color image, and then uses a pre-trained deep learning segmentation model to achieve accurate detection and segmentation to obtain a target segmentation mask map. Then, by performing centroid extraction on the target segmentation mask map, the problem of low segmentation accuracy in dynamic and complex scenes such as illumination changes, blurred target boundaries, and occlusion is solved, so that the geometric feature extraction of the target area in dynamic and complex scenes is more accurate, thereby improving the accuracy of the target object's centroid coordinates. Finally, a depth correction process is performed based on the target object's centroid coordinates, the pseudo-color depth map and the preset internal parameter matrix to obtain the target three-dimensional coordinates, and the target three-dimensional coordinates are determined as the target grabbing point data, thereby improving the accuracy of the target grabbing point data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0034] Figure 1 A schematic diagram of an application scenario of the target grab point data generation method provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram of a flow chart of a method for generating target grabbing point data provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of the application process of the target grab point data generation method provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of the structure of the target grab point data generation device provided in the embodiment of the present application Figure 1 ;

[0038] Figure 5 The structure diagram of the target grab point data generation device provided in this embodiment is as follows: Figure 2 ;

[0039] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0040] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0041] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0042] When the inventors used the methods in the prior art to obtain the grasping point data, they found that the depth map of the target object is generally first captured by a depth camera, which lacks support for accurate segmentation of the target object, resulting in blurry or deviation of the target boundary, thereby reducing the accuracy of the grasping point data obtained. In addition, in complex dynamic scenes such as changing lighting conditions or partial occlusion of the target object, the grasping point data obtained is even more inaccurate.

[0043] Therefore, the inventors proposed the following inventive concept to solve the above-mentioned technical problems: while accurately segmenting the target object, the problem of low target object detection and segmentation accuracy in complex dynamic scenes is solved through depth information correction, centroid calculation of the target mask, and alignment of the depth image and color image, thereby improving the accuracy of the final grasping point data.

[0044] Figure 1 A schematic diagram of an application scenario of the target grab point data generation method provided in an embodiment of the present application, such as Figure 1 As shown, the specific application scenario of the present application includes a humanoid robot 101 and a target object 102 .

[0045] Among them, a depth camera and a chip are installed on the humanoid robot 101, and the chip is connected to the depth camera for communication. The depth camera is used to shoot the target object 102 in real time to obtain a depth map and a color map. Then the chip receives the depth map and color map transmitted by the depth camera, and executes the relevant steps of the target grasping point data generation method for the depth map and the color map, and finally generates the target grasping point data of the humanoid robot 101 for the target object 102.

[0046] In an optional embodiment of the present application, the processor of the depth camera of the humanoid robot can also execute the relevant steps of the target grasping point data generation method, and finally generate the target grasping point data of the humanoid robot 101 for the target object 102, and transmit it to the robot control chip inside the humanoid robot 101.

[0047] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0048] Figure 2 The flowchart of the target grab point data generation method provided in the embodiment of the present application is as follows. The execution subject of the target grab point data generation method can be as follows Figure 1 The chip or processor in the humanoid robot 101 shown, or the computer-related device wirelessly connected to the humanoid robot 101, is not limited in this embodiment.

[0049] like Figure 2 As shown, the target grab point data generation method includes:

[0050] S201: Acquire a color image and a depth image of a target object captured by a shooting device in real time.

[0051] In this embodiment, the shooting device may be a depth camera, the color image refers to an RBG image, and the depth image refers to an image containing depth information of each pixel, and the depth information may be the distance between the target object and the camera.

[0052] S202: Perform alignment correction processing on the color image and the depth image to obtain a pseudo-color depth map.

[0053] In this embodiment, the alignment correction process refers to a one-to-one correspondence between all pixels in the color image and all pixels in the depth image, so as to ensure that the color information and the depth information at the same pixel position accurately correspond to each other.

[0054] Specifically, in an optional embodiment of the present application, step S202 includes:

[0055] S202a: Performing alignment processing on the color image and the depth image using a preset depth camera technology library to obtain data to be corrected, wherein the data to be corrected is color information and depth information at the same pixel position.

[0056] In this embodiment, the preset depth camera technology library may be a pyrealsense2 library, which may align a dark image with a color image to ensure that the color information and the depth information at the same pixel position accurately correspond.

[0057] S202b: Perform correction processing according to the data to be corrected and a preset depth scale factor to obtain a pseudo-color depth map.

[0058] In this embodiment, the depth value of each pixel after correction can be obtained by using the data to be corrected, the preset depth scale factor and the corresponding correction formula, and then the depth value of each pixel after correction is corresponded to the corresponding pixel of the color image to obtain a pseudo-color depth map.

[0059] In an optional embodiment of the present application, the data to be corrected includes the original depth of each pixel. Accordingly, the calculation formula used in step S202b for performing correction processing according to the data to be corrected and the preset depth scale factor is:

[0060]

[0061] In the formula, is the corrected depth value of the pixel point (x, y), Z is the original depth of the pixel point (x, y), and S is the preset depth calibration factor.

[0062] In this embodiment, it is assumed that the pixel point in the color image is (x, y), and the original depth of the pixel point in the corresponding depth image is Z. At this time, the corrected depth value can be calculated according to the calculation formula At this time, the corrected depth value is fused to the pixel (x, y) position in the color image. After all pixels go through the above process, the final image is a pseudo-color depth map.

[0063] In this embodiment, the corrected depth value is fused into the color image, which can provide a more reliable data basis for subsequent steps, so as to achieve the effect of improving the accuracy of the final target grasping point data.

[0064] S203: Input the pseudo-color depth map into a pre-trained deep learning segmentation model for detection and segmentation processing to obtain a target segmentation mask map of the target object.

[0065] In this embodiment, the pre-trained deep learning segmentation model can be a target segmentation model that has been trained using training data for complex dynamic scenes and has undergone deep learning, such as a YOLOv8 segmentation model that supports high precision and target diversity processing.

[0066] In an optional embodiment of the present application, the training optimization process of the pre-trained deep learning segmentation model is also included as follows:

[0067] S203a: Acquire a training data set, wherein the training data set includes a pseudo-color depth map under a lighting change condition, a pseudo-color depth map under an occlusion condition, and a pseudo-color depth map under a complex background condition.

[0068] S203b: Perform model training on the initial segmentation model according to the training data set to obtain a pre-trained deep learning segmentation model.

[0069] In this embodiment, the pre-trained deep learning segmentation model can more accurately detect and segment pseudo-color depth maps under conditions of changing lighting, pseudo-color depth maps under occlusion conditions, and pseudo-color depth maps under conditions of complex backgrounds, obtain target types, bounding boxes, and masks, and the generated masks are represented by binary images.

[0070] In this embodiment, by using an optimized pre-trained deep learning segmentation model to efficiently infer the pseudo-color depth map, more accurate target segmentation can be achieved in complex dynamic environments to cope with complex dynamic environments such as lighting changes, occlusion and background, thereby improving the robustness of target detection.

[0071] S204: Perform centroid extraction processing according to the target segmentation mask image to determine the centroid coordinates of the target object.

[0072] In this embodiment, the centroid coordinates of the target object can be calculated by performing morphological analysis on each target segmentation mask image and using the image moment method.

[0073] Based on the above embodiment, in an optional embodiment of the present application, step S204 includes:

[0074] S204a: Determine, according to the target segmentation mask image, the area of ​​the mask region, the first-order geometric distance about the x-axis in the mask image, and the first-order geometric distance about the y-axis in the mask image.

[0075] S204b: Determine the x-axis coordinate value of the centroid according to the area of ​​the mask region and the first-order geometric distance about the x-axis in the mask image.

[0076] S204c: Determine the y-axis coordinate value of the centroid according to the area of ​​the mask region and the first-order geometric distance about the y-axis in the mask image.

[0077] S204d: Determine the center of mass coordinates of the target object according to the center of mass x-axis coordinate value and the center of mass y-axis coordinate value.

[0078] In this embodiment, all the pixels in the image region of the target segmentation mask image are known, and the sum of the x coordinates of all the pixels in the image region of the target segmentation mask image can be obtained through image processing software, and the sum of the x coordinates of all the pixels in the image region of the target segmentation mask image is determined as the first-order geometric distance about the x-axis in the mask image. Similarly, the sum of the y coordinates of all the pixels in the image region of the target segmentation mask image can also be obtained, and the sum of the y coordinates of all the pixels in the image region of the target segmentation mask image is determined as the first-order geometric distance about the y-axis in the mask image.

[0079] In this embodiment, the calculation formula used in steps S204b and S204c is:

[0080]

[0081] In the formula, is the area of ​​the mask region, is the first-order geometric distance about the x-axis in the mask image, is the first-order geometric distance about the y-axis in the mask image, cX is the x-axis coordinate value of the center of mass, and cY is the y-axis coordinate value of the center of mass.

[0082] In this embodiment, the above calculation process can be combined with the optimized performance of OpenCV, so that the centroid extraction process can be completed in milliseconds and can fully adapt to the needs of dynamic scenes.

[0083] S205: Perform depth correction processing according to the target object's centroid coordinates, the pseudo-color depth map, and a preset internal parameter matrix to obtain the target three-dimensional coordinates, and determine the target three-dimensional coordinates as the target grabbing point data.

[0084] In this embodiment, the depth correction process can be a process of calculating the target three-dimensional coordinates through depth correction and back-projection formula to finally obtain accurate target three-dimensional coordinates. This process relies on high-precision coordinate mapping and real-time correction functions.

[0085] Specifically, in an optional embodiment of the present application, in step S205, depth correction processing is performed according to the target object centroid coordinates, the pseudo-color depth map and the preset intrinsic parameter matrix to obtain the target three-dimensional coordinates, including:

[0086] S205a: Determine the target centroid depth value according to the target object centroid coordinates and the pseudo-color depth map.

[0087] In this embodiment, the target object centroid coordinates correspond to a fixed pixel in the pseudo-color depth map. The pseudo-color depth map and the target object centroid coordinates can be combined to match the depth value of the corresponding pixel in the pseudo-color depth map, and the depth value is determined as the target centroid depth value. Assume that the target object centroid coordinates are (cX, cY), and the depth value corresponding to the target object centroid coordinates in the pseudo-color depth map is recorded as D (cX, cY).

[0088] S205b: Perform depth correction processing according to the target object's centroid coordinates, the target centroid depth value and a preset internal reference matrix to obtain the target three-dimensional coordinates.

[0089] Specifically, in an optional embodiment of the present application, in step S205, depth correction processing is performed according to the target object centroid coordinates, the target centroid depth value and the preset internal parameter matrix, and the calculation formula used to obtain the target three-dimensional coordinates is:

[0090]

[0091] In the formula, is the target three-dimensional coordinate, is the target centroid depth value, K is the preset internal parameter matrix, cX is the x-axis coordinate value in the target object centroid coordinates, and cY is the y-axis coordinate value in the target object centroid coordinates.

[0092] On the basis of the above embodiment, a method for generating grab point data provided as an optional embodiment of the present application further includes:

[0093] Step A: Perform coordinate conversion on the target grasping point data to obtain the robot grasping coordinates in the robot coordinate system.

[0094] Step B: Generate the target grasping path according to the robot grasping coordinates.

[0095] Step C: Generate a crawling instruction according to the target crawling path.

[0096] Step D: According to the grasping instruction, the robot grasping coordinates are synthesized into the real-time image captured by the shooting device and the actuator is controlled to complete the grasping of the target object according to the target grasping path.

[0097] In this embodiment, steps A to D refer to generating operation instructions for grasping points based on the target grasping point data, i.e., the target three-dimensional coordinates, in combination with the working range of the robot end effector and the coordinate system conversion rules, to control the actuator to complete the grasping of the target object according to the target grasping path, wherein the target grasping path is the movement path of the center coordinates of the actuator to move to the target three-dimensional coordinates. Finally, the target grasping point data is displayed on the color image and depth image captured by the shooting device for real-time monitoring and debugging. For example: the robot grasping point coordinates are synthesized into the real-time video screen captured by the shooting device in a video synthesis method or a picture pixel synthesis method.

[0098] Figure 3 A schematic diagram of the application flow of the target grab point data generation method provided in an embodiment of the present application.

[0099] like Figure 3 As shown, the method takes the starting step as the first step, and includes the following steps in sequence: first, the hardware and software initialization process, that is, setting the RealSense configuration of the shooting device, starting the pipeline for real-time shooting of the target object image transmission, and loading the pre-trained deep learning segmentation model (hereinafter replaced by the YOLOv8 model used in this embodiment), in order to avoid the data generated by the shooting device in the automatic parameter adjustment stage affecting the target grab point data generation efficiency and accuracy, skip the first 30 frames of depth images and color images (hereinafter represented by frame data such as depth and color frames). Then, by acquiring frame data from the shooting device in real time, aligning the depth and color frames, and then converting them into an array to generate a pseudo-color depth map. Then enter the mask detection step, that is, by using the YOLOv8 model to detect the pseudo-color depth map corresponding to each frame of data, to perform the step of whether the mask is detected.

[0100] When a mask is detected, the mask processing step is entered, which corresponds to the calculation of the center of mass coordinates of the target object, the correction of the depth information corresponding to the center of mass, and the conversion into the three-dimensional coordinates of the target object in the above embodiment, and then the target grab point data is generated. A grab instruction is generated based on the target grab point data, and then the grab point is drawn in the image of the current frame. Then the frame display step is entered, and the display frame includes the annotated image and the depth map. Then the key detection step is entered to determine whether a user or a humanoid robot automatically presses the corresponding stop key, such as whether the q key is pressed.

[0101] If yes, stop the pipeline to cut off the image transmission channel. Then close the window and end the execution of the target grab point data generation method. Otherwise, if no, return to the step of obtaining frame data.

[0102] When no mask is detected, it jumps directly to the step of displaying the frame.

[0103] In summary, the target grab point data generation method provided in the embodiment of the present application aligns and corrects the color image and the depth image collected by the depth camera to obtain a pseudo-color image, and then uses a pre-trained deep learning segmentation model to achieve accurate detection and segmentation to obtain a target segmentation mask image. Then, by performing centroid extraction on the target segmentation mask image, the problem of low segmentation accuracy in dynamic and complex scenes such as illumination changes, blurred target boundaries, and occlusion is solved, so that the geometric feature extraction of the target area in dynamic and complex scenes is more accurate, thereby improving the accuracy of the target object's centroid coordinates. Finally, a depth correction process is performed based on the target object's centroid coordinates, the pseudo-color depth map, and the preset internal parameter matrix to obtain the target three-dimensional coordinates, and the target three-dimensional coordinates are determined as the target grab point data, thereby improving the accuracy of the target grab point data.

[0104] At the same time, the accuracy of segmentation operations in dynamic scenes is improved by pre-training deep learning segmentation models, and more efficient generation of target three-dimensional coordinates is achieved, making this method adaptable to the application requirements of multiple targets and complex scenes.

[0105] At the same time, by performing centroid extraction processing and subsequent steps on the target segmentation mask image, it is ensured that the geometric feature extraction of the target area is more accurate, thereby improving the accuracy of the target grasping point data.

[0106] At the same time, the internal parameter correction and the conversion of pixels to 3D coordinates further improve the calculation accuracy of the target 3D coordinates, and can maintain high robustness even when the depth value related data is discontinuous or the noise interference is large.

[0107] At the same time, the computational overhead is reduced through efficient alignment processing and centroid extraction of the target segmentation mask image. At the same time, combined with the built-in hardware acceleration features of the depth camera and the reasoning ability of the segmentation model, high-real-time target detection and segmentation are achieved to meet the real-time needs of scenes such as industrial robots and service robots.

[0108] At the same time, the method can also be used to segment multiple target objects and generate target grasping point data, and generate independent target grasping point results based on the depth and target three-dimensional coordinates of each target object. In complex scenes, it can effectively distinguish target objects and generate corresponding grasping instructions for each target object, providing technical support for the robot's autonomous operation in multi-target scenes.

[0109] Figure 4 A schematic diagram of the structure of the target grab point data generation device provided in the embodiment of the present application Figure 1 ,like Figure 4 As shown, the target grab point data generating device provided in this embodiment includes: an image acquisition module 41, an image alignment module 42, a detection and segmentation module 43, a calculation module 44 and a depth correction module 45.

[0110] The image acquisition module 41 acquires the color image and the depth image of the target object captured by the shooting device in real time.

[0111] The image alignment module 42 is used to perform alignment correction processing on the color image and the depth image to obtain a pseudo-color depth map.

[0112] The detection and segmentation module 43 is used to input the pseudo-color depth map into the pre-trained deep learning segmentation model for detection and segmentation processing to obtain a target segmentation mask map of the target object.

[0113] The operation module 44 is used to perform centroid extraction processing according to the target segmentation mask image to determine the centroid coordinates of the target object.

[0114] The depth correction module 45 is used to perform depth correction processing according to the target object's centroid coordinates, the pseudo-color depth map and the preset internal parameter matrix to obtain the target three-dimensional coordinates, and determine the target three-dimensional coordinates as the target grabbing point data.

[0115] In one possible implementation, the image alignment module 42 is specifically used to: perform alignment processing on the color image and the depth image through a preset depth camera technology library to obtain data to be corrected, wherein the data to be corrected is the color information and depth information at the same pixel position; perform correction processing on the data to be corrected and a preset depth scale factor to obtain a pseudo-color depth map.

[0116] In a possible implementation, the data to be corrected includes the original depth of each pixel; accordingly, the image alignment module 42 is specifically configured to: perform the correction processing according to the data to be corrected and the preset depth scale factor using the following calculation formula:

[0117]

[0118] In the formula, is the corrected depth value of the pixel point (x, y), Z is the original depth of the pixel point (x, y), and S is the preset depth calibration factor.

[0119] In one possible implementation, the operation module 44 is specifically used to: determine the area of ​​the mask region, the first-order geometric distance about the x-axis in the mask image, and the first-order geometric distance about the y-axis in the mask image according to the target segmentation mask image; determine the x-axis coordinate value of the center of mass according to the area of ​​the mask region and the first-order geometric distance about the x-axis in the mask image; determine the y-axis coordinate value of the center of mass according to the area of ​​the mask region and the first-order geometric distance about the y-axis in the mask image; determine the coordinates of the center of mass of the target object according to the x-axis coordinate value of the center of mass and the y-axis coordinate value of the center of mass.

[0120] In an optional embodiment of the present application, the depth correction module 45 is specifically used to: determine the target center of mass depth value based on the target object center of mass coordinates and the pseudo-color depth map; perform depth correction processing based on the target object center of mass coordinates, the target center of mass depth value and a preset internal parameter matrix to obtain the target three-dimensional coordinates.

[0121] In an optional embodiment of the present application, the depth correction module 45 is specifically used to perform depth correction processing according to the target object centroid coordinates, the target centroid depth value and the preset internal parameter matrix, and the calculation formula used to obtain the target three-dimensional coordinates is:

[0122]

[0123] In the formula, is the target three-dimensional coordinate, is the target centroid depth value, K is the preset internal parameter matrix, cX is the x-axis coordinate value in the target object centroid coordinates, and cY is the y-axis coordinate value in the target object centroid coordinates.

[0124] Figure 5 The structure diagram of the target grab point data generation device provided in this embodiment is as follows: Figure 2 .

[0125] like Figure 5As shown, the target grasping point generating device also includes a display execution module 46. In an optional embodiment of the present application, the display execution module 46 is specifically used to: perform coordinate conversion processing on the target grasping point data to obtain the robot grasping coordinates in the robot coordinate system; generate a target grasping path according to the robot grasping coordinates; generate a grasping instruction according to the target grasping path; synthesize the robot grasping coordinates into the real-time picture captured by the shooting device according to the grasping instruction and control the actuator to complete the grasping of the target object according to the target grasping path.

[0126] The target grabbing point data generating device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0127] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device further includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.

[0128] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.

[0129] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0130] An embodiment of the present application also provides a humanoid robot, including a robot body, a shooting device arranged on a vehicle body, and an electronic device as described in the above embodiment arranged in the robot body.

[0131] In this embodiment, both being arranged on the robot body and being arranged inside the robot body can be realized by means of hardware connection structures such as bolt connection, threaded connection, etc.

[0132] The embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above target grabbing point data generating method is implemented.

[0133] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above target grasping point data generation method when executed by a processor.

[0134] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0135] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0136] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0137] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0138] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0139] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0140] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0141] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0144] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0145] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0146] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for generating target grab point data, characterized in that: include: Acquire the color image and depth image of the target object taken by the shooting device in real time; Performing alignment correction processing on the color image and the depth image to obtain a pseudo-color depth map; Inputting the pseudo-color depth map into a pre-trained deep learning segmentation model for detection and segmentation processing to obtain a target segmentation mask map of the target object; Performing centroid extraction processing according to the target segmentation mask image to determine the centroid coordinates of the target object; Depth correction processing is performed according to the target object's centroid coordinates, the pseudo-color depth map and a preset internal parameter matrix to obtain the target three-dimensional coordinates, and the target three-dimensional coordinates are determined as the target grabbing point data.

2. The method according to claim 1, characterized in that The step of performing alignment correction processing on the color image and the depth image to obtain a pseudo-color depth map includes: Performing alignment processing on the color image and the depth image through a preset depth camera technology library to obtain data to be corrected, wherein the data to be corrected is color information and depth information at the same pixel position; Correction processing is performed according to the data to be corrected and a preset depth scale factor to obtain a pseudo-color depth map.

3. The method according to claim 2, characterized in that The data to be corrected includes the original depth of each pixel; Accordingly, the calculation formula used when performing the correction process according to the data to be corrected and the preset depth scale factor is: In the formula, is the corrected depth value of the pixel point (x, y), Z is the original depth of the pixel point (x, y), and S is the preset depth calibration factor.

4. The method according to claim 1, characterized in that: The centroid extraction process is performed according to the target segmentation mask image to obtain the centroid coordinates of the target object, including: According to the target segmentation mask image, determine the area of ​​the mask region, the first-order geometric distance about the x-axis in the mask image, and the first-order geometric distance about the y-axis in the mask image; Determine the x-axis coordinate value of the centroid according to the area of ​​the mask region and the first-order geometric distance about the x-axis in the mask image; Determine the y-axis coordinate value of the centroid according to the area of ​​the mask region and the first-order geometric distance about the y-axis in the mask image; The center of mass coordinates of the target object are determined based on the center of mass x-axis coordinate value and the center of mass y-axis coordinate value.

5. The method according to claim 1, characterized in that The depth correction process is performed according to the target object centroid coordinates, the pseudo-color depth map and the preset internal parameter matrix to obtain the target three-dimensional coordinates, including: Determine the target object centroid depth value according to the target object centroid coordinates and the pseudo-color depth map; Depth correction processing is performed according to the target object's centroid coordinates, the target centroid depth value and a preset internal parameter matrix to obtain the target three-dimensional coordinates.

6. The method according to claim 5, characterized in that The calculation formula used for performing depth correction processing according to the target object centroid coordinates, the target centroid depth value and the preset internal parameter matrix to obtain the target three-dimensional coordinates is: In the formula, is the three-dimensional coordinate of the target, is the target centroid depth value, K is the preset internal parameter matrix, cX is the x-axis coordinate value in the target object centroid coordinates, and cY is the y-axis coordinate value in the target object centroid coordinates.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: Performing coordinate conversion processing on the target grasping point data to obtain the robot grasping coordinates in the robot coordinate system; Generate a target grasping path according to the robot grasping coordinates; Generate a grabbing instruction according to the target grabbing path; According to the grasping instruction, the robot grasping coordinates are synthesized into the real-time picture taken by the shooting device and the actuator is controlled to complete the grasping of the target object according to the target grasping path.

8. A target grab point data generating device, characterized in that: include: An image acquisition module acquires the color image and depth image of the target object captured by the shooting device in real time; An image alignment module, used to perform alignment correction processing according to the color image and the depth image to obtain a pseudo-color depth map; A detection and segmentation module is used to input the pseudo-color depth map into a pre-trained deep learning segmentation model for detection and segmentation processing to obtain a target segmentation mask map of the target object; A calculation module, used for performing centroid extraction processing according to the target segmentation mask image to determine the centroid coordinates of the target object; The depth correction module is used to perform depth correction processing according to the target object's centroid coordinates, the pseudo-color depth map and a preset internal parameter matrix to obtain the target three-dimensional coordinates, and determine the target three-dimensional coordinates as the target grabbing point data.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A humanoid robot, characterized in that: The invention comprises a robot body, a photographing device arranged on the robot body, and the electronic device as claimed in claim 9 arranged in the robot body.

Images