Posture estimation method, device, readable storage medium and robot

Abstract

The present invention provides a posture estimation method, device, readable storage medium and robot. The posture estimation method includes: receiving two-dimensional image data of a target object captured by a camera; inputting the two-dimensional image data into a target detection model to obtain a first number of direction vector diagrams; processing the first number of direction vector diagrams to obtain two-dimensional coordinate data of a center point and two-dimensional coordinate data of a second number of key points; processing the point cloud data of the target object to obtain a third number of three-dimensional coordinate data; determining the posture of the target object in the camera's coordinate system based on the two-dimensional coordinate data of the center point, the two-dimensional coordinate data of the second number of key points, the third number of three-dimensional coordinate data and the camera's parameter data. Since the above operations are all implemented based on two-dimensional image data, there is no need to use expensive 3D camera equipment, thereby reducing the estimation cost during posture estimation.

Description

Technical Field

[0001] The present invention relates to the field of control technology, and in particular to a posture estimation method, device, readable storage medium and robot. Background Art

[0002] Existing robotic grasping technology is of great significance to service robots and industrial robots, serving as a crucial means for robots to interact with their external environment. Object recognition, positioning, and pose estimation are key technologies for robotic grasping, with the recognition rate, positioning accuracy, and pose estimation accuracy determining the success rate of grasping.

[0003] Existing object pose estimation methods are mainly divided into 3D template matching-based methods and deep learning-based methods.

[0004] Among them, methods based on 3D template matching must ensure that the point cloud to be matched is relatively clean. Traditional recognition, segmentation, and filtering methods cannot achieve efficient and stable denoising and segmentation results, and therefore have certain limitations. Deep learning-based methods mainly use a two-stage approach, first segmenting the 3D point cloud and then estimating the pose of the segmented target. However, segmentation is often tested in scenes with simple backgrounds and is not effective when detecting multiple instances of the same target. At the same time, processing point cloud data consumes a huge amount of network computing resources, making real-time detection impossible. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0006] To this end, a first aspect of the present invention is to provide a posture estimation method.

[0007] A second aspect of the present invention is to provide a posture estimation device.

[0008] A third aspect of the present invention is to provide a posture estimation device.

[0009] A fourth aspect of the present invention provides a readable storage medium.

[0010] A fifth aspect of the present invention provides a robot.

[0011] In view of this, according to a first aspect of the present invention, the present invention provides a posture estimation method, including: receiving two-dimensional image data of a target object captured by a camera; inputting the two-dimensional image data into a target detection model to obtain a first number of direction vector graphs; processing the first number of direction vector graphs to obtain two-dimensional coordinate data of a center point and two-dimensional coordinate data of a second number of key points; processing the point cloud data of the target object to obtain a third number of three-dimensional coordinate data; determining the position and posture of the target object in the camera's coordinate system based on the two-dimensional coordinate data of the center point, the two-dimensional coordinate data of the second number of key points, the third number of three-dimensional coordinate data, and the camera's parameter data.

[0012] The technical solution of the present application proposes a posture estimation method. By running the estimation method, the posture estimation of the target object can be achieved. In the above solution, the posture estimation of the target object is achieved using two-dimensional image data. Since the above operations are all based on two-dimensional image data, there is no need to use expensive 3D camera equipment, thereby reducing the estimation cost of the posture estimation.

[0013] In addition, the above scheme uses two-dimensional image data for estimation, and two-dimensional operations have low requirements on the complexity of the target detection model. Therefore, it provides a basis for realizing rapid estimation of posture and also makes real-time estimation of posture possible.

[0014] In the above technical solution, the two-dimensional image data mentioned above can be understood as a two-dimensional picture of the target object, that is, a 2D image.

[0015] In the above technical solution, it is worth pointing out that since the pose estimation is based on the directional vector graph, it can help the network learn the rigid structure of the object, cope with the situation where the target object is occluded, improve the ability to cope with occlusion, and thus improve the robustness of the pose estimation.

[0016] In the above technical solution, the first number is 18.

[0017] In the above technical solution, the camera can be understood as the camera of the robot.

[0018] In the above technical solution, the target object may be a workpiece to be grasped, a workpiece to be processed, a tool mounted on a robot, etc.

[0019] In the above technical solution, the second number is selected as 8, that is, the number of key points is 8.

[0020] Furthermore, the number of center points is one.

[0021] In the above technical solution, the third number is equal to 9.

[0022] In the above technical solution, point cloud data, also known as point cloud, is a dataset of points in a certain coordinate system. Points contain a wealth of information, including 3D coordinates (X, Y, and Z), color, classification value, intensity value, time, and much more. In my opinion, point clouds can atomize the real world, allowing us to restore the real world through high-precision point cloud data.

[0023] Among them, point cloud is mainly obtained by data collection through 3D laser scanner, and then 3D reconstruction is performed through 2D images to obtain point cloud data during the reconstruction process. In addition, some point clouds are obtained by calculation through 3D models.

[0024] In the above technical solution, the target detection model, that is, the model for performing target detection, serves as a branch of computer vision.

[0025] In the above technical solution, the direction vector graph can be understood as a direction vector field branch pointing to the key point position, so that the branch can be used to realize the pose estimation.

[0026] In addition, the posture estimation method proposed in this application also has the following additional technical features.

[0027] In the above technical solution, the two-dimensional image data is input into the target detection model to obtain a first number of directional vector graphs, including: using the target detection model to extract features from the two-dimensional image data to obtain a feature graph; performing region of interest detection on the feature graph to obtain a target area; performing regional feature aggregation on the feature graph according to the target area to obtain an aggregation result; performing convolution and transpose convolution operations on the features in the aggregation result to obtain a first number of directional vector graphs.

[0028] In this technical solution, the process of determining the directional vector map is specifically defined, that is, the specific definition process of the directional vector field branch is specifically defined, and the target detection model is used to perform feature extraction on the input two-dimensional image data to obtain a feature map of the two-dimensional image data, that is, a Feature map. ROI (Region of Interest) detection is performed on each layer of the feature pyramid of the target detection model to obtain the target area, and after determining the target area, the target area (that is, the ROI area) is used to perform ROI Align processing on the Feature map to obtain a processing result, wherein the processing result is also the aggregation result mentioned above.

[0029] Based on the aggregation result, it enters the Vector head and performs convolution and transpose convolution operations using the features of ROI Align to obtain 18 sets of feature maps, which are the first number of directional vector maps mentioned above.

[0030] In the above technical solution, the convolution and transposed convolution operations include the following steps:

[0031] The image first undergoes four convolutions with a kernel size of 3×3 and a stride of 1 (without changing the size of the feature map); then undergoes a transposed convolution to upsample to 28×28; then undergoes two convolutions with a kernel size of 3×3 and a stride of 1 (without changing the size of the feature map); then undergoes a transposed convolution to upsample to 56×56; and finally passes through a fully connected layer to output a 56×56×18 result, where 56×56 is the size of the feature map and 18 is the number of feature vectors.

[0032] In any of the above technical solutions, the direction vector diagram includes direction vector diagrams in the X-axis direction and the Y-axis direction in a Cartesian coordinate system.

[0033] In the above technical solution, the 18 characteristic vector graphs are respectively the X or Y direction vector graphs of 9 points.

[0034] In any of the above technical solutions, a first number of directional vector maps are processed to obtain two-dimensional coordinate data of a second number of key points, including: selecting two pixel points in the directional vector map pointing to the same key point in the first number of directional vector maps, and taking the intersection of the two pixel points as the hypothetical key point; repeatedly selecting pixel points to obtain a set of hypothetical key points; voting for each hypothetical key point in the set based on all pixel points of the target object to obtain a voting result; and taking the two-dimensional coordinate data of the largest hypothetical key point in the voting result as the coordinate data of the key point.

[0035] In the above technical solution, the method of obtaining the data coordinates of the key points is specifically defined. In this solution, in the vector diagram, the vector v k (P), the vector representing the distance from pixel p to the key point x of the object k The direction of pixel p and key point x k The offset between k -P. Command Represents that each pixel tends to the key point x k direction.

[0036] Among them, two pixels are randomly selected on the X and Y direction vector graphs pointing to the same key point, and the intersection of their vectors is used as the hypothesis h k,i , as the key point x k Repeat N times to generate a set of hypotheses {h k,i |i=1,2,…,N} is used to represent the possible keypoint locations. Then, all pixels of the object vote for these hypotheses. k,i The voting score w k,i Defined as

[0037]

[0038] Among them, θ is the threshold, P is the pixel point, p∈O, which means that the pixel P belongs to the target object O. Repeat N times to get the intersection point h of the maximum score k,i As key point location.

[0039] In any of the above technical solutions, the first number of directional vector maps are processed to obtain two-dimensional coordinate data of the center point, including: selecting two pixel points in the directional vector map pointing to the center point in the first number of directional vector maps, and taking the intersection of the two pixel points as the hypothetical center point; repeatedly selecting pixel points to obtain a set of hypothetical center points; voting for each hypothetical center point in the set based on all pixel points of the target object to obtain a voting result; and taking the two-dimensional coordinate data of the largest hypothetical center point in the voting result as the two-dimensional coordinate data of the center point.

[0040] In the above technical solution, the method of obtaining the data coordinates of the center point is specifically defined. In this solution, in the vector diagram, the vector v k (P), the vector represents the distance from pixel p to the center point x of the object k The direction of pixel p and center point x k The offset between k -P. Command Represents that each pixel tends to the center point x k direction.

[0041] Among them, two pixels are randomly selected on the X and Y direction vector graphs pointing to the same center point, and the intersection of their vectors is used as the hypothesis h k,i , as the center point x k Repeat N times to generate a set of hypotheses {h k,i |i=1,2,…,N} is used to represent the possible center point locations. Then, all pixels of the object vote for these hypotheses. k,i The voting score w k,i Defined as

[0042]

[0043] Among them, θ is the threshold, P is the pixel point, and p∈O means that the pixel P belongs to the target object O. Repeat N times to get the intersection point h with the maximum score k,i As the center point position.

[0044] In any of the above technical solutions, a random sampling consensus algorithm is used for voting.

[0045] In any of the above technical solutions, it also includes: performing convolution and full connection operations on the features in the aggregation results to obtain a classification result; performing convolution and transpose convolution operations on the features in the aggregation results to obtain an image mask; determining the image mask corresponding to the classification result; and screening the first number of directional vector graphs according to the image mask corresponding to the classification result.

[0046] In this technical solution, the direction vector graph is screened to reduce the amount of calculation when extracting the center point and key points, which provides a basis for reducing the complexity of the target detection model and also provides a basis for achieving real-time estimation of posture.

[0047] Specifically, using the image mask to screen the directional vector graph can be understood as processing only the directional vector graph of the Mask portion to achieve determination of the two-dimensional coordinate data of the center point and the key points.

[0048] The above technical solution also includes: doubling the image mask, selecting the corresponding Mask as the ROI according to the category, and extracting the corresponding area on the direction vector feature map, that is, only processing the vector of the Mask part.

[0049] In the above technical solution, the classification result can be based on the RCNN Head part after convolution and full connection to achieve classification and Bounding Box regression; and the image mask can be based on the MaskHead after multiple convolutions and upsampling to restore the 28×28 mask features.

[0050] Furthermore, the above classification results and image masks are implemented based on MaskRcnn.

[0051] In the above technical solution, the target detection model is implemented using the MaskRCNN framework. Specifically, the ResNet-FPN+Fast RCNN+mask structure is adopted. In the above structure, the main network ResNet-FPN is used for feature extraction. In addition, ResNet can also be ResNet-50 or ResNet-101, etc.

[0052] In any of the above technical solutions, the third quantity of three-dimensional coordinate data includes the third quantity of three-dimensional coordinate data of the center points and the third quantity of three-dimensional coordinate data of the key points, wherein the point cloud data of the target object is processed using the farthest point sampling algorithm to obtain the third quantity of three-dimensional coordinate data of the key points.

[0053] In this technical solution, key points are scattered on the target object. The farthest point sampling (FPS) algorithm is a very common sampling algorithm. It is widely used because it can ensure uniform sampling of samples. For example, PointNet++ in the 3D point cloud deep learning framework performs FPS sampling on sample points and then clusters them as the receptive field. The 3D object detection network VoteNet performs FPS sampling on the scattered points obtained by voting and then clusters them. The 6D pose estimation algorithm PVN3D is used to select 8 feature points of the object for voting and calculate the pose.

[0054] Specifically, the principles are as follows:

[0055] 1. The input point cloud has N points. A point P0 is selected from the point cloud as the starting point, and the sampling point set S = {P0} is obtained;

[0056] 2. Calculate the distances from all points to P0, form an N-dimensional array L, select the point corresponding to the maximum value as P1, and update the sampling point set S = {P0, P1};

[0057] 3. Calculate the distance from all points to P1. For each point Pi, if its distance to P1 is less than L[i], update L[i] = d(Pi, P1). Therefore, the array L always stores the closest distance from each point to the sampling point set S.

[0058] 4. Select the point corresponding to the maximum value in L as P2, and update the sampling point set S = {P0, P1, P2};

[0059] 5. Repeat 2 to 4 until N' target sampling points are reached.

[0060] In any of the above technical solutions, the Perspective-n-Point algorithm is used to determine the position and posture of the target object in the camera's coordinate system.

[0061] In any of the above technical solutions, the target detection model includes: a Feature pyramid network model.

[0062] The Feature Pyramid Network (FPN) addresses the multi-scale nature of objects in images. It resizes images to different sizes, extracts features of corresponding sizes, and then performs predictions. This approach can address the multi-scale issue to a certain extent.

[0063] In any of the above technical solutions, it also includes: outputting the position and posture of the target object in the coordinate system of the camera.

[0064] In this technical solution, the posture is output so that the robot's grasping posture can be adjusted according to the output posture, thereby improving the accuracy of grasping the target object.

[0065] According to the second aspect of the present invention, the present invention provides a posture estimation device, comprising: a receiving unit for receiving two-dimensional image data of a target object captured by a camera; an extraction unit for inputting the two-dimensional image data into a target detection model to obtain a first number of direction vector graphs; a first processing unit for processing the first number of direction vector graphs to obtain two-dimensional coordinate data of a center point and two-dimensional coordinate data of a second number of key points; a second processing unit for processing the point cloud data of the target object to obtain a third number of three-dimensional coordinate data; a determination unit for determining the pose of the target object in the camera's coordinate system based on the two-dimensional coordinate data of the center point, the two-dimensional coordinate data of the second number of key points, the third number of three-dimensional coordinate data, and the camera's parameter data.

[0066] The technical solution of the present application proposes a posture estimation device. A robot equipped with the estimation device can estimate the posture of a target object. In the above solution, the posture estimation of the target object is achieved using two-dimensional image data. Since the above operations are all based on two-dimensional image data, there is no need to use expensive 3D camera equipment, thereby reducing the estimation cost of posture estimation.

[0067] In addition, the above scheme uses two-dimensional image data for estimation, and two-dimensional operations have low requirements on the complexity of the target detection model. Therefore, it provides a basis for realizing rapid estimation of posture and also makes real-time estimation of posture possible.

[0068] In the above technical solution, the two-dimensional image data mentioned above can be understood as a two-dimensional picture of the target object, that is, a 2D image.

[0069] In the above technical solution, it is worth pointing out that since the pose estimation is based on the directional vector graph, it can help the network learn the rigid structure of the object, cope with the situation where the target object is occluded, improve the ability to cope with occlusion, and thus improve the robustness of the pose estimation.

[0070] In the above technical solution, the first number is 18.

[0071] In the above technical solution, the camera can be understood as the camera of the robot.

[0072] In the above technical solution, the target object may be a workpiece to be grasped, a workpiece to be processed, a tool mounted on a robot, etc.

[0073] In the above technical solution, the second number is selected as 8, that is, the number of key points is 8.

[0074] Furthermore, the number of center points is one.

[0075] In the above technical solution, the third number is equal to 9.

[0076] In the above technical solution, point cloud data, also known as point cloud, is a dataset of points in a certain coordinate system. Points contain a wealth of information, including 3D coordinates (X, Y, and Z), color, classification value, intensity value, time, and much more. In my opinion, point clouds can atomize the real world, allowing us to restore the real world through high-precision point cloud data.

[0077] Among them, point cloud is mainly obtained by data collection through 3D laser scanner, and then 3D reconstruction is performed through 2D images to obtain point cloud data during the reconstruction process. In addition, some point clouds are obtained by calculation through 3D models.

[0078] In the above technical solution, the target detection model, that is, the model for performing target detection, serves as a branch of computer vision.

[0079] In the above technical solution, the direction vector graph can be understood as a direction vector field branch pointing to the key point position, so that the branch can be used to realize the pose estimation.

[0080] In addition, the posture estimation device proposed in this application also has the following additional technical features.

[0081] In the above technical solution, the extraction unit is specifically used to: use the target detection model to extract features from the two-dimensional image data to obtain a feature map; perform region of interest detection on the feature map to obtain a target area; perform regional feature aggregation on the feature map according to the target area to obtain an aggregation result; perform convolution and transpose convolution operations on the features in the aggregation result to obtain a first number of directional vector maps.

[0082] In this technical solution, the process of determining the directional vector map is specifically defined, that is, the specific definition process of the directional vector field branch is specifically defined, and the target detection model is used to perform feature extraction on the input two-dimensional image data to obtain a feature map of the two-dimensional image data, that is, a Feature map. ROI (Region of Interest) detection is performed on each layer of the feature pyramid of the target detection model to obtain the target area, and after determining the target area, the target area (that is, the ROI area) is used to perform ROI Align processing on the Feature map to obtain a processing result, wherein the processing result is also the aggregation result mentioned above.

[0083] Based on the aggregation result, it enters the Vector head and performs convolution and transpose convolution operations using the features of ROI Align to obtain 18 sets of feature maps, which are the first number of directional vector maps mentioned above.

[0084] In the above technical solution, the convolution and transposed convolution operations include the following steps:

[0085] The image first undergoes four convolutions with a kernel size of 3×3 and a stride of 1 (without changing the size of the feature map); then undergoes a transposed convolution to upsample to 28×28; then undergoes two convolutions with a kernel size of 3×3 and a stride of 1 (without changing the size of the feature map); then undergoes a transposed convolution to upsample to 56×56; and finally passes through a fully connected layer to output a 56×56×18 result, where 56×56 is the size of the feature map and 18 is the number of feature vectors.

[0086] In any of the above technical solutions, the direction vector diagram includes direction vector diagrams in the X-axis direction and the Y-axis direction in a Cartesian coordinate system.

[0087] In the above technical solution, the 18 characteristic vector graphs are respectively the X or Y direction vector graphs of 9 points.

[0088] In any of the above technical solutions, the first processing unit is specifically used to: select two pixel points in the directional vector map pointing to the same key point in the first number of directional vector maps, and use the intersection of the two pixel points as the hypothetical key point; repeatedly select pixel points to obtain a set of hypothetical key points; vote for each hypothetical key point in the set based on all the pixel points of the target object to obtain a voting result; and use the two-dimensional coordinate data of the largest hypothetical key point in the voting result as the coordinate data of the key point.

[0089] In the above technical solution, the method of obtaining the data coordinates of the key points is specifically defined. In this solution, in the vector diagram, the vector v k (P), the vector representing the distance from pixel p to the key point x of the object k The direction of pixel p and key point x k The offset between k -P. Command Represents that each pixel tends to the key point x k direction.

[0090] Among them, two pixels are randomly selected on the X and Y direction vector graphs pointing to the same key point, and the intersection of their vectors is used as the hypothesis h k,i , as the key point x k Repeat N times to generate a set of hypotheses {h k,i|i=1,2,…,N} is used to represent the possible keypoint locations. Then, all pixels of the object vote for these hypotheses. k,i The voting score w k,i Defined as

[0091]

[0092] Among them, θ is the threshold, P is the pixel point, and p∈O means that the pixel P belongs to the target object O. Repeat N times to get the intersection point h with the maximum score k,i As key point location.

[0093] In any of the above technical solutions, the first processing unit is specifically used to: select two pixel points in the directional vector map pointing to the center point in the first number of directional vector maps, and use the intersection of the two pixel points as the hypothetical center point; repeatedly select pixel points to obtain a set of hypothetical center points; vote for each hypothetical center point in the set based on all the pixel points of the target object to obtain a voting result; and use the two-dimensional coordinate data of the largest hypothetical center point in the voting result as the two-dimensional coordinate data of the center point.

[0094] In the above technical solution, the method of obtaining the data coordinates of the center point is specifically defined. In this solution, in the vector diagram, the vector v k (P), the vector represents the distance from pixel p to the center point x of the object k The direction of pixel p and center point x k The offset between k -P. Command Represents that each pixel tends to the center point x k direction.

[0095] Among them, two pixels are randomly selected on the X and Y direction vector graphs pointing to the same center point, and the intersection of their vectors is used as the hypothesis h k,i , as the center point x k Repeat N times to generate a set of hypotheses {h k,i |i=1,2,…,N} is used to represent the possible center point locations. Then, all pixels of the object vote for these hypotheses. k,i The voting score w k,i Defined as

[0096]

[0097] Among them, θ is the threshold, P is the pixel point, and p∈O means that the pixel P belongs to the target object O. Repeat N times to get the intersection point h with the maximum score k,i As the center point position.

[0098] In any of the above technical solutions, a random sampling consensus algorithm is used for voting.

[0099] In any of the above technical solutions, the extraction unit is also used to: perform convolution and full connection operations on the features in the aggregation results to obtain a classification result; perform convolution and transpose convolution operations on the features in the aggregation results to obtain an image mask; determine the image mask corresponding to the classification result; and screen the first number of directional vector graphs according to the image mask corresponding to the classification result.

[0100] In this technical solution, the direction vector graph is screened to reduce the amount of calculation when extracting the center point and key points, which provides a basis for reducing the complexity of the target detection model and also provides a basis for achieving real-time estimation of posture.

[0101] Specifically, using the image mask to screen the directional vector graph can be understood as processing only the directional vector graph of the Mask portion to achieve determination of the two-dimensional coordinate data of the center point and the key points.

[0102] The above technical solution also includes: doubling the image mask, selecting the corresponding Mask as the ROI according to the category, and extracting the corresponding area on the direction vector feature map, that is, only processing the vector of the Mask part.

[0103] In the above technical solution, the classification result can be based on the RCNN Head part after convolution and full connection to achieve classification and Bounding Box regression; and the image mask can be based on the MaskHead after multiple convolutions and upsampling to restore the 28×28 mask features.

[0104] Furthermore, the above classification results and image masks are implemented based on MaskRcnn.

[0105] In the above technical solution, the target detection model is implemented using the MaskRCNN framework. Specifically, the ResNet-FPN+Fast RCNN+mask structure is adopted. In the above structure, the main network ResNet-FPN is used for feature extraction. In addition, ResNet can also be ResNet-50 or ResNet-101, etc.

[0106] In any of the above technical solutions, the third quantity of three-dimensional coordinate data includes the third quantity of three-dimensional coordinate data of the center points and the third quantity of three-dimensional coordinate data of the key points, wherein the point cloud data of the target object is processed using the farthest point sampling algorithm to obtain the third quantity of three-dimensional coordinate data of the key points.

[0107] In this technical solution, key points are scattered on the target object. The farthest point sampling (FPS) algorithm is a very common sampling algorithm. It is widely used because it can ensure uniform sampling of samples. For example, PointNet++ in the 3D point cloud deep learning framework performs FPS sampling on sample points and then clusters them as the receptive field. The 3D object detection network VoteNet performs FPS sampling on the scattered points obtained by voting and then clusters them. The 6D pose estimation algorithm PVN3D is used to select 8 feature points of the object for voting and calculate the pose.

[0108] Specifically, the principles are as follows:

[0109] 1. The input point cloud has N points. A point P0 is selected from the point cloud as the starting point, and the sampling point set S = {P0} is obtained;

[0110] 2. Calculate the distances from all points to P0, form an N-dimensional array L, select the point corresponding to the maximum value as P1, and update the sampling point set S = {P0, P1};

[0111] 3. Calculate the distance from all points to P1. For each point Pi, if its distance to P1 is less than L[i], update L[i] = d(Pi, P1). Therefore, the array L always stores the closest distance from each point to the sampling point set S.

[0112] 4. Select the point corresponding to the maximum value in L as P2, and update the sampling point set S = {P0, P1, P2};

[0113] 5. Repeat 2 to 4 until N target sampling points are reached.

[0114] In any of the above technical solutions, the Perspective-n-Point algorithm is used to determine the position and posture of the target object in the camera's coordinate system.

[0115] In any of the above technical solutions, a random sampling consensus algorithm is used for voting.

[0116] In any of the above technical solutions, the target detection model includes: a Feature pyramid network model.

[0117] The Feature Pyramid Network (FPN) addresses the multi-scale nature of objects in images. It resizes images to different sizes, extracts features of corresponding sizes, and then performs predictions. This approach can address the multi-scale issue to a certain extent.

[0118] In any of the above technical solutions, the determination unit is further configured to: output the position and posture of the target object in the coordinate system of the camera.

[0119] In this technical solution, the posture is output so that the robot's grasping posture can be adjusted according to the output posture, thereby improving the accuracy of grasping the target object.

[0120] According to the third aspect of the present invention, the present invention provides a posture estimation device, comprising: a controller and a memory, wherein the memory stores a program or instruction, and the controller implements the steps of any of the above methods when executing the program or instruction in the memory.

[0121] According to a fourth aspect of the present invention, the present invention provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of any of the above methods are implemented.

[0122] According to a fifth aspect of the present invention, the present invention provides a robot comprising: any one of the posture estimation devices described above; or the readable storage medium described above.

[0123] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0125] Figure 1 A schematic diagram of a flow chart of a posture estimation method in an embodiment of the present invention is shown;

[0126] Figure 2 A schematic diagram of a target detection model in an embodiment of the present invention is shown;

[0127] Figure 3 A schematic diagram of a target detection model in an embodiment of the present invention is shown;

[0128] Figure 4 A schematic diagram showing three-dimensional coordinate data in an embodiment of the present invention is shown;

[0129] Figure 5 A schematic diagram showing the pose of a target object in a camera coordinate system according to an embodiment of the present invention is shown;

[0130] Figure 6 A schematic diagram of a flow chart of a posture estimation method in an embodiment of the present invention is shown;

[0131] Figure 7A schematic block diagram of a posture estimation device in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0132] In order to more clearly understand the above aspects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features therein can be combined with each other in the absence of conflict.

[0133] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0134] Refer to the following Figures 1 to 7 The present invention describes a posture estimation method, an apparatus, a readable storage medium, and a robot according to some embodiments of the present invention.

[0135] Example 1

[0136] like Figure 1 As shown, according to a first aspect of the present invention, the present invention provides a posture estimation method, comprising:

[0137] Step 102, receiving two-dimensional image data of a target object captured by a camera;

[0138] Step 104: input the two-dimensional image data into the target detection model to obtain a first number of direction vector graphs;

[0139] Step 106: Process the first number of directional vector graphs to obtain two-dimensional coordinate data of the center point and two-dimensional coordinate data of the second number of key points;

[0140] Step 108 , processing the point cloud data of the target object to obtain a third amount of three-dimensional coordinate data;

[0141] Step 110 , determining the pose of the target object in the camera's coordinate system based on the two-dimensional coordinate data of the center point, the two-dimensional coordinate data of the second number of key points, the three-dimensional coordinate data of the third number, and the camera's parameter data.

[0142] The technical solution of the present application proposes a posture estimation method. By running the estimation method, the posture estimation of the target object can be achieved. In the above solution, the posture estimation of the target object is achieved using two-dimensional image data. Since the above operations are all based on two-dimensional image data, there is no need to use expensive 3D camera equipment, thereby reducing the estimation cost of the posture estimation.

[0143] In addition, the above scheme uses two-dimensional image data for estimation, and two-dimensional operations have low requirements on the complexity of the target detection model. Therefore, it provides a basis for realizing rapid estimation of posture and also makes real-time estimation of posture possible.

[0144] In the above technical solution, the two-dimensional image data mentioned above can be understood as a two-dimensional picture of the target object, that is, a 2D image.

[0145] In the above technical solution, it is worth pointing out that since the pose estimation is based on the directional vector graph, it can help the network learn the rigid structure of the object, cope with the situation where the target object is occluded, improve the ability to cope with occlusion, and thus improve the robustness of the pose estimation.

[0146] In the above technical solution, the first number is 18.

[0147] In the above technical solution, the camera can be understood as the camera of the robot.

[0148] In the above technical solution, the target object may be a workpiece to be grasped, a workpiece to be processed, a tool mounted on a robot, etc.

[0149] In the above technical solution, the second number is selected as 8, that is, the number of key points is 8.

[0150] Furthermore, the number of center points is one.

[0151] In the above technical solution, the third number is equal to 9.

[0152] In the above technical solution, point cloud data, also known as point cloud, is a dataset of points in a certain coordinate system. Points contain a wealth of information, including 3D coordinates (X, Y, and Z), color, classification value, intensity value, time, and much more. In my opinion, point clouds can atomize the real world, allowing us to restore the real world through high-precision point cloud data.

[0153] Among them, point cloud is mainly obtained by data collection through 3D laser scanner, and then 3D reconstruction is performed through 2D images to obtain point cloud data during the reconstruction process. In addition, some point clouds are obtained by calculation through 3D models.

[0154] In the above technical solution, the target detection model, that is, the model for performing target detection, serves as a branch of computer vision.

[0155] In the above technical solution, the direction vector graph can be understood as a direction vector field branch pointing to the key point position, so that the branch can be used to realize the pose estimation.

[0156] In addition, the posture estimation method proposed in this application also has the following additional technical features.

[0157] In the above technical solution, the two-dimensional image data is input into the target detection model to obtain a first number of directional vector graphs, including: using the target detection model to extract features from the two-dimensional image data to obtain a feature graph; performing region of interest detection on the feature graph to obtain a target area; performing regional feature aggregation on the feature graph according to the target area to obtain an aggregation result; performing convolution and transpose convolution operations on the features in the aggregation result to obtain a first number of directional vector graphs.

[0158] In this technical solution, the process of determining the directional vector map is specifically defined, that is, the specific definition process of the directional vector field branch is specifically defined, and the target detection model is used to perform feature extraction on the input two-dimensional image data to obtain a feature map of the two-dimensional image data, that is, a Feature map. ROI (Region of Interest) detection is performed on each layer of the feature pyramid of the target detection model to obtain the target area, and after determining the target area, the target area (that is, the ROI area) is used to perform ROI Align processing on the Feature map to obtain a processing result, wherein the processing result is also the aggregation result mentioned above.

[0159] Based on the aggregation result, it enters the Vector head and performs convolution and transpose convolution operations using the features of ROI Align to obtain 18 sets of feature maps, which are the first number of directional vector maps mentioned above.

[0160] In the above technical solution, the convolution and transposed convolution operations include the following steps:

[0161] The image first undergoes four convolutions with a kernel size of 3×3 and a stride of 1 (without changing the size of the feature map); then undergoes a transposed convolution to upsample to 28×28; then undergoes two convolutions with a kernel size of 3×3 and a stride of 1 (without changing the size of the feature map); then undergoes a transposed convolution to upsample to 56×56; and finally passes through a fully connected layer to output a 56×56×18 result, where 56×56 is the size of the feature map and 18 is the number of feature vectors.

[0162] In any of the above technical solutions, the direction vector diagram includes direction vector diagrams in the X-axis direction and the Y-axis direction in a Cartesian coordinate system.

[0163] In the above technical solution, the 18 characteristic vector graphs are respectively the X or Y direction vector graphs of 9 points.

[0164] In any of the above technical solutions, a first number of directional vector maps are processed to obtain two-dimensional coordinate data of a second number of key points, including: selecting two pixel points in the directional vector map pointing to the same key point in the first number of directional vector maps, and taking the intersection of the two pixel points as the hypothetical key point; repeatedly selecting pixel points to obtain a set of hypothetical key points; voting for each hypothetical key point in the set based on all pixel points of the target object to obtain a voting result; and taking the two-dimensional coordinate data of the largest hypothetical key point in the voting result as the coordinate data of the key point.

[0165] In the above technical solution, the method of obtaining the data coordinates of the key points is specifically defined. In this solution, in the vector diagram, the vector v k (P), the vector representing the distance from pixel p to the key point x of the object k The direction of pixel p and key point x k The offset between k -P. Command Represents that each pixel tends to the key point x k direction.

[0166] Among them, two pixels are randomly selected on the X and Y direction vector graphs pointing to the same key point, and the intersection of their vectors is used as the hypothesis h k,i , as the key point x k Repeat N times to generate a set of hypotheses {h k,i |i=1,2,…,N} is used to represent the possible keypoint locations. Then, all pixels of the object vote for these hypotheses. k,i The voting score w k,i Defined as

[0167]

[0168] Among them, θ is the threshold, P is the pixel point, and p∈O means that the pixel P belongs to the target object O. Repeat N times to get the intersection point h with the maximum score k,i As key point location.

[0169] In any of the above technical solutions, the first number of directional vector maps are processed to obtain two-dimensional coordinate data of the center point, including: selecting two pixel points in the directional vector map pointing to the center point in the first number of directional vector maps, and taking the intersection of the two pixel points as the hypothetical center point; repeatedly selecting pixel points to obtain a set of hypothetical center points; voting for each hypothetical center point in the set based on all pixel points of the target object to obtain a voting result; and taking the two-dimensional coordinate data of the largest hypothetical center point in the voting result as the two-dimensional coordinate data of the center point.

[0170] In the above technical solution, the method of obtaining the data coordinates of the center point is specifically defined. In this solution, in the vector diagram, the vector v k (P), the vector represents the distance from pixel p to the center point x of the object k The direction of pixel p and center point x k The offset between k -P. Command Represents that each pixel tends to the center point x k direction.

[0171] Among them, two pixels are randomly selected on the X and Y direction vector graphs pointing to the same center point, and the intersection of their vectors is used as the hypothesis h k,i , as the center point x k Repeat N times to generate a set of hypotheses {h k,i |i=1,2,…,N} is used to represent the possible center point locations. Then, all pixels of the object vote for these hypotheses. k,i The voting score w k,i Defined as

[0172]

[0173] Among them, θ is the threshold, P is the pixel point, and p∈O means that the pixel P belongs to the target object O. Repeat N times to get the intersection point h with the maximum score k,i As the center point position.

[0174] In any of the above technical solutions, a random sampling consensus algorithm is used for voting.

[0175] In any of the above technical solutions, it also includes: performing convolution and full connection operations on the features in the aggregation results to obtain a classification result; performing convolution and transpose convolution operations on the features in the aggregation results to obtain an image mask; determining the image mask corresponding to the classification result; and screening the first number of directional vector graphs according to the image mask corresponding to the classification result.

[0176] In this technical solution, the direction vector graph is screened to reduce the amount of calculation when extracting the center point and key points, which provides a basis for reducing the complexity of the target detection model and also provides a basis for achieving real-time estimation of posture.

[0177] Specifically, using the image mask to screen the directional vector graph can be understood as processing only the directional vector graph of the Mask portion to achieve determination of the two-dimensional coordinate data of the center point and the key points.

[0178] The above technical solution also includes: doubling the image mask, selecting the corresponding Mask as the ROI according to the category, and extracting the corresponding area on the direction vector feature map, that is, only processing the vector of the Mask part.

[0179] In the above technical solution, if Figure 2 As shown in the figure, the classification result can be based on the RCNN Head part after convolution and full connection to achieve classification and Bounding Box regression; and the image mask can be based on the MaskHead after multiple convolutions and upsampling to restore the 28×28 mask features.

[0180] Furthermore, the above classification results and image masks are implemented based on MaskRcnn.

[0181] In the above technical solution, if Figure 3 As shown in the figure, the target detection model is implemented using the MaskRCNN framework. Specifically, the ResNet-FPN+Fast RCNN+mask structure is adopted. In the above structure, the main network ResNet-FPN is used for feature extraction. In addition, ResNet can also be ResNet-50 or ResNet-101.

[0182] In any of the above technical solutions, the third quantity of three-dimensional coordinate data includes the third quantity of three-dimensional coordinate data of the center points and the third quantity of three-dimensional coordinate data of the key points, wherein the point cloud data of the target object is processed using the farthest point sampling algorithm to obtain the third quantity of three-dimensional coordinate data of the key points.

[0183] In this technical solution, if Figure 4 As shown in the figure, key points are scattered on the target object. Among them, the farthest point sampling algorithm, also known as the farthest point sampling (FPS) algorithm, is a very common sampling algorithm. It is widely used because it can ensure uniform sampling of samples. For example, PointNet++ in the 3D point cloud deep learning framework performs FPS sampling on sample points and then clusters them as the receptive field. The 3D object detection network VoteNet performs FPS sampling on the scattered points obtained by voting and then clusters them. The 6D pose estimation algorithm PVN3D is used to select 8 feature points of the object for voting and calculate the pose.

[0184] Specifically, the principles are as follows:

[0185] 1. The input point cloud has N points. A point P0 is selected from the point cloud as the starting point, and the sampling point set S = {P0} is obtained;

[0186] 2. Calculate the distances from all points to P0, form an N-dimensional array L, select the point corresponding to the maximum value as P1, and update the sampling point set S = {P0, P1};

[0187] 3. Calculate the distance from all points to P1. For each point Pi, if its distance to P1 is less than L[i], update L[i] = d(Pi, P1). Therefore, the array L always stores the closest distance from each point to the sampling point set S.

[0188] 4. Select the point corresponding to the maximum value in L as P2, and update the sampling point set S = {P0, P1, P2};

[0189] 5. Repeat 2 to 4 until N' target sampling points are reached.

[0190] In any of the above technical solutions, the Perspective-n-Point algorithm is used to determine the position and posture of the target object in the camera's coordinate system.

[0191] In any of the above technical solutions, the target detection model includes: a Feature pyramid network model.

[0192] The Feature Pyramid Network (FPN) addresses the multi-scale nature of objects in images. It resizes images to different sizes, extracts features of corresponding sizes, and then performs predictions. This approach can address the multi-scale issue to a certain extent.

[0193] In any of the above technical solutions, if Figure 5 As shown, it also includes: outputting the position and posture of the target object in the camera's coordinate system.

[0194] In this technical solution, the posture is output so that the robot's grasping posture can be adjusted according to the output posture, thereby improving the accuracy of grasping the target object.

[0195] In one embodiment, Figure 6 As shown, the pose estimation method includes:

[0196] Step 202, obtaining a picture from a camera;

[0197] Step 204: extract image features using FPN network neural convolution processing;

[0198] Step 206: perform region proposal detection (ROI) on each layer of the FPN feature pyramid.

[0199] Step 208, using ROI to perform ROI Align on the feature map;

[0200] Step 210: Enter the RCNN head and perform convolution and full connection operations using the features of ROI Align to obtain the class and box.

[0201] Step 212: Enter the mask head and perform convolution and transpose convolution operations using the features of ROI Align to obtain a mask; output the corresponding mask according to the category detected by the RCNN head;

[0202] Step 214: Enter the Vector head and perform convolution and transpose convolution operations using the features of ROI Align to obtain 18 sets of feature maps, corresponding to the X vector map and Y vector map of 8 feature points and 1 center point;

[0203] Step 216, further extracting the valid area of ​​the feature map obtained by the Vector head based on the mask detected by the mask head;

[0204] Step 218: Use Ransac to vote on the X vector and Y vector of the same key point, and the position with the highest votes is the key point;

[0205] Step 220, using the obtained 8 2D key points, the camera intrinsic parameters, and the 8 known 3D coordinates of the key points, use PNP to calculate the pose of the object in the camera coordinate system;

[0206] Step 222, repeat the above steps until the postures of all ROI targets are calculated and the results are output.

[0207] Example 2

[0208] like Figure 7 As shown, the present invention provides a posture estimation device 300, including: a receiving unit 302, for receiving two-dimensional image data of a target object captured by a camera; an extraction unit 304, for inputting the two-dimensional image data into a target detection model to obtain a first number of direction vector graphs; a first processing unit 306, for processing the first number of direction vector graphs to obtain two-dimensional coordinate data of a center point and two-dimensional coordinate data of a second number of key points; a second processing unit 308, for processing point cloud data of the target object to obtain a third number of three-dimensional coordinate data; a determination unit 310, for determining the position and posture of the target object in the camera's coordinate system based on the two-dimensional coordinate data of the center point, the two-dimensional coordinate data of the second number of key points, the third number of three-dimensional coordinate data and the camera's parameter data.

[0209] The technical solution of the present application proposes a posture estimation device 300. A robot equipped with the estimation device can estimate the posture of a target object. In the above solution, the posture estimation of the target object is achieved using two-dimensional image data. Since the above operations are all based on two-dimensional image data, there is no need to use expensive 3D camera equipment, thereby reducing the estimation cost of the posture estimation.

[0210] In addition, the above scheme uses two-dimensional image data for estimation, and two-dimensional operations have low requirements on the complexity of the target detection model. Therefore, it provides a basis for realizing rapid estimation of posture and also makes real-time estimation of posture possible.

[0211] In the above technical solution, the two-dimensional image data mentioned above can be understood as a two-dimensional picture of the target object, that is, a 2D image.

[0212] In the above technical solution, it is worth pointing out that since the pose estimation is based on the directional vector graph, it can help the network learn the rigid structure of the object, cope with the situation where the target object is occluded, improve the ability to cope with occlusion, and thus improve the robustness of the pose estimation.

[0213] In the above technical solution, the first number is 18.

[0214] In the above technical solution, the camera can be understood as the camera of the robot.

[0215] In the above technical solution, the target object may be a workpiece to be grasped, a workpiece to be processed, a tool mounted on a robot, etc.

[0216] In the above technical solution, the second number is selected as 8, that is, the number of key points is 8.

[0217] Furthermore, the number of center points is one.

[0218] In the above technical solution, the third number is equal to 9.

[0219] In the above technical solution, point cloud data, also known as point cloud, is a dataset of points in a certain coordinate system. Points contain a wealth of information, including 3D coordinates (X, Y, and Z), color, classification value, intensity value, time, and much more. In my opinion, point clouds can atomize the real world, allowing us to restore the real world through high-precision point cloud data.

[0220] Among them, point cloud is mainly obtained by data collection through 3D laser scanner, and then 3D reconstruction is performed through 2D images to obtain point cloud data during the reconstruction process. In addition, some point clouds are obtained by calculation through 3D models.

[0221] In the above technical solution, the target detection model, that is, the model for performing target detection, serves as a branch of computer vision.

[0222] In the above technical solution, the direction vector graph can be understood as a direction vector field branch pointing to the key point position, so that the branch can be used to realize the pose estimation.

[0223] In addition, the posture estimation device proposed in this application also has the following additional technical features.

[0224] In the above technical solution, the extraction unit 304 is specifically used to: use the target detection model to extract features from the two-dimensional image data to obtain a feature map; perform region of interest detection on the feature map to obtain a target area; perform regional feature aggregation on the feature map according to the target area to obtain an aggregation result; perform convolution and transpose convolution operations on the features in the aggregation result to obtain a first number of directional vector maps.

[0225] In this technical solution, the process of determining the directional vector map is specifically defined, that is, the specific definition process of the directional vector field branch is specifically defined, and the target detection model is used to perform feature extraction on the input two-dimensional image data to obtain a feature map of the two-dimensional image data, that is, a Feature map. ROI (Region of Interest) detection is performed on each layer of the feature pyramid of the target detection model to obtain the target area, and after determining the target area, the target area (that is, the ROI area) is used to perform ROI Align processing on the Feature map to obtain a processing result, wherein the processing result is also the aggregation result mentioned above.

[0226] Based on the aggregation result, it enters the Vector head and performs convolution and transpose convolution operations using the features of ROI Align to obtain 18 sets of feature maps, which are the first number of directional vector maps mentioned above.

[0227] In the above technical solution, the convolution and transposed convolution operations include the following steps:

[0228] The image first undergoes four convolutions with a kernel size of 3×3 and a stride of 1 (without changing the size of the feature map); then undergoes a transposed convolution to upsample to 28×28; then undergoes two convolutions with a kernel size of 3×3 and a stride of 1 (without changing the size of the feature map); then undergoes a transposed convolution to upsample to 56×56; and finally passes through a fully connected layer to output a 56×56×18 result, where 56×56 is the size of the feature map and 18 is the number of feature vectors.

[0229] In any of the above technical solutions, the direction vector diagram includes direction vector diagrams in the X-axis direction and the Y-axis direction in a Cartesian coordinate system.

[0230] In the above technical solution, the 18 characteristic vector graphs are respectively the X or Y direction vector graphs of 9 points.

[0231] In any of the above technical solutions, the first processing unit 306 is specifically used to: select two pixel points in the directional vector map pointing to the same key point in the first number of directional vector maps, and use the intersection of the two pixel points as the hypothetical key point; repeatedly select pixel points to obtain a set of hypothetical key points; vote for each hypothetical key point in the set based on all the pixel points of the target object to obtain a voting result; and use the two-dimensional coordinate data of the largest hypothetical key point in the voting result as the coordinate data of the key point.

[0232] In the above technical solution, the method of obtaining the data coordinates of the key points is specifically defined. In this solution, in the vector diagram, the vector v k (P), the vector representing the distance from pixel p to the key point x of the object k The direction of pixel p and key point x k The offset between k -P. Command Represents that each pixel tends to the key point x k direction.

[0233] Among them, two pixels are randomly selected on the X and Y direction vector graphs pointing to the same key point, and the intersection of their vectors is used as the hypothesis h k,i , as the key point x k Repeat N times to generate a set of hypotheses {h k,i |i=1,2,…,N} is used to represent the possible keypoint locations. Then, all pixels of the object vote for these hypotheses. k,i The voting score w k,i Defined as

[0234]

[0235] Among them, θ is the threshold, P is the pixel point, and p∈O means that the pixel P belongs to the target object O. Repeat N times to get the intersection point h with the maximum score k,i As key point location.

[0236] In any of the above technical solutions, the first processing unit 306 is specifically used to: select two pixel points in the directional vector map pointing to the center point in the first number of directional vector maps, and use the intersection of the two pixel points as the hypothetical center point; repeatedly select pixel points to obtain a set of hypothetical center points; vote for each hypothetical center point in the set based on all the pixel points of the target object to obtain a voting result; and use the two-dimensional coordinate data of the largest hypothetical center point in the voting result as the two-dimensional coordinate data of the center point.

[0237] In the above technical solution, the method of obtaining the data coordinates of the center point is specifically defined. In this solution, in the vector diagram, the vector v k (P), the vector represents the distance from pixel p to the center point x of the object k The direction of pixel p and center point x k The offset between k -P. Command Represents that each pixel tends to the center point x k direction.

[0238] Among them, two pixels are randomly selected on the X and Y direction vector graphs pointing to the same center point, and the intersection of their vectors is used as the hypothesis h k,i , as the center point x k Repeat N times to generate a set of hypotheses {h k,i |i=1,2,…,N} is used to represent the possible center point locations. Then, all pixels of the object vote for these hypotheses. k,i The voting score w k,i Defined as

[0239]

[0240] Among them, θ is the threshold, P is the pixel point, and p∈O means that the pixel P belongs to the target object O. Repeat N times to get the intersection point h with the maximum score k,i As the center point position.

[0241] In any of the above technical solutions, a random sampling consensus algorithm is used for voting.

[0242] In any of the above technical solutions, the extraction unit 304 is also used to: perform convolution and full connection operations on the features in the aggregation results to obtain a classification result; perform convolution and transpose convolution operations on the features in the aggregation results to obtain an image mask; determine the image mask corresponding to the classification result; and filter the first number of directional vector graphs according to the image mask corresponding to the classification result.

[0243] In this technical solution, the direction vector graph is screened to reduce the amount of calculation when extracting the center point and key points, which provides a basis for reducing the complexity of the target detection model and also provides a basis for achieving real-time estimation of posture.

[0244] Specifically, using the image mask to screen the directional vector graph can be understood as processing only the directional vector graph of the Mask portion to achieve determination of the two-dimensional coordinate data of the center point and the key points.

[0245] The above technical solution also includes: doubling the image mask, selecting the corresponding Mask as the ROI according to the category, and extracting the corresponding area on the direction vector feature map, that is, only processing the vector of the Mask part.

[0246] In the above technical solution, the classification result can be based on the RCNN Head part after convolution and full connection to achieve classification and Bounding Box regression; and the image mask can be based on the MaskHead after multiple convolutions and upsampling to restore the 28×28 mask features.

[0247] Furthermore, the above classification results and image masks are implemented based on MaskRcnn.

[0248] In the above technical solution, the target detection model is implemented using the MaskRCNN framework. Specifically, the ResNet-FPN+Fast RCNN+mask structure is adopted. In the above structure, the main network ResNet-FPN is used for feature extraction. In addition, ResNet can also be ResNet-50 or ResNet-101, etc.

[0249] In any of the above technical solutions, the third quantity of three-dimensional coordinate data includes the third quantity of three-dimensional coordinate data of the center points and the third quantity of three-dimensional coordinate data of the key points, wherein the point cloud data of the target object is processed using the farthest point sampling algorithm to obtain the third quantity of three-dimensional coordinate data of the key points.

[0250] In this technical solution, key points are scattered on the target object. The farthest point sampling (FPS) algorithm is a very common sampling algorithm. It is widely used because it can ensure uniform sampling of samples. For example, PointNet++ in the 3D point cloud deep learning framework performs FPS sampling on sample points and then clusters them as the receptive field. The 3D object detection network VoteNet performs FPS sampling on the scattered points obtained by voting and then clusters them. The 6D pose estimation algorithm PVN3D is used to select 8 feature points of the object for voting and calculate the pose.

[0251] Specifically, the principles are as follows:

[0252] 1. The input point cloud has N points. A point P0 is selected from the point cloud as the starting point, and the sampling point set S = {P0} is obtained;

[0253] 2. Calculate the distances from all points to P0, form an N-dimensional array L, select the point corresponding to the maximum value as P1, and update the sampling point set S = {P0, P1};

[0254] 3. Calculate the distance from all points to P1. For each point Pi, if its distance to P1 is less than L[i], update L[i] = d(Pi, P1). Therefore, the array L always stores the closest distance from each point to the sampling point set S.

[0255] 4. Select the point corresponding to the maximum value in L as P2, and update the sampling point set S = {P0, P1, P2};

[0256] 5. Repeat 2 to 4 until N' target sampling points are reached.

[0257] In any of the above technical solutions, the Perspective-n-Point algorithm is used to determine the position and posture of the target object in the camera's coordinate system.

[0258] In any of the above technical solutions, a random sampling consensus algorithm is used for voting.

[0259] In any of the above technical solutions, the target detection model includes: a Feature pyramid network model.

[0260] The Feature Pyramid Network (FPN) addresses the multi-scale nature of objects in images. It resizes images to different sizes, extracts features of corresponding sizes, and then performs predictions. This approach can address the multi-scale issue to a certain extent.

[0261] In any of the above technical solutions, the determining unit 310 is further configured to: output the position and posture of the target object in the coordinate system of the camera.

[0262] In this technical solution, the posture is output so that the robot's grasping posture can be adjusted according to the output posture, thereby improving the accuracy of grasping the target object.

[0263] Example 3

[0264] The present invention provides a posture estimation device, comprising: a controller and a memory, wherein the memory stores a program or instruction, and the controller implements the steps of any one of the above methods when executing the program or instruction in the memory.

[0265] An embodiment of the present application proposes a posture estimation device, wherein a controller in the posture estimation device executes a program or instruction stored in a memory to implement the steps of a posture estimation method as in any one of the above embodiments. Therefore, the posture estimation device has all the beneficial technical effects of the above-mentioned posture estimation method, which will not be repeated here.

[0266] The memory may include volatile memory or non-volatile memory, or the memory may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM). The memory in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0267] Example 4

[0268] The present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of any one of the above methods are implemented.

[0269] The technical solution of the present application proposes a readable storage medium, the program or instructions in which can implement the steps of any of the above-mentioned posture estimation methods, and therefore has all the beneficial technical effects of the above-mentioned posture estimation method, which will not be repeated here.

[0270] By running this estimation method, the pose of the target object can be estimated. In the above scheme, the pose estimation of the target object is achieved using two-dimensional image data. Since the above operations are all based on two-dimensional image data, there is no need to use expensive 3D camera equipment, thereby reducing the estimation cost of pose estimation.

[0271] Example 5

[0272] The present invention provides a robot, comprising: any one of the posture estimation devices described above; or the readable storage medium described above.

[0273] An embodiment of the present application proposes a robot having the above-mentioned posture estimation device or readable storage medium. Therefore, the robot has all the beneficial technical effects of the above-mentioned posture estimation device or readable storage medium, which will not be repeated here.

[0274] In the description of the present invention, the term "plurality" refers to two or more than two. Unless otherwise expressly defined, the orientations or positional relationships indicated by the terms "upper" and "lower" are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention. The terms "connect," "install," and "fix" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0275] In the description of the present invention, the terms "one embodiment," "some embodiments," "specific embodiments," etc., mean that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In the present invention, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0276] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A posture estimation method, characterized in that: include: Receiving two-dimensional image data of a target object captured by a camera; Inputting the two-dimensional image data into a target detection model to obtain a first number of direction vector graphs; Processing the first number of directional vector graphs to obtain two-dimensional coordinate data of a center point and two-dimensional coordinate data of a second number of key points; processing the point cloud data of the target object to obtain a third amount of three-dimensional coordinate data; Determining a pose of the target object in the coordinate system of the camera according to the two-dimensional coordinate data of the center point, the two-dimensional coordinate data of the second number of key points, the three-dimensional coordinate data of the third number, and the parameter data of the camera; Inputting the two-dimensional image data into a target detection model to obtain a first number of direction vector graphs includes: Performing feature extraction on the two-dimensional image data using the target detection model to obtain a feature map; Performing region of interest detection on the feature map to obtain a target region; Performing regional feature aggregation on the feature map according to the target area to obtain an aggregation result; Perform convolution and transpose convolution operations on the features in the aggregation result to obtain the first number of directional vector graphs.

2. The posture estimation method according to claim 1, wherein The direction vector diagram includes direction vector diagrams in the X-axis direction and the Y-axis direction in a Cartesian coordinate system.

3. The posture estimation method according to claim 1, wherein Processing the first number of directional vector graphs to obtain two-dimensional coordinate data of a second number of key points includes: Selecting two pixel points in the directional vector graphs pointing to the same key point in the first number of directional vector graphs, and taking the intersection of the two pixel points as a hypothetical key point; Repeatedly selecting the pixel points to obtain the set of hypothetical key points; Voting for each of the hypothetical key points in the set according to all pixel points of the target object to obtain a voting result; The two-dimensional coordinate data of the largest hypothetical key point in the voting result is used as the coordinate data of the key point.

4. The posture estimation method according to claim 1, wherein Processing the first number of directional vector graphs to obtain two-dimensional coordinate data of the center point includes: Selecting two pixel points in the directional vector graphs pointing to the center point in the first number of directional vector graphs, and taking the intersection of the two pixel points as the assumed center point; Repeatedly selecting the pixel points to obtain a set of the hypothetical center points; Voting for each of the hypothetical center points in the set based on all pixel points of the target object to obtain a voting result; The two-dimensional coordinate data of the largest hypothetical center point in the voting results is used as the two-dimensional coordinate data of the center point.

5. The posture estimation method according to claim 1, wherein Also includes: Performing convolution and full connection operations on the features in the aggregation result to obtain a classification result; Performing convolution and transposed convolution operations on the features in the aggregation result to obtain an image mask; Determining an image mask corresponding to the classification result; The first number of directional vector graphs are screened according to the image mask corresponding to the classification result.

6. The posture estimation method according to any one of claims 1 to 5, characterized in that The third amount of three-dimensional coordinate data includes a fourth amount of three-dimensional coordinate data of center points and a fifth amount of three-dimensional coordinate data of key points. The point cloud data of the target object is processed using a farthest point sampling algorithm to obtain three-dimensional coordinate data of the fifth number of key points.

7. The posture estimation method according to any one of claims 1 to 5, characterized in that The perspective-n-point algorithm is used to determine the position and orientation of the target object in the camera's coordinate system.

8. The posture estimation method according to claim 4, characterized in that Voting is performed using a random sampling consensus algorithm.

9. The posture estimation method according to any one of claims 1 to 5, characterized in that The target detection model includes: a Feature pyramid network model.

10. The posture estimation method according to claim 1 or 2, characterized in that: Also includes: Output the pose of the target object in the camera's coordinate system.

11. A posture estimation device, characterized in that: include: A receiving unit, configured to receive two-dimensional image data of a target object captured by a camera; an extraction unit, configured to input the two-dimensional image data into a target detection model to obtain a first number of directional vector graphs; a first processing unit, configured to process the first number of directional vector graphs to obtain two-dimensional coordinate data of a center point and two-dimensional coordinate data of a second number of key points; a second processing unit, configured to process the point cloud data of the target object to obtain a third amount of three-dimensional coordinate data; a determining unit, configured to determine a position and posture of the target object in the coordinate system of the camera based on the two-dimensional coordinate data of the center point, the two-dimensional coordinate data of the second number of key points, the three-dimensional coordinate data of the third number, and the parameter data of the camera; The extraction unit is further configured to: extract features from the two-dimensional image data using a target detection model to obtain a feature map; perform region of interest detection on the feature map to obtain a target region; and perform regional feature aggregation on the feature map according to the target region to obtain an aggregation result; Convolution and transposed convolution operations are performed on the features in the aggregation result to obtain a first number of direction vector maps.

12. A posture estimation device, characterized in that: include: A controller and a memory, wherein the memory stores a program or instruction, and the controller implements the steps of the method according to any one of claims 1 to 10 when executing the program or instruction in the memory.

13. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A robot, characterized in that: include: The posture estimation device according to claim 11 or 12; or The readable storage medium of claim 13.

Images