6D pose estimation method based on full-flow bidirectional fusion network

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a 6D pose estimation method based on a full-flow bidirectional fusion network, and the method comprises the steps: S1, collecting an RGB-D image of an estimation object, and converting the RGB-D image into point cloud data according to the internal reference of a camera; s2, extracting RGB texture features corresponding to the point cloud data, and processing the point cloud data to obtain spliced local-global features; s3, carrying out feature bidirectional fusion, wherein the feature bidirectional fusion comprises a pixel-to-point bidirectional fusion process and a point-to-pixel bidirectional fusion process; s4, carrying out pixel dense fusion, and combining a fusion result with the global features of dense fusion to obtain a prediction pose of each pixel point; and S5, inputting the predicted poses of all the pixel points into the final multilayer perceptron, continuously reducing the value of the overall loss function of the estimated object through training learning, and finally selecting the output with the minimum overall loss function value as the overall estimated pose of the estimated object. According to the method, the accuracy and robustness of pose estimation can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a 6D pose estimation method based on a full-flow bidirectional fusion network. Background Art

[0002] 6D pose estimation is to obtain a rigid transformation from the target object coordinate system to the camera coordinate system, including the 3D rotation matrix R and the 3D translation matrix T. 6D pose estimation is an important component in many real-world applications. For example, in the field of industrial robots, accurately identifying the 6D pose of the object to be grasped can improve the accuracy of grasping or assembly; in the field of augmented reality, accurate 6D pose estimation can accurately place virtual objects in the real world and enhance the immersive experience. In the field of autonomous driving, accurate 6D pose estimation can accurately identify the position and direction of other vehicles, pedestrians and obstacles in the surrounding environment, helping autonomous vehicles make safe decisions.

[0003] From the perspective of practical applications, the current 6D pose estimation methods can be divided into three types. The first category uses RGB images, which provide rich color information and are conducive to feature extraction and object recognition. However, due to the lack of depth information, it is sensitive to illumination changes and occlusion, which may lead to poor performance in the case of lack of texture or repeated texture. The second category is to use only depth images or point clouds, which can provide accurate three-dimensional spatial information, but is sensitive to some noise in the point cloud data itself and difficult to handle when the point cloud is sparse or occluded. The third category is to use RGB-D images, combining the color information of RGB images and the precise distance measurement of depth sensors. He Y (He Y.FFB6D:A Full Flow Bidirectional Fusion Networkfor 6D Pose Estimation.[J].2021) proposed a full-flow bidirectional fusion network that fuses each encoding and decoding layer during the learning process of RGB-D images. After learning the appearance and geometric information, the texture and geometric information of the object model are used to obtain the 3D key points of the object, and then the PnP algorithm is used to calculate the 6D pose parameters. However, the lack of texture information and surface reflection problems on the workpiece make it difficult to extract the 3D key points of the workpiece, which affects the effect of the network. Summary of the invention

[0004] The purpose of this application is to provide a 6D pose estimation method based on a full-flow bidirectional fusion network to solve the technical problems raised in the above background technology.

[0005] To achieve the above objectives, the present application discloses the following technical solution: a 6D pose estimation method based on a full-flow bidirectional fusion network, the method comprising the following steps:

[0006] Step S1: Collect an RGB-D image of the estimated object and convert the RGB-D image into point cloud data according to the camera intrinsic parameters;

[0007] Step S2: Input the RGB-D image into the pre-trained ResNet-34 network for encoding; then input the encoded data into the PSPNet network to decode the image and obtain the feature image; input the converted point cloud data into the PointNet++ network for encoding and decoding to obtain the spliced ​​local-global point cloud features;

[0008] Step S3: performing bidirectional feature fusion, wherein the bidirectional feature fusion includes a pixel-to-point and a point-to-pixel bidirectional fusion process;

[0009] The pixel-to-point fusion process specifically includes: in the pixel-to-point fusion module from image features to point cloud features, find the K coordinates of each 3D point in the XYZ map. r2p The nearest point, records the appearance feature F of the nearest point in the RGB feature map ri ; Use maximum pooling and multi-layer perceptron to integrate compressed appearance features F ri , and get the integrated appearance feature F r2p ; The point cloud feature F point and integrated appearance features F r2p Connect and obtain fused point features through MLPS

[0010] The point-to-pixel fusion process specifically includes: in the pixel-to-point fusion module from point cloud features to image features, find the nearest point of each feature pixel with XYZ coordinates, and record the geometric features F of the nearest point in the point cloud feature map. pj ; Use maximum pooling and multi-layer perceptron to integrate compressed geometric features F pj , get the integrated point feature F p2r , the image feature F rgb And the integrated point feature F p2r Connect to get the fused image features

[0011] Step S4: Perform pixel-dense fusion to associate the geometric information of each point with the projected pixel obtained using the camera intrinsic parameters; connect the point features of the corresponding pixels and image features Get feature F fr ; The connected feature F fr The input is integrated and compressed into a multi-layer perceptron, and the integrated features are input into the symmetric reduction function to obtain a global feature vector F of a fixed size. g ; Splicing feature F frand the global eigenvector F g Get pixel-by-pixel feature F frg ; The pixel-by-pixel feature F frg The input is sent to the final network to predict the estimated pose of each pixel. Through training and learning, the value of the loss function is continuously reduced, and finally the one with the smallest loss function value is selected as the predicted pose of each pixel.

[0012] Step S5: The predicted poses of all pixels are input into the final multi-layer perceptron, the confidence is averaged and maximum pooled, and the 6D pose of the estimated object is predicted by regression. Through training and learning, the value of the overall loss function of the estimated object is continuously reduced, and finally the output with the smallest overall loss function value is selected as the overall estimated pose of the estimated object.

[0013] Preferably, in step S3, the use of maximum pooling and multi-layer perceptron to integrate the compressed appearance feature F ri , and get the integrated appearance feature F r2p , which is calculated by the following formula:

[0014]

[0015] Among them, F ri is the i-th nearest pixel of the RGB texture feature.

[0016] Preferably, in step S3, the point cloud feature F point and integrated appearance features F r2p Connect and obtain fused point features through MLPS It is calculated by the following formula:

[0017]

[0018] in, It is a connection operation.

[0019] Preferably, in step S3, the maximum pooling and multi-layer perceptron are used to integrate the compressed geometric features F pj , get the integrated point feature F p2r , which is calculated by the following formula:

[0020]

[0021] Among them, F pi represents the jth nearest point feature.

[0022] Preferably, in step S3, the image feature F rgb And the integrated point feature F p2r Connect to get the fused image features It is calculated by the following formula:

[0023]

[0024] in, It is a connection operation.

[0025] Preferably, in step S3, when performing bidirectional feature fusion, the extracted RGB image features and point cloud features are effectively fused, and the pixel-level features are associated with the point-level features through the intrinsic parameter matrix; the XYZ image size is adjusted using the nearest neighbor interpolation algorithm to ensure the correspondence between the feature pixels and the 3D coordinates.

[0026] Preferably, in step S4, feature F fr It is expressed as:

[0027]

[0028] Among them, cat(·) represents the channel concatenation operation;

[0029] Global eigenvector F g It is expressed as:

[0030] F g =AP(MLP(F fr ))

[0031] Where AP(·) represents the average pooling operation;

[0032] Pixel-by-pixel feature F frg It is expressed as:

[0033] F frg =cat(F fr , F g ).

[0034] Preferably, in step S4, when the estimated object is an asymmetric object, the loss of posture estimation is defined as the distance between a point sampled from the object model under the actual posture and a corresponding point on the same model converted by the predicted posture, and the specific formula is:

[0035]

[0036] Among them, L i is the distance, x j represents the jth point among M 3D points randomly selected from the 3D model of the object, [R g |T g ] is the real posture, is the predicted pose generated from the fusion of the i-th dense pixel:

[0037] Preferably, in step S4, when the estimated object is a symmetrical object, the loss of posture estimation is defined as minimizing the distance between each point in the estimated model direction and the nearest point on the real model. The specific formula is:

[0038]

[0039] Among them, L i is the distance, x j represents the jth point among M 3D points randomly selected from the 3D model of the object, [R g |T g ] is the real posture, is the predicted pose generated from the fusion of the i-th dense pixels.

[0040] Preferably, in step S5, the estimated object overall loss function is formed by combining the estimated object's pose estimation loss function with the confidence score, and the calculation formula of the estimated object overall loss function L is:

[0041]

[0042] Among them, ci is the confidence of the predicted pose of the i-th sampling pixel point, ω is the weight of the confidence, which is a balanced hyperparameter, and N is the number of randomly extracted dense pixel features.

[0043] Beneficial effects: The 6D pose estimation method based on the full-flow bidirectional fusion network of the present application introduces pixel-to-point and point-to-pixel bidirectional fusion modules, which not only makes full use of the visual information in the RGB image, but also deeply mines the geometric features in the point cloud data, thereby achieving a high degree of information complementarity and enhancement at the feature level; at the same time, it also uses pixel-intensive fusion technology to independently process and predict each pixel feature, significantly improving the accuracy and robustness of pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A flowchart of a 6D pose estimation method based on a full-flow bidirectional fusion network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0047] In this article, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprising..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0048] This embodiment provides Figure 1 A 6D pose estimation method based on a full-flow bidirectional fusion network is shown, which specifically includes the following steps:

[0049] Step S1: Collect the RGB-D image of the estimated object and convert the RGB-D image into point cloud data according to the camera intrinsic parameters. The calculation formula is:

[0050]

[0051] Where d represents the depth value in the depth map, s is the scale factor, x' and y' are the pixel coordinates in the depth map, and c x 、c y 、f x 、f y are the intrinsic parameters of the camera, and (x, y, z) are the converted coordinates.

[0052] Step S2: Extract RGB texture features, input the RGB image into the pre-trained ResNet-34 network, encode the RGB image; input the encoded data into the PSPNet network to decode the image; extract point cloud features, input the point cloud data into the PointNet++ network for encoding and decoding, and obtain the spliced ​​local-global features.

[0053] Step S3: Implement bidirectional feature fusion, including pixel-to-point and point-to-pixel bidirectional fusion processes; in the pixel-to-point fusion module from image features to point cloud features, find the K coordinates of each 3D point in the XYZ map. r2p The nearest points, record their appearance features F in the RGB feature map ri ; Use max pooling and multi-layer perceptrons (MLPs) to integrate and compress F ri , and get the integrated appearance feature F r2p, the formula is:

[0054]

[0055] Among them, F ri is the i-th nearest pixel of the RGB feature, F r2p It is an integrated feature.

[0056] The appearance feature F r2p With point cloud feature F point Connect and obtain fused point features through MLPs The formula is:

[0057]

[0058] in It is a connection operation.

[0059] In the pixel-to-point fusion module from point cloud features to image features, find the nearest point of each feature pixel with XYZ coordinates and record their geometric features F in the point cloud feature map pj ; Use max pooling and multi-layer perceptrons (MLPs) to integrate the compressed point features F pj , get the integrated point feature F p2r , the formula is:

[0060]

[0061] Among them, F pj represents the jth nearest point feature, F p2r Represents the integrated point features.

[0062] The image feature F rgb and point feature F p2r Connect to get the fused image features The formula is:

[0063]

[0064] The extracted RGB image features and point cloud features are effectively fused, and the pixel-level features are associated with the point-level features through the intrinsic parameter matrix; the nearest neighbor interpolation algorithm is used to adjust the size of the XYZ image to ensure the correspondence between the feature pixels and the 3D coordinates.

[0065] Step S4: Implement pixel-dense fusion, associate the geometric information of each point with the projected pixel obtained using the camera intrinsic parameters; connect the point features of the corresponding pixels and image features Get feature F fr , feature F fr It is expressed as:

[0066]

[0067] Among them, cat(·) represents the channel concatenation operation;

[0068] The connected feature F fr The input is integrated and compressed into a multi-layer perceptron, and the integrated features are input into the symmetric reduction function to obtain a global feature vector F of a fixed size. g ; Splicing feature F fr and the global eigenvector F g Get pixel-by-pixel feature F frg ; The feature F frg Input to the final network to predict the estimated pose of each pixel. Among them, the global feature vector F g It is expressed as:

[0069] F g =AP(MLP(F fr )) (9)

[0070] Where AP(·) represents the average pooling operation;

[0071] Pixel-by-pixel feature F frg It is expressed as:

[0072] F frg =cat(F fr ,F g ) (10)

[0073] If the estimated object is an asymmetric object, the loss of pose estimation is defined as the distance between the point sampled from the object model at the actual pose and the corresponding point on the same model converted by the predicted pose, calculated as:

[0074]

[0075] where x j represents the jth point among M 3D points randomly selected from the 3D model of the object; [Rg|T g ] is the real posture, is the predicted pose generated from the fusion of the i-th dense pixels.

[0076] If the estimated object is a symmetric object, the loss is defined as minimizing the distance between each point in the estimated model direction and the nearest point on the true model. The calculation formula is:

[0077]

[0078] Through training and learning, the value of the loss function is continuously reduced, and finally the one with the smallest loss function value is selected as the predicted pose for each pixel.

[0079] Step S5: The estimated poses of all pixels are input into the final multi-layer perceptron, and then the confidence is averaged and max-pooled, and the 6D pose of the object is predicted by regression.

[0080] The pose estimation loss function of the object is combined with the confidence score to form the loss function of the object's overall 6D pose estimation. The calculation formula of the overall loss function L is:

[0081]

[0082] Among them, ci is the confidence of the predicted pose of the i-th sampled pixel point, ω is the weight of the confidence, which is a balanced hyperparameter, and N is the number of randomly extracted dense pixel features.

[0083] Through training and learning, the value of the overall loss function of the object is continuously reduced, and finally the output with the smallest overall loss function value is selected as the overall estimated pose of the object.

[0084] To summarize, the 6D pose estimation method based on the full-flow bidirectional fusion network of this embodiment introduces pixel-to-point and point-to-pixel bidirectional fusion modules, which not only makes full use of the visual information in the RGB image, but also deeply mines the geometric features in the point cloud data, thereby achieving a high degree of information complementarity and enhancement at the feature level; at the same time, it also uses pixel-dense fusion technology to independently process and predict each pixel feature, which significantly improves the accuracy and robustness of pose estimation.

[0085] In the embodiments provided in the present application, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code or any appropriate combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: application specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), processor, controller, microcontroller, microprocessor, other electronic units designed to implement the functions described herein or their combination. For software implementation, part or all of the flow of the embodiment can be completed by instructing the relevant hardware through a computer program. When implemented, the above program can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a computer can access. The computer-readable storage medium can include but is not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer.

[0086] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A 6D pose estimation method based on a full-flow bidirectional fusion network, characterized in that: The method comprises the following steps: Step S1: Collect an RGB-D image of the estimated object and convert the RGB-D image into point cloud data according to the camera intrinsic parameters; Step S2: Input the RGB-D image into the pre-trained ResNet-34 network for encoding; then input the encoded data into the PSPNet network to decode the image and obtain the feature image; input the converted point cloud data into the PointNet++ network for encoding and decoding to obtain the spliced ​​local-global point cloud features; Step S3: performing bidirectional feature fusion, wherein the bidirectional feature fusion includes a pixel-to-point and a point-to-pixel bidirectional fusion process; The pixel-to-point fusion process specifically includes: in the pixel-to-point fusion module from image features to point cloud features, find the K coordinates of each 3D point in the XYZ map. r2p The nearest point, records the appearance feature F of the nearest point in the RGB feature map ri ; Use maximum pooling and multi-layer perceptron to integrate compressed appearance features F ri , and get the integrated appearance feature F r2p ; The point cloud feature F point and integrated appearance features F r2p Connect and obtain fused point features through MLPS The point-to-pixel fusion process specifically includes: in the pixel-to-point fusion module from point cloud features to image features, find the nearest point of each feature pixel with XYZ coordinates, and record the geometric features F of the nearest point in the point cloud feature map. pj ; Use maximum pooling and multi-layer perceptron to integrate compressed geometric features F pj , get the integrated point feature F p2r , the image feature F rgb And the integrated point feature F p2r Connect to get the fused image features Step S4: Perform pixel-dense fusion to associate the geometric information of each point with the projected pixel obtained using the camera intrinsic parameters; connect the point features of the corresponding pixels and image features Get feature F fr ; The connected feature F fr The input is integrated and compressed into a multi-layer perceptron, and the integrated features are input into the symmetric reduction function to obtain a global feature vector F of a fixed size. g ; Splicing feature F fr and the global eigenvector F g Get pixel-by-pixel feature F frg ; The pixel-by-pixel feature F frg The input is sent to the final network to predict the estimated pose of each pixel. Through training and learning, the value of the loss function is continuously reduced, and finally the one with the smallest loss function value is selected as the predicted pose of each pixel. Step S5: The predicted poses of all pixels are input into the final multi-layer perceptron, the confidence is averaged and maximum pooled, and the 6D pose of the estimated object is predicted by regression. Through training and learning, the value of the overall loss function of the estimated object is continuously reduced, and finally the output with the smallest overall loss function value is selected as the overall estimated pose of the estimated object.

2. The 6D pose estimation method based on the full-flow bidirectional fusion network according to claim 1 is characterized in that: In step S3, the maximum pooling and multi-layer perceptron are used to integrate the compressed appearance features F ri , and get the integrated appearance feature F r2p , which is calculated by the following formula: Among them, F ri is the i-th nearest pixel of the RGB texture feature.

3. The 6D pose estimation method based on the full-flow bidirectional fusion network according to claim 2 is characterized in that: In step S3, the point cloud feature F point and integrated appearance features F r2p Connect and obtain fused point features through MLPS It is calculated by the following formula: in, It is a connection operation.

4. The 6D pose estimation method based on the full-flow bidirectional fusion network according to claim 1 is characterized in that: In step S3, the maximum pooling and multi-layer perceptron are used to integrate the compressed geometric features F pj , get the integrated point feature F p2r , which is calculated by the following formula: Among them, F pj represents the jth nearest point feature.

5. The 6D pose estimation method based on the full-flow bidirectional fusion network according to claim 2 is characterized in that: In step S3, the image feature F rgb And the integrated point feature F p2r Connect to get the fused image features It is calculated by the following formula: in, It is a connection operation.

6. The 6D pose estimation method based on the full-flow bidirectional fusion network according to claim 3 or 5, characterized in that: In step S3, when performing bidirectional feature fusion, the extracted RGB image features and point cloud features are effectively fused, and the pixel-level features are associated with the point-level features through the intrinsic parameter matrix; the XYZ image size is adjusted using the nearest neighbor interpolation algorithm to ensure the correspondence between the feature pixels and the 3D coordinates.

7. The 6D pose estimation method based on a full-flow bidirectional fusion network according to claim 1, characterized in that: In step S4, feature F fr It is expressed as: Among them, cat(·) represents the channel concatenation operation; Global eigenvector F g It is expressed as: F g =AP(MLP(F fr )) Where AP(·) represents the average pooling operation; Pixel-by-pixel feature F frg It is expressed as: F frg =cat(F fr ,F g )。 8. The 6D pose estimation method based on a full-flow bidirectional fusion network according to claim 1, characterized in that: In step S4, when the estimated object is an asymmetric object, the loss of posture estimation is defined as the distance between the point sampled from the object model under the actual posture and the corresponding point on the same model converted by the predicted posture, and the specific formula is: Among them, L i is the distance, x j represents the jth point among M 3D points randomly selected from the 3D model of the object, [R g |T g ] is the real posture, is the predicted pose generated from the fusion of the i-th dense pixels.

9. The 6D pose estimation method based on a full-flow bidirectional fusion network according to claim 1 or 8, characterized in that: In step S4, when the estimated object is a symmetrical object, the loss of posture estimation is defined as minimizing the distance between each point in the estimated model direction and the nearest point on the real model. The specific formula is: Among them, L i is the distance, x j represents the jth point among M 3D points randomly selected from the 3D model of the object, [R g |T g ] is the real posture, is the predicted pose generated from the fusion of the i-th dense pixels.

10. The 6D pose estimation method based on full-flow bidirectional fusion network according to claim 1, characterized in that: In step S5, the estimated object overall loss function is formed by combining the estimated object's pose estimation loss function with the confidence score, and the calculation formula of the estimated object overall loss function L is: Among them, ci is the confidence of the predicted pose of the i-th sampling pixel point, ω is the weight of the confidence, which is a balanced hyperparameter, and N is the number of randomly extracted dense pixel features.

Images