Six-degree-of-freedom object pose estimation method and system based on point cloud deformation and optimization
Patent Information
- Application Number
- CN202410069396.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-18
- Publication Date
- 2026-08-18
- Estimated Expiration
- 2044-01-18
AI Technical Summary
[0003]在实际场景中，虽然物体处于相同类别，但是它们之间的尺度大小与外形差异较大，这给实际场景下的物体位姿估计带来了巨大的挑战
[0025]This invention provides a six-DOF object pose estimation method and system based on point cloud deformation and optimization. The invention uses RGB images and point cloud data (object instance point cloud and prior point cloud of known categories) to estimate the pose of an unknown object. It performs deformation and feature fusion of the instance point cloud and prior point cloud twice through the application of a feature deformation network, and further iteratively optimizes the prior point cloud by deforming and refining its shape to match the shape of the target object. Based on the similarity transformation between the deformed prior point cloud and the instance point cloud, the six-DOF pose of the object is calculated. Compared with existing technologies, the proposed solution for 6D object pose estimation has higher accuracy.
Smart Images
Figure CN117911496B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a category-level six-DOF object pose estimation method and system based on multi-point cloud deformation and optimization. Background Technology
[0002] Six-DOF object pose estimation is widely used in fields such as robot grasping, autonomous driving, and augmented reality. Object pose estimation requires predicting the rotational and translational components of an object in the camera coordinate system. Traditional pose estimation mainly focuses on instance-level methods, but these methods require a prior 3D model of the object. When facing unknown scenes, obtaining a 3D model is difficult, resulting in poor generalization in real-world scenarios. To address this issue, researchers have proposed category-level object pose estimation methods, which can predict the poses of unknown objects of the same category. These methods do not rely on a precise 3D model of the object and exhibit stronger generalization and reliability in practical applications.
[0003] In real-world scenarios, although objects may belong to the same category, their scale and shape can vary significantly, posing a considerable challenge to object pose estimation. While researchers have proposed various category-level object pose estimation methods, there is still considerable room for improvement in pose estimation accuracy, requiring further investigation. Summary of the Invention
[0004] The purpose of this invention is to provide a method and system for estimating the pose of a six-degree-of-freedom object based on point cloud deformation and optimization, which can greatly improve the accuracy of six-degree-of-freedom pose prediction.
[0005] To achieve the above objectives, the present invention provides the following solution:
[0006] In a first aspect, this application provides a six-degree-of-freedom object pose estimation method based on point cloud deformation and optimization, characterized in that it includes:
[0007] Acquire the scene RGB image and scene depth image of the target object;
[0008] The scene RGB image is input into an instance segmentation network to obtain the category and segmentation mask of the target object; the prior point cloud of known category objects is determined based on the category of the target object; the known category objects refer to objects that are consistent with the category of the target object;
[0009] The scene RGB image is cropped according to the segmentation mask of the target object to obtain the RGB image block of the target object; the instance point cloud of the target object is determined according to the depth value at the corresponding position in the scene depth image based on the segmentation mask of the target object.
[0010] Feature extraction is performed on the RGB image patch, the instance point cloud, and the prior point cloud of the target object to obtain the RGB image features, instance point cloud features, and prior point cloud features of the target object.
[0011] The RGB image features and the instance point cloud features are input into a feature deformation network to obtain deformed RGB image map features and deformed instance point cloud map features; local instance features are obtained based on the deformed RGB image map features and the deformed instance point cloud map features; and the first global instance features are obtained based on the local instance features.
[0012] The first global instance feature and the prior point cloud feature are input into the feature deformation network to obtain a second global instance feature and local category features with graph structure information; the global category feature is obtained based on the local category features; the feature deformation network includes a first self-attention module, a second self-attention module, a third self-attention module, a fourth self-attention module, a cross-attention module, a first transformer-based graph module, and a second transformer-based graph module; the input of the first self-attention module is the RGB image feature or the first global instance feature; the input of the second self-attention module is the instance point cloud feature or the prior point cloud feature; the input of the third self-attention module is the output of the first self-attention module; the input of the fourth self-attention module is the output of the second self-attention module; the input of the cross-attention module is the output of the third self-attention module and the output of the fourth self-attention module; the two inputs of the cross-attention module are the inputs of the first transformer-based graph module and the second transformer-based graph module, respectively; the output of the first transformer-based graph module is the deformed RGB image graph feature or the second global instance feature; the output of the second transformer-based graph module is the deformed instance point cloud graph feature or the local category feature;
[0013] The prior point cloud is iteratively optimized and deformed based on the first global instance feature, the second global instance feature, and the global category feature to obtain the NOCS model of the target object;
[0014] Point cloud matching calculations are performed based on the NOCS model of the target object and the instance point cloud to obtain the six-degree-of-freedom pose and scale of the target object.
[0015] Secondly, this application provides a six-DOF object pose estimation system based on point cloud deformation and optimization, comprising:
[0016] The image acquisition subsystem is used to acquire the scene RGB image and scene depth image of the target object;
[0017] The prior point cloud acquisition subsystem is used to input the scene RGB image into the instance segmentation network to obtain the category and segmentation mask of the target object; and to determine the prior point cloud of known category objects based on the category of the target object; the known category objects refer to objects with the same category as the target object.
[0018] The instance point cloud acquisition subsystem is used to crop the scene RGB image and the scene depth image according to the segmentation mask of the target object to obtain RGB image blocks and depth image blocks of the target object; and to determine the instance point cloud of the target object according to the depth value of the depth image block.
[0019] The feature extraction subsystem is used to extract features from the RGB image blocks, instance point clouds, and prior point clouds of the target object to obtain the RGB image features, instance point cloud features, and prior point cloud features of the target object.
[0020] A global instance feature determination subsystem is used to input the RGB image features and the instance point cloud features into a feature deformation network to obtain deformed RGB image map features and deformed instance point cloud map features; obtain local instance features based on the deformed RGB image map features and deformed instance point cloud map features; and obtain the first global instance feature based on the local instance features.
[0021] A global category feature determination subsystem is used to input the first global instance feature and the prior point cloud feature into the feature deformation network to obtain a second global instance feature and local category features with graph structure information; and to obtain the global category feature based on the local category feature; the feature deformation network includes a first self-attention module, a second self-attention module, a third self-attention module, a fourth self-attention module, a cross-attention module, a first transformer-based graph module, and a second transformer-based graph module; the input of the first self-attention module is the RGB image feature or the first global instance feature; the input of the second self-attention module is the instance point cloud feature or the prior point cloud feature; The input to the third self-attention module is the output of the first self-attention module; the input to the fourth self-attention module is the output of the second self-attention module; the input to the cross-attention module is the output of the third self-attention module and the output of the fourth self-attention module; the two inputs to the cross-attention module are the inputs of the first transformer-based graph module and the second transformer-based graph module, respectively; the output of the first transformer-based graph module is the deformed RGB image graph feature or the second global instance feature; the output of the second transformer-based graph module is the deformed instance point cloud graph feature or the local category feature;
[0022] The iterative optimization subsystem is used to iteratively optimize and deform the prior point cloud based on the first global instance features, the second global instance features, and the global category features to obtain the NOCS model of the target object.
[0023] The pose calculation subsystem is used to perform point cloud matching calculations based on the NOCS model of the target object and the instance point cloud to obtain the six-degree-of-freedom pose and scale of the target object.
[0024] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0025] This invention provides a six-DOF object pose estimation method and system based on point cloud deformation and optimization. The invention uses RGB images and point cloud data (object instance point cloud and prior point cloud of known categories) to estimate the pose of an unknown object. It performs deformation and feature fusion of the instance point cloud and prior point cloud twice through the application of a feature deformation network, and further iteratively optimizes the prior point cloud by deforming and refining its shape to match the shape of the target object. Based on the similarity transformation between the deformed prior point cloud and the instance point cloud, the six-DOF pose of the object is calculated. Compared with existing technologies, the proposed solution for 6D object pose estimation has higher accuracy. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0027] Figure 1 The flowchart of a six-degree-of-freedom object pose estimation method based on point cloud deformation and optimization is provided in Embodiment 1 of the present invention.
[0028] Figure 2 This is a conceptual diagram of a six-DOF object pose estimation method based on point cloud deformation and optimization provided in Embodiment 1 of the present invention.
[0029] Figure 3 This is a schematic diagram of the object segmentation process provided in Embodiment 1 of the present invention;
[0030] Figure 4 This is a schematic diagram of the self-attention module and cross-attention module structure provided in Embodiment 1 of the present invention;
[0031] Figure 5 This is a schematic diagram of the structure of the graph module provided in Embodiment 1 of the present invention;
[0032] Figure 6 This is a schematic diagram of the prior point cloud cyclic optimization process provided in Embodiment 1 of the present invention;
[0033] Figure 7 This is a schematic diagram of the object pose estimation result provided in Embodiment 1 of the present invention;
[0034] Figure 8 This is a block diagram of a six-degree-of-freedom object pose estimation system based on point cloud deformation and optimization, provided in Embodiment 2 of the present invention. Detailed Implementation
[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0036] The purpose of this invention is to provide a method and system for estimating the pose of a six-degree-of-freedom object based on point cloud deformation and optimization, which can greatly improve the accuracy of six-degree-of-freedom pose prediction.
[0037] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0038] Example 1
[0039] like Figure 1 and 2 As shown, this embodiment provides a six-DOF object pose estimation method based on point cloud deformation and optimization, including:
[0040] S1: Obtain the scene RGB image and scene depth image of the target object.
[0041] A depth camera is used to acquire information about objects in a scene, including RGB images and depth images of the objects. Any camera capable of acquiring both RGB image and depth information can be used.
[0042] S2: Input the scene RGB image into the instance segmentation network to obtain the category of the target object and the segmentation mask; determine the prior point cloud of known category objects based on the category of the target object; the known category objects refer to objects that are consistent with the category of the target object.
[0043] As an optional implementation, the instance segmentation network can be a Mask-RCNN network, or any existing instance segmentation method. Figure 3 The object segmentation process is shown.
[0044] S3: Crop the scene RGB image according to the segmentation mask of the target object to obtain the RGB image block of the target object; determine the instance point cloud of the target object according to the depth value at the corresponding position in the scene depth image based on the segmentation mask of the target object.
[0045] The instance point cloud of the target object is determined based on the depth value of the depth image patch and the camera intrinsic parameter matrix.
[0046] S4: Perform feature extraction on the RGB image patch, the instance point cloud, and the prior point cloud of the target object to obtain the RGB image features F of the target object. rgb Example point cloud features F o and prior point cloud features F r .
[0047] The neural network CNN used to extract RGB image features is PSPNet based on ResNet-18, and the neural network used to extract the feature of the solid instance point cloud and prior point cloud is PointNet++.
[0048] S5: Transfer the RGB image features F rgb and the instance point cloud features F o The input is fed into a feature deformation network to obtain deformed RGB image map features and deformed instance point cloud map features; based on the deformed RGB image map features and the deformed instance point cloud map features, local instance features are obtained. Based on the local instance features Obtain the first global instance features
[0049] The feature deformation network includes a first self-attention module SA1, a second self-attention module SA2, a third self-attention module SA3, a fourth self-attention module SA4, a cross-attention module CA, a first transformer-based graph module GM1, and a second transformer-based graph module GM2. The input of the first self-attention module is the RGB image features; the input of the second self-attention module is the instance point cloud features; the input of the third self-attention module is the output of the first self-attention module; the input of the fourth self-attention module is the output of the second self-attention module; the input of the cross-attention module is the output of the third self-attention module and the output of the fourth self-attention module; the two inputs of the cross-attention module are the inputs of the first transformer-based graph module and the second transformer-based graph module, respectively; the output of the first transformer-based graph module is the deformed RGB image graph features; and the output of the second transformer-based graph module is the deformed instance point cloud graph features.
[0050] like Figure 4 and Figure 5The structure of the feature deformation network is shown, including an encoding part, a decoding part, and a graph structure information extraction part. The encoding part includes a first self-attention module and a second self-attention module; the decoding part includes a third self-attention module, a fourth self-attention module, and a cross-attention module; and the graph structure information extraction part includes a first transformer-based graph module and a second transformer-based graph module. The encoding uses a transformer to perform feature extraction, and the decoding part uses a transformer to predict the deformable features of RGB features and point cloud features, realizing feature fusion and adjustment of point cloud features.
[0051] Figure 4 The diagram illustrates the specific attention mechanisms of each transformer-based self-attention module and the specific cross-attention methods of the transformer-based cross-attention module. Specifically, the transformer-based self-attention module is used to perform feature adjustment and optimization, extracting global features; the transformer-based cross-attention module is used to perform multi-source feature fusion, adjusting and optimizing the fused features; and the transformer-based graph module is used to find the intrinsic relationships between multi-source features, constructing graph features among multi-source features, and deepening the network's understanding and use of multi-source features.
[0052] Step S5, "inputting the RGB image features and the instance point cloud features into the feature deformation network to obtain deformed RGB image map features and deformed instance point cloud map features," specifically includes:
[0053] (1) Input the RGB image features into the first self-attention module to obtain encoded RGB image features, and input the encoded RGB image features into the third self-attention module to obtain decoded RGB image features.
[0054] (2) Input the instance point cloud features into the second self-attention module to obtain the encoded instance point cloud features, and input the encoded instance point cloud features into the fourth self-attention module to obtain the decoded instance point cloud features.
[0055] (3) Input the decoded RGB image features and the decoded instance point cloud features into the cross attention module to obtain cross RGB image features and cross instance point cloud features; and obtain deformed RGB image features based on the cross RGB image features and the RGB image features; and obtain deformed instance point cloud features based on the cross instance point cloud features and the instance point cloud features.
[0056] (4) Input the deformed RGB image features into the first transformer-based graph module to obtain the deformed RGB image graph features; input the deformed instance point cloud features into the second transformer-based graph module to obtain the deformed instance point cloud graph features.
[0057] Among them, such as Figure 5 As shown, the deformed RGB image features are input into the first transformer-based graph module to obtain the deformed RGB image graph features, specifically including:
[0058] (4-1) Obtain the matrix q, matrix k, and matrix v in the attention mechanism based on the deformed RGB image features;
[0059] As an example, one could choose to apply the features of the deformed RGB image to a multilayer perceptron (MLP) to generate the three parameters q, k, v of the transformer.
[0060] (4-2) Perform a softmax operation on the matrix k and combine it with the matrix v to obtain the attention matrix.
[0061] (4-3) Adjust the matrix q using the attention matrix to obtain the adjustment matrix.
[0062] (4-4) Randomly determine the adjacency matrix of the graph structure, and multiply the adjacency matrix with the matrix v to obtain the graph features of matrix v.
[0063] (4-5) Add the graph features of matrix v to matrix q to obtain a first addition result (self-connection features of the graph); multiply the first addition result by matrix q to obtain a multiplication result (adjustment of graph features); multiply the multiplication result by the adjustment matrix. Add them together to get the second result.
[0064] (4-6) The deformed RGB image features are obtained by adding the second addition result to the deformed RGB image features.
[0065] When generating local instance features and global instance features, the deformed RGB image map features and the deformed instance point cloud map features are concatenated, and the local instance features are obtained through an MLP layer. Then, for local instance features Perform average pooling (AVE) to generate the first global instance feature.
[0066] In this step, RGB image features and instance point cloud features are input into the feature deformation network for coarse deformation operation. The transformer is used to adjust and fuse the feature attention of RGB image features and instance point cloud features, and predict the deformation features of instance point cloud to complete coarse deformation and point cloud completion of instance point cloud. Based on RGB image features and instance point cloud features, local instance features and then global instance features are obtained.
[0067] S6: Transfer the first global instance feature and the prior point cloud features F r The input is fed into the feature deformation network to obtain a second global instance feature with graph structure information. and local category features Based on the local category features Obtain global category features
[0068] When the first global instance feature and the prior point cloud feature are input into the feature deformation network, the input of the first self-attention module is the first global instance feature; the input of the second self-attention module is the prior point cloud feature; the output of the first transformer-based graph module is the second global instance feature; and the output of the second transformer-based graph module is the local category feature.
[0069] In this step, the first global instance features and category point cloud features (prior point cloud features) are input into the feature deformation network for fine deformation operations. The instance features and category features are fused through the transformer to predict the deformation features of the prior point cloud, thereby generating local category features and global category features.
[0070] S7: Based on the first global instance feature Second global instance feature and the global category features For the prior point cloud p r The NOCS model of the target object is obtained by performing iterative optimization deformation.
[0071] Based on global instance features and global category features, the prior point cloud after fine deformation is further deformed and adjusted, and an iterative optimization process is performed to finally generate the NOCS model of the target object. The idea of iterative optimization is as follows: predict the deformation offset of the prior point cloud based on global category features; further adjust and optimize the shape of the prior point cloud based on the deformation offset; predict the transformation matrix based on global category features and global instance features; finally, obtain the NOCS model of the object based on the final optimized prior point cloud, deformation offset, and transformation matrix.
[0072] Specifically, such as Figure 6 As shown, step S7 includes:
[0073] (a) The first global instance feature and the second global instance feature By concatenating the features, we obtain the third global instance feature.
[0074] (b) Let the global category feature be the initial value of the global category feature in the iterative optimization process, that is, the global category feature in the 0th iteration. Let the prior point cloud be the initial value of the prior point cloud in the iterative optimization process, that is, the prior point cloud in the 0th iteration. The initial value of the deformation offset of the prior point cloud is determined based on the initial value of the global category feature, i.e., the deformation offset in the 0th iteration. Based on the third global instance feature Determine the initial value of the transformation matrix of the prior point cloud, i.e., the transformation matrix of the 0th iteration.
[0075] (c) Determine whether the current iteration number i is less than the iteration number threshold N; i = 1, 2, 3, ...
[0076] (d) If not, output the prior point cloud for the (i-1)th iteration.
[0077] (e) If so, then the deformation offset of the (i-1)th iteration is used. Update the prior point cloud for the (i-1)th iteration. The prior point cloud of the i-th iteration is obtained.
[0078] (f) Obtain the local category features of the i-th iteration using the prior point cloud of the i-th iteration; obtain the global category features of the i-th iteration based on the local category features of the i-th iteration.
[0079] (g) Determine the transformation matrix of the i-th iteration based on the third global instance feature and the global category feature of the i-th iteration; determine the deformation offset of the i-th iteration based on the global category feature of the i-th iteration and the global category feature of the 0th iteration.
[0080] One possible implementation for determining the transformation matrix and the deformation offset of the i-th iteration is as follows:
[0081] The intermediate transformation matrix for the i-th iteration is determined based on the third global instance feature and the global category feature of the i-th iteration. The transformation matrix for the i-th iteration is obtained based on the intermediate transformation matrix of the i-th iteration and the transformation matrix of the (i-1)-th iteration. Correspondingly... Figure 6 In
[0082] The intermediate deformation offset for the i-th iteration is determined based on the global category features of the i-th iteration and the global category features of the 0-th iteration. The deformation offset for the i-th iteration is determined based on the intermediate deformation offset of the i-th iteration and the deformation offset of the (i-1)-th iteration. Correspondingly... Figure 6 In
[0083] (h) Let i = i + 1, and return to step (c): "Determine whether the current iteration number i is less than the maximum iteration number".
[0084] (i) Based on the transformation matrix of the last iteration Deformation offset of the last iteration and the prior point cloud of the last iteration Determine the NOCS model of the target object.
[0085] The expression for the NOCS model of the target object is as follows:
[0086]
[0087] Here, softmax() represents the activation function; This represents the transformation matrix of the last iteration; This represents the deformation offset from the last iteration; This represents the prior point cloud of the last iteration.
[0088] S8: Perform point cloud matching calculations based on the NOCS model of the target object and the instance point cloud to obtain the six-degree-of-freedom pose and scale of the target object.
[0089] Specifically, step S8 includes:
[0090] Based on the NOCS model of the target object and the instance point cloud, a similarity transformation is performed to obtain the six-degree-of-freedom pose and scale of the target object.
[0091] The expression for the similarity transformation is: T = f(P) o ,P NOCS ).
[0092] Among them, P oLet T represent the instance point cloud of the target object, T represent the six-DOF pose of the target object, including the three-DOF rotational components and the three-DOF translational components of the target object, and f represent the Umeyama algorithm used in the similarity transformation.
[0093] Figure 7 The visualization results of the object pose prediction are shown. Figure 7 All objects are unknown beforehand, and their poses are represented using 3D bounding boxes. Figure 7 The diagram contains two 3D bounding boxes: a dashed bounding box representing the ground truth and a solid bounding box representing the predicted ground truth. If the predicted 3D bounding box closely matches the ground truth, the object pose estimation can be considered relatively accurate. It is evident that the predicted pose matches the ground truth well.
[0094] In this embodiment, the pose estimation of an unknown object is achieved using RGB images and point clouds. Coarse and fine deformation operations are used to deform and fuse the instance point cloud and prior point cloud, respectively. Furthermore, a cyclic optimization process is employed to further deform and refine the shape of the prior point cloud, ensuring it matches the shape of the target object. A similarity transformation is performed between the deformed prior point cloud and the instance point cloud to calculate the object's six-degree-of-freedom pose. Compared to existing technologies, the proposed solution achieves higher accuracy in estimating the 6D pose of the object.
[0095] Example 2
[0096] like Figure 8 As shown, this embodiment provides a six-DOF object pose estimation system based on point cloud deformation and optimization, including:
[0097] The image acquisition subsystem 100 is used to acquire the scene RGB image and scene depth image of the target object.
[0098] The prior point cloud acquisition subsystem 200 is used to input the scene RGB image into the instance segmentation network to obtain the category and segmentation mask of the target object; and to determine the prior point cloud of known category objects based on the category of the target object; the known category objects refer to objects that are consistent with the category of the target object.
[0099] The instance point cloud acquisition subsystem 300 is used to crop the scene RGB image according to the segmentation mask of the target object to obtain the RGB image block of the target object; and to determine the instance point cloud of the target object according to the depth value at the corresponding position in the scene depth image based on the segmentation mask of the target object.
[0100] The feature extraction subsystem 400 is used to extract features from the RGB image blocks, instance point clouds, and prior point clouds of the target object to obtain the RGB image features, instance point cloud features, and prior point cloud features of the target object.
[0101] A global instance feature determination subsystem 500 is used to input the RGB image features and the instance point cloud features into a feature deformation network to obtain deformed RGB image map features and deformed instance point cloud map features; obtain local instance features based on the deformed RGB image map features and the deformed instance point cloud map features; and obtain the first global instance feature based on the local instance features.
[0102] A global category feature determination subsystem 600 is used to input the first global instance feature and the prior point cloud feature into the feature deformation network to obtain a second global instance feature and local category features with graph structure information; and to obtain the global category feature based on the local category feature; the feature deformation network includes a first self-attention module, a second self-attention module, a third self-attention module, a fourth self-attention module, a cross-attention module, a first transformer-based graph module, and a second transformer-based graph module; the input of the first self-attention module is the RGB image feature or the first global instance feature; the input of the second self-attention module is the instance point cloud feature or the prior point cloud feature; The input to the third self-attention module is the output of the first self-attention module; the input to the fourth self-attention module is the output of the second self-attention module; the input to the cross-attention module is the output of the third self-attention module and the output of the fourth self-attention module; the two inputs to the cross-attention module are the inputs of the first transformer-based graph module and the second transformer-based graph module, respectively; the output of the first transformer-based graph module is the deformed RGB image graph feature or the second global instance feature; the output of the second transformer-based graph module is the deformed instance point cloud graph feature or the local category feature.
[0103] The iterative optimization subsystem 700 is used to iteratively optimize and deform the prior point cloud based on the first global instance features, the second global instance features, and the global category features to obtain the NOCS model of the target object.
[0104] The pose calculation subsystem 800 is used to perform point cloud matching calculations based on the NOCS model of the target object and the instance point cloud to obtain the six-degree-of-freedom pose and scale of the target object.
[0105] Example 3
[0106] This embodiment provides an electronic device, including a memory and a processor. The memory is used to store computer programs, and the processor runs the computer programs to enable the electronic device to execute a six-degree-of-freedom object pose estimation method based on point cloud deformation and optimization according to Embodiment 1.
[0107] Alternatively, the aforementioned electronic device may be a server.
[0108] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements a six-degree-of-freedom object pose estimation method based on point cloud deformation and optimization according to Embodiment 1.
[0109] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0110] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0113] Each embodiment focuses on describing the differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.
[0114] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A six-DOF object pose estimation method based on point cloud deformation and optimization, characterized in that, include: Acquire the scene RGB image and scene depth image of the target object; The scene RGB image is input into an instance segmentation network to obtain the category and segmentation mask of the target object; Based on the category of the target object, determine the prior point cloud of known category objects; the known category objects refer to objects that are of the same category as the target object; The scene RGB image is cropped according to the segmentation mask of the target object to obtain the RGB image block of the target object; The instance point cloud of the target object is determined based on the depth value at the corresponding position in the scene depth image according to the segmentation mask of the target object; Feature extraction is performed on the RGB image patch, the instance point cloud, and the prior point cloud of the target object to obtain the RGB image features, instance point cloud features, and prior point cloud features of the target object. The RGB image features and the instance point cloud features are input into the feature deformation network to obtain deformed RGB image map features and deformed instance point cloud map features; Based on the deformed RGB image features and the deformed instance point cloud features, local instance features are obtained; based on the local instance features, first global instance features are obtained. The first global instance feature and the prior point cloud feature are input into the feature deformation network to obtain a second global instance feature and local category features with graph structure information; the global category feature is obtained based on the local category features; the feature deformation network includes a first self-attention module, a second self-attention module, a third self-attention module, a fourth self-attention module, a cross-attention module, a first transformer-based graph module, and a second transformer-based graph module; the input of the first self-attention module is the RGB image feature or the first global instance feature; The input to the second self-attention module is the instance point cloud feature or the prior point cloud feature; the input to the third self-attention module is the output of the first self-attention module; the input to the fourth self-attention module is the output of the second self-attention module; the input to the cross-attention module is the output of the third self-attention module and the output of the fourth self-attention module; the two outputs of the cross-attention module are the inputs of the first transformer-based graph module and the second transformer-based graph module, respectively; the output of the first transformer-based graph module is the deformed RGB image graph feature or the second global instance feature; The output of the second transformer-based graph module is either the point cloud feature of the deformed instance or the local category feature; The prior point cloud is iteratively optimized and deformed based on the first global instance feature, the second global instance feature, and the global category feature to obtain the NOCS model of the target object; Point cloud matching calculations are performed based on the NOCS model of the target object and the instance point cloud to obtain the six-degree-of-freedom pose and scale of the target object.
2. The six-degree-of-freedom object pose estimation method based on point cloud deformation and optimization according to claim 1, characterized in that, The RGB image features and the instance point cloud features are input into a feature deformation network to obtain deformed RGB image map features and deformed instance point cloud map features, specifically including: The RGB image features are input into the first self-attention module to obtain encoded RGB image features, and the encoded RGB image features are input into the third self-attention module to obtain decoded RGB image features; The instance point cloud features are input into the second self-attention module to obtain encoded instance point cloud features, and the encoded instance point cloud features are input into the fourth self-attention module to obtain decoded instance point cloud features; The decoded RGB image features and the decoded instance point cloud features are input into the cross-attention module to obtain cross-RGB image features and cross-instance point cloud features; deformed RGB image features are obtained based on the cross-RGB image features and the RGB image features; deformed instance point cloud features are obtained based on the cross-instance point cloud features and the instance point cloud features. The deformed RGB image features are input into the first transformer-based graph module to obtain the deformed RGB image graph features; the deformed instance point cloud features are input into the second transformer-based graph module to obtain the deformed instance point cloud graph features.
3. The six-degree-of-freedom object pose estimation method based on point cloud deformation and optimization according to claim 2, characterized in that, The deformed RGB image features are input into the first transformer-based graph module to obtain the deformed RGB image graph features, specifically including: The matrix in the attention mechanism is obtained based on the deformed RGB image features. q ,matrix k sum matrix v ; For the matrix k implement soft The max operation is combined with the matrix. v Obtain the attention matrix; Using the attention matrix to the matrix q Adjustments are made to obtain the adjustment matrix. ; Randomly determine the adjacency matrix of the graph structure, and then combine the adjacency matrix with the matrix. v Multiply to obtain a matrix v Graph features; The matrix v The graph features and the matrix q Perform the addition to obtain a first addition result; and then add the first addition result to the matrix. q Multiply to obtain the multiplication result; then multiply the result with the adjustment matrix. Add them together to get the second result. The deformed RGB image features are obtained by adding the second addition result to the deformed RGB image features.
4. The six-degree-of-freedom object pose estimation method based on point cloud deformation and optimization according to claim 1, characterized in that, The prior point cloud is iteratively optimized and deformed based on the first global instance features, the second global instance features, and the global category features to obtain the NOCS model of the target object, specifically including: The first global instance feature and the second global instance feature are concatenated to obtain the third global instance feature; Let the global category feature be the initial value of the global category feature in the iterative optimization process, i.e., the global category feature of the 0th iteration; let the prior point cloud be the initial value of the prior point cloud in the iterative optimization process, i.e., the prior point cloud of the 0th iteration; determine the initial value of the deformation offset of the prior point cloud based on the initial value of the global category feature, i.e., the deformation offset of the 0th iteration; determine the initial value of the transformation matrix of the prior point cloud based on the third global instance feature, i.e., the transformation matrix of the 0th iteration. Determine the current iteration number i Is it less than the iteration count threshold? i =1, 2, 3, ...; If not, then output the first... i Prior point cloud of -1 iteration; If so, then use the first i- The deformation offset is updated in the first iteration. i The prior point cloud of the -1st iteration yields the... i The prior point cloud of the next iteration; Using the first i The prior point cloud of the second iteration is obtained. i Local category features of the nth iteration; based on the nth iteration i The local class features of the iteration are obtained at the 1st iteration. i Global category features in the next iteration; Based on the third global instance feature and the first i The global category features of the iteration determine the first i The transformation matrix of the nth iteration; according to the nth iteration i The global category features of the 0th iteration and the global category features of the 1st iteration determine the 2nd iteration. i The deformation offset of the next iteration; make i = i +1, and return to the step "Determine the current iteration number". i Is it less than the maximum number of iterations? The NOCS model of the target object is determined based on the transformation matrix of the last iteration, the deformation offset of the last iteration, and the prior point cloud of the last iteration.
5. The six-degree-of-freedom object pose estimation method based on point cloud deformation and optimization according to claim 4, characterized in that, Based on the third global instance feature and the first i The global category features of the iteration determine the first i The transformation matrix of the nth iteration; according to the nth iteration i The global category features of the 0th iteration and the global category features of the 1st iteration determine the 2nd iteration. i The deformation offset of the next iteration specifically includes: Based on the third global instance feature and the first i The global category features of the iteration determine the first i The intermediate transformation matrix of the next iteration; According to the first i The intermediate transformation matrix of the nth iteration and the nth iteration i The transformation matrix of the -1st iteration yields the first... i The transformation matrix of the next iteration; According to the first i The global category features of the 0th iteration and the global category features of the 1st iteration determine the 2nd iteration. i The intermediate deformation offset of the next iteration; According to the first i The intermediate deformation offset of the second iteration and the first i The deformation offset of the -1st iteration determines the first... i The deformation offset of the next iteration.
6. The six-degree-of-freedom object pose estimation method based on point cloud deformation and optimization according to claim 4, characterized in that, The expression for the NOCS model of the target object is: in, soft max() represents the activation function; This represents the transformation matrix of the last iteration; This represents the deformation offset from the last iteration; This represents the prior point cloud of the last iteration.
7. The six-degree-of-freedom object pose estimation method based on point cloud deformation and optimization according to claim 6, characterized in that, Point cloud matching calculations are performed based on the NOCS model of the target object and the instance point cloud to obtain the six-DOF pose and scale of the target object, specifically including: Based on the NOCS model of the target object and the instance point cloud, a similarity transformation is performed to obtain the six-DOF pose and scale of the target object; the expression for the similarity transformation is: ; An instance point cloud representing the target object. This represents the six-degree-of-freedom pose of the target object, including its three-degree-of-freedom rotational and three-degree-of-freedom translational components. This indicates the Umeyama algorithm used in the similarity transformation.
8. A six-DOF object pose estimation system based on point cloud deformation and optimization, characterized in that, include: The image acquisition subsystem is used to acquire the scene RGB image and scene depth image of the target object; The prior point cloud acquisition subsystem is used to input the scene RGB image into the instance segmentation network to obtain the category and segmentation mask of the target object; and to determine the prior point cloud of known category objects based on the category of the target object; the known category objects refer to objects with the same category as the target object. The instance point cloud acquisition subsystem is used to crop the scene RGB image according to the segmentation mask of the target object to obtain the RGB image block of the target object; The instance point cloud of the target object is determined based on the depth value at the corresponding position in the scene depth image according to the segmentation mask of the target object; The feature extraction subsystem is used to extract features from the RGB image blocks, instance point clouds, and prior point clouds of the target object to obtain the RGB image features, instance point cloud features, and prior point cloud features of the target object. A global instance feature determination subsystem is used to input the RGB image features and the instance point cloud features into a feature deformation network to obtain deformed RGB image map features and deformed instance point cloud map features; Based on the deformed RGB image features and the deformed instance point cloud features, local instance features are obtained; based on the local instance features, first global instance features are obtained. A global category feature determination subsystem is used to input the first global instance feature and the prior point cloud feature into the feature deformation network to obtain a second global instance feature and local category features with graph structure information; and to obtain the global category feature based on the local category feature; the feature deformation network includes a first self-attention module, a second self-attention module, a third self-attention module, a fourth self-attention module, a cross-attention module, a first transformer-based graph module, and a second transformer-based graph module; the input of the first self-attention module is the RGB image feature or the first global instance feature; The input to the second self-attention module is the instance point cloud feature or the prior point cloud feature; the input to the third self-attention module is the output of the first self-attention module; the input to the fourth self-attention module is the output of the second self-attention module; the input to the cross-attention module is the output of the third self-attention module and the output of the fourth self-attention module; the two outputs of the cross-attention module are the inputs of the first transformer-based graph module and the second transformer-based graph module, respectively; the output of the first transformer-based graph module is the deformed RGB image graph feature or the second global instance feature; The output of the second transformer-based graph module is either the point cloud feature of the deformed instance or the local category feature; The iterative optimization subsystem is used to iteratively optimize and deform the prior point cloud based on the first global instance features, the second global instance features, and the global category features to obtain the NOCS model of the target object. The pose calculation subsystem is used to perform point cloud matching calculations based on the NOCS model of the target object and the instance point cloud to obtain the six-degree-of-freedom pose and scale of the target object.