Training method of object pose estimation model, object pose estimation method and device thereof

Abstract

The present disclosure provides a method for training an object pose estimation model, an object pose estimation method and an apparatus thereof. The method comprises: obtaining a pose training set, the pose training set comprising a plurality of target object training images and an object prior vector corresponding to each target object training image; for each target object training image, performing convolution processing on the target object training image using a first two-dimensional convolution model to obtain an object global feature; processing the object global feature and the object prior vector using a deformation network to obtain deformation point cloud information; processing the object global feature and the deformation point cloud information using a convolutional neural network to obtain a two-dimensional feature vector and a three-dimensional feature vector; performing multi-level feature matching processing on the two-dimensional feature vector and the three-dimensional feature vector to obtain a three-dimensional correspondence parameter; and iteratively adjusting network parameters of an initial pose estimation model according to the three-dimensional correspondence parameter and a three-dimensional key point heat map to obtain a trained object pose estimation model.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more specifically to a training method for an object pose estimation model, an object pose estimation method, an apparatus, a device, a medium, and a program product. Background Technology

[0002] Object pose estimation, a core task in computer vision, is crucial for various practical applications. In autonomous driving, accurate object pose estimation helps vehicles accurately perceive their surroundings, avoid obstacles, and improve driving safety and efficiency. In industrial manufacturing, accurate pose estimation is key to robot operation and automated production, improving production efficiency, reducing costs, and ensuring worker safety. Furthermore, object pose estimation technology plays a vital role in intelligent monitoring, virtual reality, and other fields, providing essential support for the implementation of various intelligent systems.

[0003] In realizing the present invention, the inventors discovered that the accuracy and reliability of the estimated data in the related technologies for pose estimation are poor. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a training method for an object pose estimation model, an object pose estimation method, an apparatus, a device, a medium, and a program product.

[0005] According to a first aspect of this disclosure, a method for training an object pose estimation model is provided, comprising:

[0006] Obtain a pose training set, wherein the pose training set includes multiple target object training images and object prior vectors corresponding to each of the target object training images;

[0007] For each of the above target object training images, the first two-dimensional convolution model is used to perform convolution processing on the above target object training images to obtain the global features of the object;

[0008] By using a deformation network to process the global features and prior vectors of the above objects, deformation point cloud information is obtained.

[0009] By using a convolutional neural network to process the global features of the above-mentioned objects and the above-mentioned deformation point cloud information, two-dimensional feature vectors and three-dimensional feature vectors are obtained.

[0010] Multi-level feature matching processing is performed on the above two-dimensional feature vector and the above three-dimensional feature vector to obtain three-dimensional correspondence parameters, wherein the above three-dimensional correspondence parameters characterize the correlation and correspondence between pixels in the above target object training image and point clouds in the above deformed point cloud information.

[0011] The network parameters of the initial pose estimation model are iteratively adjusted based on the above three-dimensional correspondence parameters and the three-dimensional key point heatmap to obtain the trained object pose estimation model. The above initial pose estimation model includes the above first two-dimensional convolutional model, the above deformation network, the above convolutional neural network and the object detection model. The above three-dimensional key point heatmap is obtained by processing the above target object training image and the above object prior vector using the above object detection model.

[0012] According to embodiments of this disclosure, deformation point cloud information is obtained by processing the global features of the object and the prior vector of the object using a deformation network, including:

[0013] The deformation vector is obtained by processing the global features of the object using the deformation network described above.

[0014] Based on the aforementioned deformation vector and the aforementioned object prior vector, the aforementioned deformation point cloud information is generated.

[0015] According to embodiments of this disclosure, the convolutional neural network includes a second two-dimensional convolutional model and a three-dimensional convolutional model.

[0016] According to embodiments of this disclosure, a convolutional neural network is used to process the global features of the object and the deformation point cloud information to obtain two-dimensional feature vectors and three-dimensional feature vectors, including:

[0017] The above-mentioned second two-dimensional convolution model is used to perform feature extraction processing on the global features of the above-mentioned object to obtain the above-mentioned two-dimensional feature vector;

[0018] The above-mentioned three-dimensional convolution model is used to perform feature extraction processing on the above-mentioned deformed point cloud information to obtain the above-mentioned three-dimensional feature vector.

[0019] According to embodiments of this disclosure, multi-level feature matching processing is performed on the above-mentioned two-dimensional feature vectors and the above-mentioned three-dimensional feature vectors to obtain three-dimensional correspondence parameters, including:

[0020] Based on the above two-dimensional feature vectors and the above three-dimensional feature vectors, a first relation matrix is ​​generated, wherein the first relation matrix represents the correspondence between each pixel and all point clouds;

[0021] For each of the above pixels, extract multiple correspondence coefficients corresponding to the above pixels from the first relation matrix;

[0022] Generate corresponding parameters based on multiple of the above-mentioned correspondence coefficients and the point cloud coordinates corresponding to each of the above-mentioned correspondence coefficients;

[0023] Based on the above-mentioned corresponding parameters, the above-mentioned three-dimensional correspondence parameters are generated.

[0024] According to embodiments of this disclosure, generating a first relation matrix based on the aforementioned two-dimensional feature vectors and the aforementioned three-dimensional feature vectors includes:

[0025] The similarity between the two-dimensional feature vectors and the three-dimensional feature vectors is calculated to obtain a similarity score.

[0026] The similarity scores are normalized to obtain the first relation matrix.

[0027] According to embodiments of this disclosure, the above-mentioned three-dimensional key point heatmap is obtained in the following manner:

[0028] The above-mentioned target detection model is used to process the training images of the target objects and the prior vectors of the objects to obtain the query matrix;

[0029] Based on the query matrix and key matrix mentioned above, the above three-dimensional key point heatmap is generated, wherein the key matrix is ​​obtained by mapping the above three-dimensional feature vectors.

[0030] According to embodiments of this disclosure, the network parameters of the initial pose estimation model are iteratively adjusted based on the aforementioned three-dimensional correspondence parameters and three-dimensional keypoint heatmaps to obtain a trained object pose estimation model, including:

[0031] Based on the above 3D key point heatmap and point cloud data, generate 3D key point coordinates;

[0032] Based on the above three-dimensional correspondence parameters and the above three-dimensional key point coordinates, generate consistency constraint values;

[0033] The network parameters of the initial pose estimation model are iteratively adjusted based on the consistency constraint values ​​to obtain the object pose estimation model.

[0034] According to embodiments of this disclosure, consistency constraint values ​​are generated based on the aforementioned three-dimensional correspondence parameters and the aforementioned three-dimensional keypoint coordinates, including:

[0035] Perform bilinear interpolation on the above three-dimensional correspondence parameters to obtain linear interpolation results;

[0036] Based on the above linear interpolation results and the above three-dimensional key point coordinates, the above consistency constraint values ​​are generated.

[0037] According to a second aspect of this disclosure, an object pose estimation method is provided, comprising:

[0038] Obtain the image to be processed containing the objects to be processed and the average shape vector of the object category to which the objects to be processed belong;

[0039] The above-mentioned image to be processed and the above-mentioned average shape vector are processed using the object pose estimation model to obtain the predicted global features, predicted point cloud information and predicted 3D correspondence parameters;

[0040] The above-mentioned predicted global features are subjected to regression processing to obtain the initial pose data;

[0041] Based on the initial pose data, the predicted point cloud information, and the predicted 3D correspondence parameters, a rendering atlas is generated, wherein the rendering atlas includes the pose rendering map, mask map, and depth map of the image to be processed.

[0042] According to a third aspect of this disclosure, a training apparatus for an object pose estimation model is provided, comprising:

[0043] The first acquisition module is used to acquire a pose training set, wherein the pose training set includes multiple target object training images and object prior vectors corresponding to each of the target object training images.

[0044] The first convolution module is used to perform convolution processing on the training images of the target objects for each of the above-mentioned target objects using the first two-dimensional convolution model to obtain the global features of the objects.

[0045] The deformation module is used to process the global features and prior vectors of the above-mentioned objects using the deformation network to obtain deformation point cloud information.

[0046] The second convolutional module is used to process the global features of the above-mentioned objects and the above-mentioned deformation point cloud information using a convolutional neural network to obtain two-dimensional feature vectors and three-dimensional feature vectors.

[0047] The feature matching module is used to perform multi-level feature matching processing on the above two-dimensional feature vector and the above three-dimensional feature vector to obtain three-dimensional correspondence parameters, wherein the above three-dimensional correspondence parameters characterize the association and correspondence relationship between pixels in the above target object training image and point clouds in the above deformed point cloud information.

[0048] The training module is used to iteratively adjust the network parameters of the initial pose estimation model based on the above-mentioned three-dimensional correspondence parameters and the three-dimensional key point heatmap to obtain the trained object pose estimation model. The above-mentioned initial pose estimation model includes the above-mentioned first two-dimensional convolutional model, the above-mentioned deformation network, the above-mentioned convolutional neural network and the object detection model. The above-mentioned three-dimensional key point heatmap is obtained by processing the above-mentioned target object training image and the above-mentioned object prior vector using the above-mentioned object detection model.

[0049] According to a fourth aspect of this disclosure, an object pose estimation apparatus is provided, comprising:

[0050] The second acquisition module is used to acquire the image to be processed containing the object to be processed and the average shape vector of the object category to which the object to be processed belongs;

[0051] The processing module is used to process the above-mentioned image to be processed and the above-mentioned average shape vector using the object pose estimation model to obtain the predicted global features, predicted point cloud information and predicted three-dimensional correspondence parameters.

[0052] The regression module is used to perform regression processing on the above-mentioned predicted global features to obtain the initial pose data;

[0053] The generation module is used to generate a rendering atlas based on the initial pose data, the predicted point cloud information, and the predicted 3D correspondence parameters. The rendering atlas includes the pose rendering map, mask map, and depth map of the image to be processed.

[0054] A fifth aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the training method or object pose estimation method of the object pose estimation model described above.

[0055] A sixth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the training method or object pose estimation method of the above-described object pose estimation model.

[0056] The seventh aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the training method or object pose estimation method of the above-described object pose estimation model.

[0057] According to embodiments of this disclosure, the object pose estimation model utilizes multimodal data of the target object training image and object prior vectors during training, thereby providing richer and more comprehensive global features and deformation point cloud information of the object. This can improve the estimation accuracy and robustness of the object pose estimation model. Furthermore, by performing multi-level feature matching processing on the two-dimensional and three-dimensional feature vectors obtained from the convolutional neural network, the limitations of single-modal data can be compensated, and the impact of data noise and uncertainty on pose estimation can be reduced, thereby effectively improving the prediction accuracy of the object pose estimation model. Attached Figure Description

[0058] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0059] Figure 1This illustration schematically shows a training method for an object pose estimation model or an application scenario of an object pose estimation method according to embodiments of the present disclosure.

[0060] Figure 2 A flowchart illustrating a method for training an object pose estimation model according to an embodiment of the present disclosure is shown schematically.

[0061] Figure 3 A flowchart illustrating a method for training an object pose estimation model according to another embodiment of the present disclosure is shown schematically.

[0062] Figure 4 A flowchart illustrating an object pose estimation method according to an embodiment of the present disclosure is shown schematically.

[0063] Figure 5 This schematic diagram illustrates a structural block diagram of a training apparatus for an object pose estimation model according to an embodiment of the present disclosure;

[0064] Figure 6 A schematic block diagram of an object pose estimation apparatus according to an embodiment of the present disclosure is shown; and

[0065] Figure 7 A block diagram of an electronic device suitable for implementing a training method or object pose estimation method for an object pose estimation model according to embodiments of the present disclosure is shown schematically. Detailed Implementation

[0066] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0067] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0068] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0069] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0070] The primary goal of 6D pose estimation is to estimate the 3D translation and rotation of an object in a coordinate system by analyzing input image, video, or point cloud data. Traditional object pose estimation methods are divided into two categories: instance-level object pose estimation and category-level object pose estimation. Generally, current instance-level object pose estimation often relies on geometric information. These methods utilize points, edges, or other types of geometric representations constrained by projection under a perspective camera. Then, the pose is solved through optimization. A large amount of work uses correspondence to represent geometric relationships, which can be divided into sparse keypoints and dense correspondences. The former defines keypoints by sampling the corners of the 3D bounding box, hand-designed keypoints, and farthest points. On the other hand, dense correspondence methods predict pixel-level 3D coordinates within a cropped 2D region. Most existing geometry-based methods follow a two-stage strategy: first, determine the 2D-3D correspondence, and then solve the pose using the PnP algorithm. However, instance-level object pose estimation is usually for known object categories, meaning that the model can only identify and estimate object categories and instances existing in the training set, and cannot accurately estimate new categories or new instances.

[0071] Compared to instance-level 6D pose estimation, category-level pose estimation does not require training on each specific object instance but instead models the object category. Therefore, the model has greater generalization ability and can adapt to objects in different scenes and instances. Traditional category-level work proposes Normalized Object Coordinate Space (NOCS), aligning different objects of the same category into a unified normalized 3D space. Based on NOCS graph prediction, the least squares method is used to solve for 6D pose. Following this research path, subsequent work focuses on learning more accurate NOCS representations. Besides using NOCS, 6D pose estimation can also be achieved by directly regressing the pose estimate, estimating keypoint locations, or reconstructing objects and then aligning them. However, the category-level pose estimation problem is more unconstrained and challenging due to the large intra-class shape variations. These methods require training on manually labeled real or synthetic datasets to ensure their generalization ability and accuracy across instance scenes.

[0072] In summary, self-supervised class-level pose estimation methods are of great significance in the field of object pose estimation. These methods learn object representations from unlabeled data, utilizing the natural constraints and structural information within the data to guide model training. In this way, the model can better adapt to various complex scenarios and the emergence of new categories, improving the algorithm's generalization ability and adaptability. Furthermore, self-supervised class-level pose estimation methods can help address the overfitting and insufficient labeled data problems inherent in traditional methods, providing entirely new ideas and methods for the development of object pose estimation technology.

[0073] In view of this, embodiments of the present disclosure provide a training method for an object pose estimation model, an object pose estimation method, and an apparatus thereof. The method includes: acquiring a pose training set, which includes multiple target object training images and object prior vectors corresponding to each target object training image; for each target object training image, performing convolution processing on the target object training image using a first two-dimensional convolutional model to obtain global object features; processing the global object features and object prior vectors using a deformation network to obtain deformation point cloud information; processing the global object features and deformation point cloud information using a convolutional neural network to obtain two-dimensional feature vectors and three-dimensional feature vectors; performing multi-level feature matching processing on the two-dimensional feature vectors and three-dimensional feature vectors to obtain three-dimensional correspondence parameters; iteratively adjusting the network parameters of the initial pose estimation model based on the three-dimensional correspondence parameters and a three-dimensional keypoint heatmap to obtain a trained object pose estimation model.

[0074] Figure 1 The illustration shows a training method for an object pose estimation model or an application scenario of an object pose estimation method according to embodiments of the present disclosure.

[0075] like Figure 1 As shown, application scenario 100 according to this embodiment may include autonomous vehicles. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0076] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0077] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0078] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0079] It should be noted that the training method and object pose estimation method provided in the embodiments of this disclosure can generally be executed by server 105. Correspondingly, the training device and object pose estimation device provided in the embodiments of this disclosure can generally be located in server 105. The training method and object pose estimation method provided in the embodiments of this disclosure can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the training device and object pose estimation device provided in the embodiments of this disclosure can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0080] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0081] Figure 2 A flowchart illustrating a training method for an object pose estimation model according to an embodiment of the present disclosure is shown.

[0082] like Figure 2 As shown, the training method for the object pose estimation model in this embodiment includes operations S210 to S260, which can be executed by an electronic device.

[0083] In operation S210, a pose training set is obtained, which includes multiple target object training images and object prior vectors corresponding to each target object training image.

[0084] In operation S220, for each target object training image, the first two-dimensional convolutional model is used to perform convolution processing on the target object training image to obtain the global features of the object.

[0085] In operation S230, the deformation network is used to process the global features of the object and the prior vector of the object to obtain the deformation point cloud information;

[0086] In operation S240, a convolutional neural network is used to process the global features and deformation point cloud information of the object to obtain two-dimensional feature vectors and three-dimensional feature vectors.

[0087] In operation S250, multi-level feature matching processing is performed on the two-dimensional feature vector and the three-dimensional feature vector to obtain the three-dimensional correspondence parameter. The three-dimensional correspondence parameter represents the correlation and correspondence between pixels in the training image of the target object and point clouds in the deformed point cloud information.

[0088] In operation S260, the network parameters of the initial pose estimation model are iteratively adjusted according to the three-dimensional correspondence parameters and the three-dimensional key point heatmap to obtain the trained object pose estimation model. The initial pose estimation model includes a first two-dimensional convolutional model, a deformation network, a convolutional neural network, and an object detection model. The three-dimensional key point heatmap is obtained by processing the target object training image and the object prior vector using the object detection model.

[0089] According to embodiments of this disclosure, the pose training set includes multiple target object training images I and an object prior vector P∈R corresponding to each target object training image. 3×N This refers to prior knowledge of the category to which the object belongs. The object detection model can be an end-to-end object detection (DETR with Improved deNoising anchOr boxes, DINO) model.

[0090] According to embodiments of this disclosure, a first two-dimensional convolutional neural network (2D-CNN) is used to perform convolution processing on the training image of the target object to extract global style features, i.e., the global features of the object S∈R. s Here, deformable networks can refer to deformable convolutional neural networks. Simultaneously, deformable networks are used to process the global features S∈R of an object. s and the prior vector of the object P∈R 3×N Deformation point cloud information is obtained.

[0091] According to embodiments of this disclosure, a convolutional neural network is used to process the global features S∈R of an object. s and deformable point cloud information Obtain the two-dimensional eigenvector F img and three-dimensional feature vector F pc For the two-dimensional eigenvector F img and three-dimensional feature vector F pc Multi-level feature matching is performed to obtain the three-dimensional correspondence parameter C.3D Subsequently, an end-to-end object detection model can be used to train the target object image I and the corresponding object prior vector P∈R. 3×N After processing, a three-dimensional heatmap of key points can be obtained. Based on this, according to the three-dimensional correspondence parameter C 3D and 3D key point heatmap Consistency calculations are performed to iteratively adjust the network parameters of the initial pose estimation model, resulting in a trained object pose estimation model.

[0092] According to embodiments of this disclosure, the object pose estimation model utilizes multimodal data of the target object training image and object prior vectors during training, thereby providing richer and more comprehensive global features and deformation point cloud information of the object. This can improve the estimation accuracy and robustness of the object pose estimation model. Furthermore, by performing multi-level feature matching processing on the two-dimensional and three-dimensional feature vectors obtained from the convolutional neural network, the limitations of single-modal data can be compensated, and the impact of data noise and uncertainty on pose estimation can be reduced, thereby effectively improving the prediction accuracy of the object pose estimation model.

[0093] Figure 3 A flowchart illustrating a training method for an object pose estimation model according to another embodiment of the present disclosure is shown.

[0094] According to embodiments of this disclosure, deformation point cloud information is obtained by processing global features and prior vectors of an object using a deformation network, including:

[0095] The deformation vector is obtained by processing the global features of the object using a deformation network.

[0096] Deformation point cloud information is generated based on the deformation vector and the object's prior vector.

[0097] According to embodiments of this disclosure, referring to Figure 3 The obtained global features S∈R of the object s The deformation network is fed into the model to predict the point-by-point deformation vector ΔP. Based on the deformation vector and the object's prior vector P, the deformed point cloud information, i.e., the deformed point cloud information, is generated. See formula (1) for details:

[0098]

[0099] According to embodiments of this disclosure, the convolutional neural network includes a second two-dimensional convolutional model and a three-dimensional convolutional model.

[0100] According to embodiments of this disclosure, a convolutional neural network is used to process global features and deformed point cloud information of an object to obtain two-dimensional feature vectors and three-dimensional feature vectors, including:

[0101] The second two-dimensional convolutional model is used to extract features from the global features of the object to obtain a two-dimensional feature vector.

[0102] A three-dimensional convolutional model is used to extract features from deformed point cloud information to obtain three-dimensional feature vectors.

[0103] According to embodiments of this disclosure, the second two-dimensional convolutional model is structurally similar to the first two-dimensional convolutional model (Two-Dimensional Convolutional Neural Network, 2D-CNN). The three-dimensional convolutional model can refer to a 3D-CNN (Three-Dimensional Convolutional Neural Network) model.

[0104] According to embodiments of this disclosure, multimodal feature alignment plays a crucial role in pose estimation. By aligning multimodal information from different sensors or data sources, richer and more comprehensive feature representations can be provided, thereby improving the accuracy and robustness of pose estimation. For example, combining data from vision, inertial, and depth sensors can yield more comprehensive environmental perception information, helping to accurately estimate the position, pose, and motion of objects in complex scenes. Multimodal feature alignment can also compensate for the limitations of single-modal data, providing more comprehensive and robust feature representations and mitigating the impact of data noise and uncertainty on pose estimation. Therefore, multimodal feature alignment provides more information sources and higher accuracy for pose estimation.

[0105] According to embodiments of this disclosure, firstly, feature extraction needs to be performed on the multimodal data. For each modality of data (such as images, depth, etc.), a corresponding feature extractor needs to be designed. These feature extractors can be pre-trained deep neural network models or model structures customized according to specific tasks. For image data, ResNet18 is used to extract dense image pixel features, i.e., the two-dimensional feature vector F in this disclosure. img For the deformed point cloud information, i.e. deformed point cloud information The PointNet model can be used to extract point cloud features, i.e., the three-dimensional feature vector F, from the point cloud. pc .

[0106] According to embodiments of this disclosure, given the global features S of an object and the deformed point cloud information You can use E img This represents the feature extractor corresponding to the image modality, i.e., the second two-dimensional convolutional model, E. pc This represents the feature extractor corresponding to the depth modality, i.e., the 3D convolutional model. The dense features are extracted pixel-by-pixel and point-by-point using these two feature extractors respectively, and the 2D feature vector F can be obtained by referring to formula (2).img and three-dimensional feature vector F pc :

[0107] F img = E img (S) (2)

[0108]

[0109] According to embodiments of this disclosure, after passing through different feature extractors, data from each modality is mapped into a corresponding feature space. These features are then mapped into a shared feature space to achieve the association and fusion of data from different modalities. A loss function is used to constrain the feature space alignment of features from different modalities. The relationship between pairs of data from different modalities is measured based on the similarity or distance of the data. By optimizing the loss function, i.e., the consistency constraint value, similar data are brought closer together in the shared space, while dissimilar data are moved further apart.

[0110] According to embodiments of this disclosure, dense 2D-3D correspondence plays a crucial role in determining the position, orientation, and size of objects in the fields of pose estimation and object recognition. By mapping feature points in an image to a 3D model, the pose of an object can be accurately estimated, providing important support for object recognition and localization.

[0111] The specific operational plan consists of several steps. The first step is the similarity calculation stage. After obtaining the dense image features and point cloud features, the similarity between the image features and point cloud features is calculated using a suitable similarity metric, such as cosine similarity. Then, a matching calculation is performed to obtain the similarity score between them.

[0112] Next comes the matching and filtering stage, where appropriate thresholds are set based on similarity scores to filter matching pairs. Simultaneously, noise is removed from the matching pairs to improve accuracy and robustness. In the matching verification stage, geometric consistency checks are performed on the matched point pairs. For example, the transformation matrix is ​​estimated using the RANSAC algorithm, and the consistency of the matches is verified. Visual inspection is also conducted to check the distribution of matching points and the quality of the matches, followed by manual verification and adjustments. Finally, in the optimization and application stage, the algorithm parameters are fine-tuned, and the matching results are applied to real-world scenarios, such as 3D reconstruction, virtual reality, and augmented reality, providing fundamental support and technical assurance for related applications.

[0113] Through the above steps, dense 2D-3D correspondence matching between images and point clouds can be achieved, providing accurate and reliable matching results for applications in related fields and providing technical support for solving practical problems.

[0114] According to embodiments of this disclosure, the multimodal feature alignment technology achieves the association and fusion of features of different modal data through steps such as feature extraction, feature alignment model design, and loss function constraints, providing an effective solution for the processing and analysis of multimodal data.

[0115] According to embodiments of this disclosure, multi-level feature matching processing is performed on two-dimensional feature vectors and three-dimensional feature vectors to obtain three-dimensional correspondence parameters, including:

[0116] Based on the two-dimensional and three-dimensional feature vectors, a first relation matrix is ​​generated, which represents the correspondence between each pixel and all point clouds. This step includes:

[0117] Similarity scores are obtained by calculating the similarity between two-dimensional and three-dimensional feature vectors.

[0118] The similarity scores are normalized to obtain the first relation matrix;

[0119] For each pixel, extract multiple correspondence coefficients corresponding to the pixel from the first relation matrix;

[0120] Generate corresponding parameters based on multiple correspondence coefficients and the point cloud coordinates corresponding to each correspondence coefficient;

[0121] Generate three-dimensional correspondence parameters based on multiple corresponding parameters.

[0122] According to embodiments of this disclosure, in order to fully utilize the complementarity of features from different modalities, we calculate the similarity between them to more accurately capture the correlation between data. In the obtained dense image features (i.e., two-dimensional feature vector F) img ) and point cloud features (i.e., three-dimensional feature vectors F) pc After that, the similarity between image features and point cloud features is calculated using a suitable similarity metric, such as cosine similarity. Then, matching is performed to obtain a similarity score between them. The similarity score is calculated as shown in formula (3):

[0123]

[0124] Among them, f pixel That is, F img f vertex That is, F pc .

[0125] According to embodiments of this disclosure, the similarity score is normalized using the Softmax function to obtain the correspondence matrix between each pixel and all vertices, i.e., the first relation matrix M. image-mesh The specific calculation is shown in formula (4):

[0126]

[0127] Where, N vertices represents the number of points in the point cloud mesh, where i represents the pixel index and j represents the point cloud index. Similarly, the similarity score is normalized using the Softmax function to obtain the correspondence matrix between each pixel and all vertices, i.e., the second relation matrix M. mesh-image The specific calculation is shown in formula (5):

[0128]

[0129] Where, N pixels It refers to the number of pixels.

[0130] According to embodiments of this disclosure, in order to transform an uncertain correspondence distribution into a definite correspondence relationship, the calculated similarity distribution needs further processing. By using the calculated similarity as weights, pixel coordinates or point cloud coordinates can be weighted, reducing the uncertainty of the correspondence distribution. The purpose of this is to emphasize pixels or point clouds with high similarity, thereby generating more accurate correspondence relationships.

[0131] According to embodiments of this disclosure, from M mesh-image Extract the correspondence distribution of point cloud j from the data. Based on this correspondence, the weighted average corresponds to the pixel coordinates U. U is the 2D position corresponding to all foreground points in the RGB image, ranging from... Thus, for all points in the point cloud, the 2D correspondence C of the point cloud is calculated according to formula (6). 2d :

[0132]

[0133] Where, N pixels It represents the number of pixels in the grid. U i It is a pair containing the x and y coordinates. Similarly, for pixel i in a 2D image, from the first relation matrix M... image-mesh Extract the correspondence coefficients corresponding to pixel i from the data. The point cloud coordinates W corresponding to the weighted average, Therefore, based on multiple corresponding parameters, the corresponding parameters are generated by referring to formula (7). Based on multiple corresponding parameters Generate three-dimensional correspondence parameters C 3d .

[0134]

[0135] Where, N verticesThis represents the number of point clouds in the reconstructed model. V i It consists of triples containing x, y, and z coordinates. These weighted average coordinates serve as the final correspondence, representing a definitive correspondence between pixels and point clouds. This weighted method effectively minimizes uncertainty and improves the accuracy and stability of the correspondence. By considering similarity weights, we place greater emphasis on pixels or point clouds with high similarity, thus making the correspondence determination more reliable. This approach allows us to better understand and utilize the relationship between images and point clouds, providing a more reliable foundation for subsequent data analysis and applications.

[0136] According to embodiments of this disclosure, the three-dimensional key point heatmap is obtained in the following manner:

[0137] The target object training image and the object prior vector are processed using an object detection model to obtain the query matrix;

[0138] A 3D keypoint heatmap is generated based on the query matrix and the key matrix, where the key matrix is ​​obtained by mapping the 3D feature vectors.

[0139] According to embodiments of this disclosure, to alleviate the under-constraint of correspondences under unsupervised settings, the query vectors of the DINO model can be used to model more robust correspondence relationships, thereby enhancing the constraint of dense correspondences. Specifically, the DINO model processes the target object training image and the object prior vector, and the generated feature vectors are used as query vectors to establish correspondence relationships at the keypoint level. These query vectors are denoted as f. k ={k1,…,k pts}∈R pts×d Where pts is the number of keypoints and d is the feature dimension. These feature vectors can capture the semantic and visual information of an image, possessing strong expressive power and robustness. Query vector f k With image features f pixel Interactive, adaptively capturing key point information, query vector f k The image features f are used as the learnable query matrix Q. pixel Mapped to the key matrix K and value matrix V respectively, and the key point heatmap is generated according to formula (8):

[0140]

[0141]

[0142] Where, d k It controls the temperature coefficient of the canonical system. It is the query vector f k With image features f pixelThe interaction generates a keypoint heatmap. This heatmap also enables dynamic updates to the query vector. A simple matrix multiplication between the keypoint heatmap and the image pixel coordinates U generates pts specific 2D keypoint coordinates U. kpts As shown in formula (9):

[0143]

[0144] Similarly, for point cloud modalities, the query vector Q can also be related to the point cloud features f. vertex Interactive update. Q is the query vector that has been interactively updated with image features, where f is the point cloud feature. vertex Mapped to the key matrix K and value matrix V respectively, a three-dimensional key point heatmap is generated by referring to formula (10):

[0145]

[0146]

[0147] in, It is the query vector Q and the point cloud feature f vertex A 3D key point heatmap generated through interaction.

[0148] According to embodiments of this disclosure, the network parameters of an initial pose estimation model are iteratively adjusted based on 3D correspondence parameters and a 3D keypoint heatmap to obtain a trained object pose estimation model, including:

[0149] Generate the coordinates of three-dimensional key points based on the 3D key point heatmap and point cloud data;

[0150] Generate consistency constraint values ​​based on the 3D correspondence parameters and the coordinates of the 3D key points;

[0151] The network parameters of the initial pose estimation model are iteratively adjusted based on the consistency constraint value to obtain the object pose estimation model.

[0152] According to embodiments of this disclosure, a consistency constraint value is generated based on three-dimensional correspondence parameters and three-dimensional keypoint coordinates, including:

[0153] Perform bilinear interpolation on the three-dimensional correspondence parameters to obtain linear interpolation results;

[0154] Based on the linear interpolation results and the coordinates of the three key points, consistency constraint values ​​are generated.

[0155] According to embodiments of this disclosure, a three-dimensional key point heatmap Performing a matrix dot product with the point cloud coordinates W can also generate pts specific 3D keypoint coordinates W. kpts As shown in formula (11):

[0156]

[0157] Here, the point cloud coordinates W can refer to the position of the target object in the known prior vector P of the object.

[0158] According to embodiments of this disclosure, the above calculations generate pts one-to-one 2D coordinate and 3D coordinate pairs, which are the keypoint-level correspondence relationships C. pts For correspondences at the same location, keypoint-level correspondences should maintain consistency with dense correspondences. Therefore, for the 3D correspondence parameter C... 3d Perform bilinear interpolation to obtain the linear interpolation result W. bilinear As shown in formula (12):

[0159] W bilinear = BiLinear(C 3d (12)

[0160] Where BiLinear() represents the bilinear interpolation operation, W bilinear Representative according to U kpts For C 3d The result of bilinear interpolation. Keypoint-level correspondences are obtained through a weighted average of heatmaps, therefore bilinear interpolation is needed to obtain the 3D coordinates of the corresponding 2D locations from the dense correspondences. The consistency constraint value of the correspondence prediction can be calculated according to formula (13):

[0161] L consistency =∑(i,j)|W bilinear (i,j)-W kpts (i,j)| (13)

[0162] In the process of modeling correspondence relationships using the DINO model, unsupervised learning methods can be employed. By maximizing the similarity between matching image and point cloud features, the model becomes more robust and can automatically discover correspondence relationships between images, thereby improving the accuracy and stability of the correspondence relationships.

[0163] According to embodiments of this disclosure, a dense correspondence estimation technique is proposed. Compared to traditional correspondence estimation methods, the introduction of semantic key points enhances the robustness of correspondence relationships, thereby improving the correspondence quality under self-supervised settings.

[0164] According to embodiments of this disclosure, addressing the difficulty in obtaining labeled data during supervised training in existing technologies, this disclosure provides a training framework based on 3D object reconstruction and rendering techniques. This framework reconstructs 3D object models to generate sufficiently accurate correspondence estimation results and uses reconstruction loss to indirectly optimize the parameters of the entire network. This allows the network to maintain good performance even under weakly supervised self-supervised conditions and significantly reduces the need for labeled training data.

[0165] According to embodiments of this disclosure, this patent introduces an advanced multimodal feature alignment technology to address the differences between different modal features, aiming to achieve effective fusion and correlation of cross-modal feature information. This technology can integrate information from multiple sensors or data sources, enabling the correlation and fusion of various modal features during analysis. This approach not only improves the efficiency of data integration and utilization but also enhances the accuracy and reliability of data analysis, providing a solid foundation for system performance optimization and application.

[0166] Figure 4 A flowchart illustrating an object pose estimation method according to an embodiment of the present disclosure is shown schematically.

[0167] like Figure 4 As shown, object pose estimation methods include:

[0168] Obtain the image to be processed, which contains the object to be processed, and the average shape vector of the object category to which the object to be processed belongs;

[0169] The object pose estimation model is used to process the image to be processed and the average shape vector to obtain the predicted global features, predicted point cloud information and predicted 3D correspondence parameters.

[0170] Regression processing is performed on the predicted global features to obtain the initial pose data;

[0171] Based on the initial pose data, predicted point cloud information, and predicted 3D correspondence parameters, a rendering atlas is generated, which includes the pose rendering map, mask map, and depth map of the image to be processed.

[0172] According to embodiments of this disclosure, the RGBDI of the image to be processed and the average shape vector P are input into a trained object pose estimation model, thereby obtaining predicted global features, predicted point cloud information, and predicted 3D correspondence parameters. Subsequently, regression processing is performed on the predicted global features to obtain a coarse initial pose data [R|t]. Finally, the initial pose data [R|t], the predicted point cloud information, and the predicted 3D correspondence parameters are combined to obtain the pose rendering map, mask map, and depth map of the image to be processed.

[0173] According to embodiments of this disclosure, supervised category-level pose estimation has wide applications in the field of autonomous driving. First, a multi-camera system and LiDAR mounted on an autonomous vehicle can be used to acquire image data and 3D point cloud data of the road environment from different angles and distances. Next, the image data is preprocessed, including lens distortion removal and image registration, to ensure image accuracy and consistency. The LiDAR data is filtered and transformed to improve data quality. Finally, using the dense correspondence estimation technology, 3D object reconstruction and rendering technology, and multimodal feature alignment technology introduced in this patent, real-time perception and pose estimation of the surrounding road environment are achieved, providing support for precise navigation and obstacle avoidance in autonomous vehicles.

[0174] According to embodiments of this disclosure, the object pose estimation model utilizes multimodal data of the target object training image and object prior vectors during training, thereby providing richer and more comprehensive global features and deformation point cloud information of the object. This can improve the estimation accuracy and robustness of the object pose estimation model. Furthermore, by performing multi-level feature matching processing on the two-dimensional and three-dimensional feature vectors obtained from the convolutional neural network, the limitations of single-modal data can be compensated, and the impact of data noise and uncertainty on pose estimation can be reduced, thereby effectively improving the prediction accuracy of the object pose estimation model.

[0175] Figure 5 A schematic block diagram of a training apparatus for an object pose estimation model according to an embodiment of the present disclosure is shown.

[0176] like Figure 5 As shown, the training device for the object pose estimation model in this embodiment includes a first acquisition module 510, a first convolution module 520, a deformation module 530, a second convolution module 540, a feature matching module 550, and a training module 560.

[0177] The first acquisition module 510 is used to acquire a pose training set, wherein the pose training set includes multiple target object training images and object prior vectors corresponding to each target object training image;

[0178] The first convolutional module 520 is used to perform convolution processing on the target object training image for each target object training image using the first two-dimensional convolutional model to obtain the global features of the object.

[0179] The deformation module 530 is used to process the global features and prior vectors of the object using the deformation network to obtain the deformation point cloud information;

[0180] The second convolutional module 540 is used to process global features and deformed point cloud information of an object using a convolutional neural network to obtain two-dimensional feature vectors and three-dimensional feature vectors.

[0181] The feature matching module 550 is used to perform multi-level feature matching processing on two-dimensional feature vectors and three-dimensional feature vectors to obtain three-dimensional correspondence parameters. The three-dimensional correspondence parameters characterize the correlation and correspondence between pixels in the training image of the target object and point clouds in the deformed point cloud information.

[0182] The training module 560 is used to iteratively adjust the network parameters of the initial pose estimation model based on the 3D correspondence parameters and the 3D key point heatmap to obtain the trained object pose estimation model. The initial pose estimation model includes a first 2D convolutional model, a deformation network, a convolutional neural network, and an object detection model. The 3D key point heatmap is obtained by processing the target object training image and the object prior vector using the object detection model.

[0183] According to embodiments of this disclosure, the object pose estimation model utilizes multimodal data of the target object training image and object prior vectors during training, thereby providing richer and more comprehensive global features and deformation point cloud information of the object. This can improve the estimation accuracy and robustness of the object pose estimation model. Furthermore, by performing multi-level feature matching processing on the two-dimensional and three-dimensional feature vectors obtained from the convolutional neural network, the limitations of single-modal data can be compensated, and the impact of data noise and uncertainty on pose estimation can be reduced, thereby effectively improving the prediction accuracy of the object pose estimation model.

[0184] According to embodiments of this disclosure, the deformation module 530 includes:

[0185] Deformation unit, used to process global features of an object using deformation network to obtain deformation vector;

[0186] The first generation unit is used to generate deformed point cloud information based on the deformation vector and the object prior vector.

[0187] According to embodiments of this disclosure, the convolutional neural network includes a second two-dimensional convolutional model and a three-dimensional convolutional model.

[0188] According to embodiments of this disclosure, the second convolution module 540 includes:

[0189] The first feature extraction unit is used to perform feature extraction processing on the global features of the object using the second two-dimensional convolution model to obtain a two-dimensional feature vector.

[0190] The second feature extraction unit is used to extract features from the deformed point cloud information using a three-dimensional convolutional model to obtain a three-dimensional feature vector.

[0191] According to embodiments of this disclosure, the feature matching module 550 includes:

[0192] The second generation unit is used to generate a first relation matrix based on the two-dimensional feature vector and the three-dimensional feature vector, wherein the first relation matrix represents the correspondence between each pixel and all point clouds;

[0193] The extraction unit is used to extract multiple correspondence coefficients corresponding to each pixel from the first relation matrix for each pixel.

[0194] The third generation unit is used to generate corresponding parameters based on multiple correspondence coefficients and the point cloud coordinates corresponding to each correspondence coefficient;

[0195] The fourth generation unit is used to generate three-dimensional correspondence parameters based on multiple corresponding parameters.

[0196] According to embodiments of this disclosure, the second generation unit includes:

[0197] The calculation subunit is used to calculate the similarity between two-dimensional and three-dimensional feature vectors to obtain a similarity score;

[0198] The normalization sub-unit is used to normalize the similarity scores to obtain the first relation matrix.

[0199] According to embodiments of this disclosure, the three-dimensional key point heatmap is obtained in the following manner:

[0200] The detection unit is used to process the target object training image and the object prior vector using the target detection model to obtain the query matrix;

[0201] The fifth generation unit is used to generate a 3D key point heatmap based on the query matrix and the key matrix, where the key matrix is ​​obtained by mapping the 3D feature vectors.

[0202] According to embodiments of this disclosure, the training module 560 includes:

[0203] The sixth generation unit is used to generate three-dimensional key point coordinates based on the three-dimensional key point heatmap and point cloud data;

[0204] The seventh generation unit is used to generate consistency constraint values ​​based on the three-dimensional correspondence parameters and the coordinates of the three-dimensional key points;

[0205] The training unit is used to iteratively adjust the network parameters of the initial pose estimation model according to the consistency constraint value to obtain the object pose estimation model.

[0206] According to embodiments of this disclosure, the seventh generation unit includes:

[0207] The interpolation subunit is used to perform bilinear interpolation on the three-dimensional correspondence parameters to obtain linear interpolation results.

[0208] Generate sub-units to generate consistency constraint values ​​based on the linear interpolation results and the coordinates of the three-dimensional key points.

[0209] Figure 6 A schematic block diagram of an object pose estimation apparatus according to an embodiment of the present disclosure is shown.

[0210] like Figure 6 As shown, the object pose estimation device 600 includes a second acquisition module 610, a processing module 620, a regression module 630, and a generation module 640.

[0211] The second acquisition module 610 is used to acquire the image to be processed containing the object to be processed and the average shape vector of the object category to which the object to be processed belongs;

[0212] The processing module 620 is used to process the image to be processed and the average shape vector using the object pose estimation model to obtain the predicted global features, predicted point cloud information and predicted 3D correspondence parameters.

[0213] The regression module 630 is used to perform regression processing on the predicted global features to obtain the initial pose data;

[0214] The generation module 640 is used to generate a rendering atlas based on the initial pose data, predicted point cloud information and predicted 3D correspondence parameters. The rendering atlas includes the pose rendering map, mask map and depth map of the image to be processed.

[0215] According to embodiments of this disclosure, the object pose estimation model utilizes multimodal data of the target object training image and object prior vectors during training, thereby providing richer and more comprehensive global features and deformation point cloud information of the object. This can improve the estimation accuracy and robustness of the object pose estimation model. Furthermore, by performing multi-level feature matching processing on the two-dimensional and three-dimensional feature vectors obtained from the convolutional neural network, the limitations of single-modal data can be compensated, and the impact of data noise and uncertainty on pose estimation can be reduced, thereby effectively improving the prediction accuracy of the object pose estimation model.

[0216] According to embodiments of this disclosure, any multiple modules selected from the first acquisition module 510, the first convolution module 520, the deformation module 530, the second convolution module 540, the feature matching module 550, the training module 560, or the second acquisition module 610, the processing module 620, the regression module 630, and the generation module 640 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the following: the first acquisition module 510, the first convolution module 520, the deformation module 530, the second convolution module 540, the feature matching module 550, the training module 560, or the second acquisition module 610, the processing module 620, the regression module 630, and the generation module 640, can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the first acquisition module 510, the first convolution module 520, the deformation module 530, the second convolution module 540, the feature matching module 550, the training module 560, or the second acquisition module 610, the processing module 620, the regression module 630, and the generation module 640 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0217] Figure 7 A block diagram of an electronic device suitable for implementing a training method or object pose estimation method for an object pose estimation model according to embodiments of the present disclosure is shown schematically.

[0218] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0219] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0220] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0221] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0222] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0223] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the training method or object pose estimation method of the object pose estimation model provided in the embodiments of this disclosure.

[0224] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0225] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0226] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0227] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0228] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A training method for an object pose estimation model, characterized in that, The method includes: Obtain a pose training set, wherein the pose training set includes multiple target object training images and object prior vectors corresponding to each target object training image; For each training image of the target object, the training image of the target object is convolved using a first two-dimensional convolution model to obtain the global features of the object. The deformation point cloud information is obtained by processing the global features of the object and the prior vector of the object using a deformation network. The global features of the object and the deformation point cloud information are processed using a convolutional neural network to obtain two-dimensional feature vectors and three-dimensional feature vectors. Multi-level feature matching processing is performed on the two-dimensional feature vector and the three-dimensional feature vector to obtain three-dimensional correspondence parameters, wherein the three-dimensional correspondence parameters characterize the association and correspondence between pixels in the training image of the target object and point clouds in the deformed point cloud information; The network parameters of the initial pose estimation model are iteratively adjusted based on the three-dimensional correspondence parameters and the three-dimensional keypoint heatmap to obtain a trained object pose estimation model. The initial pose estimation model includes the first two-dimensional convolutional model, the deformation network, the convolutional neural network, and the object detection model. The three-dimensional keypoint heatmap is obtained by processing the target object training image and the object prior vector using the object detection model.

2. The method according to claim 1, characterized in that, The deformation point cloud information is obtained by processing the global features of the object and the prior vector of the object using a deformation network, including: The deformation network is used to process the global features of the object to obtain the deformation vector; The deformation point cloud information is generated based on the deformation vector and the object prior vector.

3. The method according to claim 1 or 2, characterized in that, The convolutional neural network includes a second two-dimensional convolutional model and a three-dimensional convolutional model; Specifically, a convolutional neural network is used to process the global features of the object and the deformation point cloud information to obtain two-dimensional feature vectors and three-dimensional feature vectors, including: The second two-dimensional convolution model is used to perform feature extraction processing on the global features of the object to obtain the two-dimensional feature vector; The three-dimensional convolutional model is used to perform feature extraction processing on the deformed point cloud information to obtain the three-dimensional feature vector.

4. The method according to claim 1, characterized in that, Multi-level feature matching is performed on the two-dimensional feature vector and the three-dimensional feature vector to obtain three-dimensional correspondence parameters, including: Based on the two-dimensional feature vector and the three-dimensional feature vector, a first relation matrix is ​​generated, wherein the first relation matrix represents the correspondence between each pixel and all point clouds; For each pixel, extract multiple corresponding relationship coefficients from the first relationship matrix; Generate corresponding parameters based on the multiple corresponding coefficients and the point cloud coordinates corresponding to each of the corresponding coefficients; The three-dimensional correspondence parameters are generated based on the multiple corresponding parameters.

5. The method according to claim 4, characterized in that, Based on the two-dimensional feature vector and the three-dimensional feature vector, a first relation matrix is ​​generated, including: The similarity between the two-dimensional feature vector and the three-dimensional feature vector is calculated to obtain a similarity score; The similarity scores are normalized to obtain the first relationship matrix.

6. The method according to claim 1, characterized in that, The three-dimensional key point heatmap was obtained in the following way: The target detection model is used to process the training image of the target object and the prior vector of the object to obtain a query matrix; The three-dimensional key point heatmap is generated based on the query matrix and the key matrix, wherein the key matrix is ​​obtained by mapping the three-dimensional feature vector.

7. The method according to claim 6, characterized in that, The network parameters of the initial pose estimation model are iteratively adjusted based on the three-dimensional correspondence parameters and the three-dimensional keypoint heatmap to obtain a trained object pose estimation model, including: Based on the three-dimensional key point heatmap and point cloud data, generate three-dimensional key point coordinates; Based on the three-dimensional correspondence parameters and the coordinates of the three key points, a consistency constraint value is generated; The network parameters of the initial pose estimation model are iteratively adjusted according to the consistency constraint value to obtain the object pose estimation model; Specifically, based on the three-dimensional correspondence parameters and the coordinates of the three-dimensional key points, consistency constraint values ​​are generated, including: Perform bilinear interpolation on the three-dimensional correspondence parameters to obtain linear interpolation results; The consistency constraint value is generated based on the linear interpolation result and the coordinates of the three key points.

8. A method for estimating object pose, characterized in that, The method includes: Obtain the image to be processed containing the object to be processed and the average shape vector of the object category to which the object to be processed belongs; The object pose estimation model is used to process the image to be processed and the average shape vector to obtain predicted global features, predicted point cloud information and predicted three-dimensional correspondence parameters, wherein the object pose estimation model is trained using the method described in any one of claims 1 to 7. The predicted global features are subjected to regression processing to obtain the initial pose data; Based on the initial pose data, the predicted point cloud information, and the predicted 3D correspondence parameters, a rendering atlas is generated, wherein the rendering atlas includes the pose rendering map, mask map, and depth map of the image to be processed.

9. A training device for an object pose estimation model, characterized in that, The device includes: The first acquisition module is used to acquire a pose training set, wherein the pose training set includes multiple target object training images and object prior vectors corresponding to each target object training image; The first convolution module is used to perform convolution processing on the training image of the target object for each target object training image using a first two-dimensional convolution model to obtain global features of the object. The deformation module is used to process the global features of the object and the prior vector of the object using a deformation network to obtain deformation point cloud information; The second convolution module is used to process the global features of the object and the deformation point cloud information using a convolutional neural network to obtain two-dimensional feature vectors and three-dimensional feature vectors. The feature matching module is used to perform multi-level feature matching processing on the two-dimensional feature vector and the three-dimensional feature vector to obtain three-dimensional correspondence parameters, wherein the three-dimensional correspondence parameters characterize the association and correspondence between pixels in the training image of the target object and point clouds in the deformed point cloud information; The training module is used to iteratively adjust the network parameters of the initial pose estimation model according to the three-dimensional correspondence parameters and the three-dimensional keypoint heatmap to obtain a trained object pose estimation model. The initial pose estimation model includes the first two-dimensional convolutional model, the deformation network, the convolutional neural network, and the object detection model. The three-dimensional keypoint heatmap is obtained by processing the target object training image and the object prior vector using the object detection model.

10. An object pose estimation device, characterized in that, The device includes: The second acquisition module is used to acquire the image to be processed containing the object to be processed and the average shape vector of the object category to which the object to be processed belongs; The processing module is used to process the image to be processed and the average shape vector using an object pose estimation model to obtain predicted global features, predicted point cloud information and predicted three-dimensional correspondence parameters, wherein the object pose estimation model is trained using the method described in any one of claims 1 to 7. The regression module is used to perform regression processing on the predicted global features to obtain initial pose data; The generation module is used to generate a rendering atlas based on the initial pose data, the predicted point cloud information, and the predicted 3D correspondence parameters, wherein the rendering atlas includes the pose rendering map, mask map, and depth map of the image to be processed.

Images