6D pose estimation method based on cross-modal reconstruction self-supervised training

Abstract

The invention discloses a 6D pose estimation method based on cross-modal reconstruction self-supervised training, and the method comprises the steps: obtaining a scene image and a depth image, cutting the scene image and the depth image to obtain a target image containing a target object and the depth image, processing the target image into a mask image, extracting image features and point cloud features, carrying out the training of a cross-modal cross attention mechanism to obtain image point cloud fusion features, and carrying out the self-supervised training. And reconstructing an image and outputting a pose estimation feature pair to realize self-supervision, and outputting a 6D estimation pose through a pose decoder by combining the trained model weight with the pose estimation feature pair after loading. According to the method, information from different sensor modalities is effectively fused, redundant information is removed by using a selected channel fusion strategy, a more accurate 6D pose is obtained, and accurate position information is provided for subsequent further control identification. The characteristics of an object can be understood from different angles and dimensions, information of various sensors is combined for use under the conditions of shielding, insufficient illumination and the like, and better robustness is achieved compared with single-mode input.

Description

Technical Field

[0001] The present invention relates to a 6D pose estimation method based on cross-modal reconstruction self-supervised training, and belongs to the field of computer vision and multimodal information fusion. Background Art

[0002] Multimodal instance-level 6D pose estimation is a key technology that achieves high-precision estimation of the position and orientation of known object instances in three-dimensional space by fusing multiple sensor data (such as RGB images, depth information, point clouds, etc.). It determines the position and orientation of objects in three-dimensional space by analyzing the depth map and RGB image captured by the depth camera, with a total of six degrees of freedom. Therefore, mastering these six degrees of freedom is crucial for computers to accurately understand and manipulate target objects and deeply analyze complex scenes. Although multimodal object pose estimation is very important and has been studied to a certain extent, there is still room for improvement in the fusion of RGB information and depth information in the pose estimation algorithm.

[0003] Currently, there are two main ways to fuse depth information and image information. One is to process and convert the relevant information in the depth map, and then use it as an additional supplementary channel to splice it with the RGB image. However, the resolution of the depth map and the RGB image may be inconsistent, and splicing the two may result in a loss of spatial resolution. This is not conducive to image detail recovery and edge preservation, and will lead to a decrease in algorithm accuracy. Another fusion method is to convert the depth map into a point cloud image, extract spatial features through a point cloud processing network, and then splice it with the image features. Point cloud data is highly robust to lighting changes and occlusion, and the point cloud processing network can also be flexibly combined with other types of network structures. Therefore, this fusion method has been widely used.

[0004] However, most applications of fusing depth and image information involve extracting features using the corresponding networks of the two modalities separately, then performing global or hierarchical concatenation, and finally supervising the network learning through a task loss. This training approach results in minimal interaction between the two networks, lacking sufficient flexibility and adaptability in the feature extraction process. The network may not fully learn how to adjust the processing of the original modal information based on heterogeneous input information. When faced with occlusion, illumination variations, and lack of texture, RGB information cannot fully describe object features. If the network cannot adequately adjust the extracted image features based on point cloud features, the model's performance in these complex situations will be limited. Therefore, the interaction between the corresponding networks of different modalities needs to be further considered to better integrate point cloud and image features for pose estimation. Furthermore, redundant information generated during the fusion process needs to be removed to avoid the introduction of noise. Therefore, to meet these two technical requirements, a 6D pose estimation method based on cross-modal reconstruction and self-supervised training is urgently needed. Summary of the Invention

[0005] The content of this application is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this application is not intended to identify key features or essential features of the technical solution for which protection is sought, nor is it intended to limit the scope of the technical solution for which protection is sought.

[0006] In response to the problems and shortcomings of the prior art, the present invention aims to provide a 6D pose estimation method based on cross-modal reconstruction self-supervised training, aiming to address the problems of background interference, object occlusion, and poor fusion of different modalities in existing 6D pose estimation methods. By introducing a self-supervised algorithm that actively masks image information and reconstructs occlusion information across modalities, information from different sensor modalities is effectively fused to improve the accuracy and robustness of pose estimation. A channel-selective fusion strategy is also used to remove redundant information and reduce noise interference. This is used to address the problems raised in the above-mentioned background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The present invention discloses a 6D pose estimation method based on cross-modal reconstruction self-supervised training, comprising the following steps:

[0009] Step 1: Get the scene graph and depth map, and use the detection anchor box to select and crop the target image and depth map containing the target object;

[0010] Step 2: Performing slice mask processing on the target image to obtain a mask image, and inputting the mask image and depth image into an encoder to extract image features and point cloud features;

[0011] Step 3: The image features and point cloud features are trained using a cross-modal cross-attention mechanism to obtain image-point cloud fusion features;

[0012] Step 4: Output pose estimation feature pairs based on the image point cloud fusion features, and reconstruct the image through a self-supervised training strategy;

[0013] Step 5: Introduce reconstruction loss and feature pair prediction loss to optimize neural network parameters and save the trained model weights;

[0014] Step 6: Load the trained model weights and combine them with the pose estimation feature pairs to output a 6D estimated pose through a pose decoder.

[0015] Preferably, step 3 comprises the following steps:

[0016] Step 3.1, extracting correlation features between the point cloud features and the image features according to the cross attention mechanism;

[0017] Step 3.2, selecting the relevant feature with the highest similarity to the point cloud feature and the image feature among the relevant features;

[0018] Step 3.3, concatenating and fusing the related features with the highest similarity into cross-modal features;

[0019] Step 3.4, sending the cross-modal features to a point cloud and image decoder to decode and restore point cloud data and image data;

[0020] Step 3.5: Integrate the point cloud data and image data to generate image point cloud fusion features.

[0021] Preferably, the step 3.1 further comprises the following steps:

[0022] Step 3.1.1, calculating the point cloud to image and image to point cloud attention heat maps based on the point cloud features and image features;

[0023] Step 3.1.2, multiplying the point cloud feature and the point cloud to image attention heat map to obtain the point cloud to image feature;

[0024] In step 3.1.3, the image feature and the image-to-point cloud attention heat map are multiplied to obtain the image-to-point cloud feature.

[0025] Preferably, the step 3.2 further comprises the following steps:

[0026] Step 3.2.1, calculating the cosine similarity between the point cloud to image, image to point cloud features, and image and point cloud features, and obtaining a depth similarity matrix respectively;

[0027] Step 3.2.2, calculating the mean of the similarity scores for each row in the depth similarity matrix using the channelized mean;

[0028] Step 3.2.3, extract the top K related features with the highest similarity scores in the mean.

[0029] Preferably, in step 3.3, the related features with the highest similarity are spliced and fused into cross-modal features, which are expressed as follows:

[0030] Extracting K related features that are most similar to the image-to-point cloud features and the point cloud features, and fusing them with the point cloud features;

[0031] K related features that are most similar to the point cloud to the image features are extracted and merged with the image features.

[0032] Preferably, in step 5, the reconstruction loss L Rec Expressed as,

[0033]

[0034] in, is the fully reconstructed image, X t is the original image before any masking is applied.

[0035] Preferably, the feature pair prediction loss function L Code Expressed as,

[0036]

[0037] Among them, w j is the importance weight of each binary code, represents the binary vertex code predicted at training step t, represents the true binary vertex code at training step t.

[0038] Preferably, after obtaining the 6D estimated pose through step 6, it is necessary to compare the 6D estimated pose with the true pose and calculate the error between them to optimize the accuracy of the estimation.

[0039] As a second aspect of the present application, the present invention further discloses an electronic device, comprising:

[0040] at least one processor, and a memory communicatively coupled to the at least one processor;

[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the above-mentioned 6D pose estimation method based on cross-modal reconstruction self-supervised training.

[0042] As a third aspect of the present application, the present invention also discloses a computer storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned 6D pose estimation method based on cross-modal reconstruction self-supervised training are implemented.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention provides a 6D pose estimation method based on cross-modal reconstruction and self-supervised training, which can effectively handle occluded environments, deeply fuse point cloud and image features, and improve the robustness and accuracy of 6D pose estimation. By combining image and point cloud data, the characteristics of the object can be understood from different angles and dimensions. When facing situations such as occlusion and insufficient lighting, multiple sensor information is used in combination, which has better robustness than single modality input. Using the self-supervised learning method, the model can be trained without a large amount of labeled data, which enhances the adaptability to different environments and conditions and reduces the cost of data collection and annotation. After partially masking the input image, the original image is reconstructed using the remaining image and point cloud. The reconstruction of the occluded part of the fused image can achieve self-supervision and promote two-way communication between different modality networks. The hierarchical cross-modal channel selection and cross-attention mechanism are adopted to effectively fuse the features of the image and point cloud, and improve the ability of feature expression. At the same time, a similar feature selection strategy is used. By optimizing the algorithm and model structure, the amount of calculation is reduced and the speed of training and reasoning is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings that constitute part of this application are used to provide a further understanding of this application and make other features, objects and advantages of this application more apparent. The illustrative embodiment drawings of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. In the drawings:

[0046] Figure 1 This is a block diagram showing the steps of the 6D pose estimation method according to an embodiment of the present invention;

[0047] Figure 2 6D pose estimation method according to an embodiment of the present invention;

[0048] Figure 3 This is a flowchart of the steps of fusion feature output in an embodiment of the present invention;

[0049] Figure 4 Flowchart of the steps of similarity feature fusion in an embodiment of the present invention;

[0050] Figure 5 This is a visualization effect diagram of the cropped scene graph and depth map in an embodiment of the present invention;

[0051] Figure 6 This is a visualization effect diagram of the scene graph, mask graph, and reconstruction graph in an embodiment of the present invention;

[0052] Figure 7 This is a visualization diagram of the 6D pose estimation result in an embodiment of the present invention;

[0053] Figure 8 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0055] It should also be noted that, for ease of description, only the parts relevant to the relevant invention are shown in the accompanying drawings. The embodiments and features in the embodiments of the present disclosure may be combined with each other unless they conflict. The present invention discloses a 6D pose estimation method based on cross-modal reconstruction and self-supervised training. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0056] Reference Figure 1 and Figure 2 As shown, the following steps are included:

[0057] Step 1: Get the scene graph and depth map, and use the detection anchor box to select and crop the target image and depth map containing the target object;

[0058] Step 2: Perform tile masking on the target image to obtain a mask image. The mask image and depth image are input into the encoder to extract image features and point cloud features.

[0059] Step 3: The image features and point cloud features are trained using a cross-modal attention mechanism to obtain image-point cloud fusion features;

[0060] Step 4: Output pose estimation feature pairs based on the image point cloud fusion features, and reconstruct the image through a self-supervised training strategy;

[0061] Step 5: Introduce reconstruction loss and feature pair prediction loss to optimize neural network parameters and save the trained model weights;

[0062] Step 6: Load the trained model weights and combine them with the pose estimation features to output the 6D estimated pose through the pose decoder.

[0063] For step 1, obtain the scene graph and depth map, which are taken by the depth camera. The depth camera can capture color images (RGB) and depth maps (Depth) at the same time. The scene graph is a conventional RGB image, and the depth map contains the distance information from each pixel in the target scene to the camera. At the same time, it is necessary to input the analogy of the scene graph to ensure accurate identification of the target object in the scene graph for subsequent work. Subsequently, the target detection algorithm is used to crop out the target object in the image, and the training weights are obtained using the target detection algorithm with better training effect using the target data set. Using the training weights and the target detection algorithm, the target object is identified and located in the scene graph. According to the target object detection results, the detection anchor frame is used to select a small range of images and depth data containing the target object, and the corresponding anchor frame coordinates are output. By cropping the scene graph and depth map obtained by the depth camera according to the anchor frame coordinates, the cropped target graph and depth map containing the target object can be obtained. The cropping results are as follows: Figure 5 This operation helps reduce the impact of background noise on subsequent processing, allowing the model to focus more on the characteristics of the target object. It also converts the cropped depth map into point cloud data, which can more intuitively represent the three-dimensional structure of the scene.

[0064] Then, the target image after the anchor frame selection and cropping is sliced, that is, the target image is divided into Image patches of size n×n are randomly discarded to form a mask, and the image data after discarding some of the image patches is reassembled into an image with a height of H and a width of W. The missing information is replaced with black RGB values, thus generating a masked target image, which provides content for subsequent self-supervised learning and reconstruction of the lost information. Specifically, the target image is divided into 10×10 image patches, which are randomly discarded with a probability of 60%, and the remaining image patches are reassembled to the size of the original target image as input. By randomly discarding image information, the feature extraction network is able to ignore background noise and focus more on target-related information.

[0065] The cropped depth map is then converted into point cloud data. Point cloud data is a high-precision digital representation of the surface geometry and attribute information of an object in the form of discrete points in three-dimensional space. It is represented as a set of points in three-dimensional space, each containing position information (x, y, z). RandLA-Net is used as the point cloud encoder to obtain the point cloud feature f pRandLA-Net is an efficient and lightweight neural network architecture that can directly infer the semantics of each point in large-scale point clouds. RandLA-Net significantly improves model efficiency and reduces computing resource consumption by replacing time-consuming sampling methods such as FPS with random sampling. The ConvNext model is used as an image encoder to process the masked target image to obtain the image feature f r The ConvNeXt model is a pure convolutional neural network, while the ConvNext model can process masked information in the input image. It outperforms the Transformer model on various tasks while retaining its simplicity and efficiency.

[0066] Reference Figure 3 and Figure 4 As described in step 3, the image feature and point cloud feature input model is trained using a cross-modal cross-attention mechanism to obtain fusion features, further including the following steps:

[0067] Step 3.1, extract the correlation features between point cloud features and image features based on the cross attention mechanism;

[0068] Step 3.2: Select the relevant features with the highest similarity to the point cloud features and image features;

[0069] Step 3.3: Concatenate and fuse the relevant features with the highest similarity into cross-modal features;

[0070] Step 3.4: Send the cross-modal features to the point cloud and image decoder to decode and restore the point cloud data and image data;

[0071] Step 3.5: Integrate point cloud data and image data to generate image point cloud fusion features.

[0072] Specifically, the key information that can characterize each modality is extracted from the encoded point cloud features and image features, and the relevant features of the two are found based on this information. Among them, the image feature descriptor includes information such as color, texture, edge, shape, position and relative position; the point cloud feature descriptor includes information such as point position, normal, curvature, spatial relationship between points, surface shape, and three-dimensional structure of the object. The specific steps include:

[0073] Step 3.1.1: Calculate the point cloud to image and image to point cloud attention heatmaps based on the point cloud features and image features.

[0074] Step 3.1.2, multiply the point cloud features and the point cloud to image attention heat map to obtain the point cloud to image features;

[0075] In step 3.1.3, the image features and the image-to-point cloud attention heatmap are multiplied to obtain the image-to-point cloud features.

[0076] The cross-modal attention mechanism is used to find relevant features between the two modalities (point cloud and image). This paper does not directly use the common features of the two modalities as the fusion output, but adopts a novel hierarchical similarity feature search method based on this common feature to select more representative and more similar heterogeneous information for fusion. First, the attention heat map between the two modalities is calculated, which is the point cloud to image attention heat map H p→r And image to point cloud attention heat map H r→p , expressed as:

[0077]

[0078] Among them, W Q Used to map point cloud or image features to the query space for similarity calculation. k Used to map point cloud or image features to key space and match with query vector. V Used to map point cloud or image features into a value space, where the values are weighted according to the attention scores. k is the scaling factor used to stabilize the gradient during training, d k With W k The dimensions are the same. Softmax is used to convert similarity scores into a probability distribution, representing the relative importance of different features. Through the attention mechanism, the model can learn the associations between point cloud and image features, understanding which point cloud features are associated with which areas in the image. Attention heatmaps can be used to enhance or suppress certain features, allowing the model to focus more on information that is useful for the task.

[0079] In multimodal learning, this attention mechanism helps to effectively integrate information from different modalities. After obtaining the attention heat map, the image feature f r , point cloud features f p Multiplying with the attention heat map, we can get the transformed related features, which are expressed as:

[0080] f p→r =f p *H p→r ,

[0081] f r→p =f r *H r→p ;

[0082] Among them, f p→r Represents the transformation from point cloud to image features. This transformation can make the point cloud feature f pAccording to the image feature f r The importance of f is weighted to obtain a richer point cloud feature containing image features. r→p Represents the transformed feature from image to point cloud. This transformation can make the image feature f r According to the point cloud feature f p The importance of is weighted to obtain richer image features that include point cloud features. Since the attention mechanism can identify the most relevant parts of the task in different modalities, the feature f obtained after transformation p→r and f r→p This achieves the fusion and enhancement of cross-modal features. This transformation combines the attention heatmap with the original features through multiplication, so that the feature representation can focus more on information that is useful for the task while suppressing unimportant information.

[0083] After getting the relevant features f p→r and f r→p Finally, through hierarchical similarity feature selection, the features with the highest similarity to the original modality in the relevant features are searched and selected, thereby enriching the point cloud and image data. Different from the existing method, the existing method directly uses the relevant features f obtained by attention similarity weights to obtain the most similar features. p→r and f r→p With the original feature f r and f p Splicing is performed without more detailed differentiation and evaluation. It is easy to cause features with large similarity differences between the converted features and the original features to be spliced into the modal features and then input into the next layer of feature extractors, resulting in the introduction of noise from heterogeneous modalities, which damages the purity of the original features, makes the features processed by the network more noisy, and reduces the effectiveness of the network's extracted features. Therefore, the hierarchical similarity feature selection proposed in the present invention is very important for multimodal feature fusion in 6D pose estimation. Specifically, it includes the following steps:

[0084] Step 3.2.1, calculate the cosine similarity between point cloud to image, image to point cloud features, and image and point cloud features, and obtain the deep similarity matrix respectively;

[0085] In step 3.2.2, the mean of the similarity scores is calculated using the channelized mean for each row in the depth similarity matrix.

[0086] In step 3.2.3, extract the top K relevant features with the highest similarity scores in the mean.

[0087] Specifically, the hierarchical similarity feature selection in the embodiment of the present invention is performed by calculating the converted feature f p→r and f r→p With the original feature f r and f pThe similarity between the two modalities is calculated to generate a deeper similarity consideration matrix. Then, by comparing the elements in this matrix, the features with the highest similarity to the original modal features are selected. These selected features are then further fused with the original modal features to generate the final cross-modal feature representation. This method can effectively improve the expressiveness of features because it not only considers the similarity between features through the attention score matrix, but also optimizes the selection of features through a hierarchical cosine similarity matrix. This method can also be used for multimodal data fusion to improve data expressiveness and application effects.

[0088] First, according to the point cloud to image feature f p→r And image to point cloud features f r→p Calculate image features f r , point cloud features f p With point cloud to image feature f p→r And image to point cloud features f r→p The cosine similarity between them is used to obtain the depth similarity matrix. The depth similarity matrix is not simply a link between the point cloud features f p and image features f r , but link the point cloud to the image features f p→r With image feature f r , image to point cloud features f r→p With point cloud features f p , is a deeper level of similarity consideration. After obtaining the similarity matrix, we can measure the point cloud to image feature f p→r And image to point cloud features f r→p and the fused image feature f r , point cloud features f p The correlation between the point cloud and the image is obtained by the correlation matrix S p→r And the image to point cloud correlation matrix S r→p Representative, the specific process is as follows:

[0089]

[0090] Each element of the correlation matrix represents the point cloud to image feature f p→r , image to point cloud features f r→p and image features f r , point cloud features f p For each row in the similarity matrix, the channelized mean calculation method is used to calculate the mean of all similarity scores in the row. This mean reflects the average similarity between the relevant features and the original features. Specifically, if S is the similarity matrix, then the mean M of the i-th channel is i It can be expressed as Among them, Sij Represents the similarity between the relevant feature i and the original feature j, and n is the number of dimensions of the feature. The top K most similar relevant features in the mean are selected to represent the transformed features, that is, the channels with cosine similarity closer to 1 are denoted as TopK(S r→p ,K) and TopK(S p→r ,K). If the feature channelized mean similarity is not in TopK(S r→p ,K) or TopK(S p→r ,K), then discard it. If the feature channelization mean similarity is within TopK(S r→p ,K) or TopK(S p→r ,K), then extract the most similar K features and image features f respectively r and point cloud features f p Stitching and fusion to enhance the expressive power of image data.

[0091] Furthermore, extracting image to point cloud features f r→p and point cloud features f p The K features with the highest correlation. The channel numbers of these K features are related to the image to point cloud correlation matrix S r→p Before

[0092] K similarity features TopK(S r→p ,K) number is the same. Extract image to point cloud features f r→p and point cloud features f p The most similar K features and point cloud features f p Splicing and fusion are used to enhance the expression ability of point cloud data and form the final output feature. The output is represented as follows:

[0093] f poutput =Concat(f p ,f r→p (n)),n=TopK(S r→p ,K),

[0094] The spliced feature vector contains point cloud features that enhance the image expression ability, which can improve the expression ability of two-dimensional image data and make it contain more geometric information. Similarly, extracting point cloud conversion image features f p→r In the image f r The K most correlated features, the channel numbers of these K features and the correlation matrix S p→r Before

[0095] K similarity features TopK(S p→r ,K) number is the same. Extract f p→r Zhong and f r The K most similar features and image features fr The splicing and fusion are performed to enhance the expressive power of the image data and form the final output feature. The output is represented as follows:

[0096] f routput =Concat(f r ,f p→r (n)),n=TopK(S p→r ,K);

[0097] The spliced feature vector contains the geometric information of the point cloud features and the texture information related to the image, which can improve the expression ability of the point cloud data and make it contain more visual information. poutput and f routput The fused cross-modal features are then fed into the image decoder and point cloud decoder, respectively, to decode and restore the point cloud and image data. The decoded point cloud and image data are then integrated to generate an image-point cloud fusion feature that incorporates information from both modalities. This improves the data's expressiveness, enabling it to more comprehensively reflect the characteristics of the original scene. This image-point cloud fusion feature is then used for subsequent image reconstruction and pose estimation matching pair calculations. This is a critical step in the point cloud and image fusion process, ensuring that the fused features can be effectively converted back into data from both modalities, guaranteeing the information validity of the image-point cloud fusion feature.

[0098] The previously processed data, including fusion features, point cloud features, and image features, are used to calculate the output pose estimation feature pair. The output pose estimation feature pair is the key information for the subsequent calculation of the target 6D pose, because it involves extracting feature information that can represent the target pose from the model. The self-supervised training strategy is used to reconstruct the image, which helps the model reduce its dependence on labeled data. At the same time, the cross-modal point cloud reconstruction promotes the network model's in-depth understanding of the two models. The graphics are partially obscured and the reconstruction visualization results are as follows. Figure 6 shown.

[0099] Reconstruction loss and feature pair prediction loss are introduced to optimize the neural network parameters and save the model weights after training. Rec The difference between the model prediction output and the true value is expressed as,

[0100]

[0101] The mean square error (MSE) loss is used to evaluate the quality of the reconstruction, where is the fully reconstructed image, X t is the original image before any masking is applied. And the feature pair prediction loss L CodeWe focus on the prediction accuracy of the model for feature pairs, and use the loss function based on Hamming distance to calculate the hierarchical binary prediction loss according to the pairing results.

[0102]

[0103] Among them, w j is the importance weight of each binary code, represents the binary vertex code predicted at training step t; represents the true binary vertex code at training step t. λ is a constant used to balance the difference between the current step and the previous step. avg represents the average operation of all pixels within the predicted object mask. Using these two loss functions, the model can learn more accurate feature representations, thereby improving its ability to estimate the target 6D pose, ensuring that the model can learn from the data and continuously improve its prediction performance. Saving the model weights ensures that the model maintains its predictive power and accuracy when faced with new data, while also facilitating model deployment and application.

[0104] Finally, the trained model weights are loaded and combined with the pose estimation feature pairs to output the 6D estimated pose through the pose prediction decoder. The pose estimation feature pairs usually include information such as key points, edges, corner points, etc. of the object in the image. This information can help the system understand and identify the position and orientation of the object in space. By analyzing these feature pairs, the system can more accurately calculate the 6D pose of the target object, that is, the position and posture of the object in three-dimensional space, which usually includes three position coordinates (x, y, z) and three rotation angles (usually expressed as Euler angles or quaternions). The output 6D pose estimation result can be represented by the projection of the point cloud on the input two-dimensional image after being transformed by the pose matrix, where the red projection represents the projection of the point cloud on the two-dimensional image after being transformed by the pose estimation matrix. The specific visualization results are as follows. Figure 7 Finally, the system compares the estimated pose with the true pose to determine the accuracy of the estimate. The error calculation can help the system understand the performance of the current model and provide a basis for further optimization.

[0105] In order to implement the above embodiment, the present application also discloses an electronic device. Figure 8As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0106] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 8 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0107] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer storage medium, and the computer program includes program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network via the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0108] It should be noted that the computer storage medium described above in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0109] In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0110] The computer storage medium may be included in the electronic device or may exist independently without being incorporated into the electronic device. The computer storage medium carries one or more programs that, when executed by the electronic device, enable the electronic device to implement a 6D pose estimation method based on cross-modal reconstruction self-supervised training.

[0111] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0112] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the part of the module, program segment or code includes one or more executable instructions for realizing the logical function of the specification. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings.

[0113] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0114] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A 6D pose estimation method based on cross-modal reconstruction self-supervised training, characterized in that: The steps include: Step 1: Get the scene graph and depth map, and use the detection anchor box to select and crop the target image and depth map containing the target object; Step 2: Performing slice mask processing on the target image to obtain a mask image, and inputting the mask image and depth image into an encoder to extract image features and point cloud features; Step 3: The image features and point cloud features are trained using a cross-modal cross-attention mechanism to obtain image-point cloud fusion features; Step 4: Output pose estimation feature pairs based on the image point cloud fusion features, and reconstruct the image through a self-supervised training strategy; Step 5: Introduce reconstruction loss and feature pair prediction loss to optimize neural network parameters and save the trained model weights; Step 6: Load the trained model weights and combine them with the pose estimation feature pairs to output a 6D estimated pose through a pose decoder.

2. A 6D pose estimation method based on cross-modal reconstruction self-supervised training according to claim 1, characterized in that The step 3 comprises the following steps: Step 3.1, extracting correlation features between the point cloud features and the image features according to the cross attention mechanism; Step 3.2, selecting the relevant feature with the highest similarity to the point cloud feature and the image feature among the relevant features; Step 3.3, concatenating and fusing the related features with the highest similarity into cross-modal features; Step 3.4, sending the cross-modal features to a point cloud and image decoder to decode and restore point cloud data and image data; Step 3.5: Integrate the point cloud data and image data to generate image point cloud fusion features.

3. A 6D pose estimation method based on cross-modal reconstruction self-supervised training according to claim 2, characterized in that The step 3.1 further comprises the following steps: Step 3.1.1, calculating the point cloud to image and image to point cloud attention heatmaps based on the point cloud features and the image features; Step 3.1.2, multiplying the point cloud features and the point cloud to image attention heatmaps to obtain the point cloud to image features; In step 3.1.3, the image feature and the image-to-point cloud attention heat map are multiplied to obtain the image-to-point cloud feature.

4. A 6D pose estimation method based on cross-modal reconstruction self-supervised training according to claim 3, characterized in that The step 3.2 further comprises the following steps: Step 3.2.1, calculating the cosine similarity between the point cloud to image, image to point cloud features, and image and point cloud features, and obtaining a depth similarity matrix respectively; Step 3.2.2, calculating the mean of the similarity scores for each row in the depth similarity matrix using the channelized mean; Step 3.2.3, extract the top K related features with the highest similarity scores in the mean.

5. A 6D pose estimation method based on cross-modal reconstruction self-supervised training according to claim 4, characterized in that: In step 3.3, the related features with the highest similarity are spliced and fused into cross-modal features, which are expressed as follows: Extracting K related features that are most similar to the image-to-point cloud features and the point cloud features, and fusing them with the point cloud features; K related features that are most similar to the point cloud to the image features are extracted and merged with the image features.

6. The 6D pose estimation method based on cross-modal reconstruction and self-supervised training according to claim 1, characterized in that: The reconstruction loss L in step 5 is Rec Expressed as, in, is the fully reconstructed image, X t is the original image before any masking is applied.

7. The 6D pose estimation method based on cross-modal reconstruction and self-supervised training according to claim 6, characterized in that: The feature pair prediction loss function L in step 5 Code Expressed as, Among them, w j is the importance weight of each binary code, represents the binary vertex code predicted at training step t, represents the true binary vertex code at training step t.

8. The 6D pose estimation method based on cross-modal reconstruction and self-supervised training according to claim 5, characterized in that: After obtaining the 6D estimated pose through step 6, it is necessary to compare the 6D estimated pose with the true pose and calculate the error between them to optimize the accuracy of the estimation.

9. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 8.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps according to any one of claims 1 to 8 are implemented.

Images