A 6-dof pose estimation method and system for reflective and transparent objects

Abstract

The application discloses a 6-DOF pose estimation method and system for reflecting and transparent objects, and belongs to the technical field of intelligent robot environment perception, wherein high-frequency contour features and semantic context features are extracted through a depth feature extraction model to generate depth image fusion features, a matching cost volume is constructed, attention mixed filtering and a recurrent neural network are adopted to perform cyclic fine adjustment on parallax, high-reliability target depth images are obtained, and RGB depth fusion feature maps are obtained by aligning RGB images, thereby avoiding depth holes and distortion on the surface of high-reflecting and transparent objects. Candidate coarse poses are quickly obtained through uniform sampling and three-dimensional structure model matching, then the model is cut and rendered for each candidate coarse pose, pose residual errors between model pose feature maps and actual pose feature maps are calculated, and the pose is iteratively updated until convergence, so that local texture ambiguity caused by high reflection is effectively eliminated, and high-precision pose estimation for weak texture and transparent objects is realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot environmental perception technology, specifically relating to a 6-DOF pose estimation method and system for facing reflective and transparent objects. Background Technology

[0002] Six-DOF pose estimation is a core technology in computer vision and robotic environmental perception, widely used in industrial grasping, autonomous driving, and augmented reality. Its goal is to determine the position (translation vector T) and orientation (rotation matrix R) of a target object in three-dimensional space. Existing pose estimation methods are mainly divided into three categories: The first category is correspondence-based methods, including traditional feature point methods (such as SIFT, ORB combined with PnP) and dense correspondence methods (such as RBP, CDPN). These methods are highly dependent on the richness of the object's surface texture, making it difficult to extract and match stable feature points for low-texture or textureless objects (such as highly reflective metal parts, transparent glassware). The second category is direct regression-based methods (such as PoseNet), which use neural networks to directly regress pose parameters from images. However, their generalization ability is weak, the weights of rotation and translation loss functions are difficult to balance, and their accuracy is usually lower than that of geometric correspondence methods. The third category is iterative optimization-based methods (such as DeepIM, FoundationPose), which improve accuracy through iterative optimization of the initial pose, but are extremely dependent on depth information provided by depth sensors. However, existing consumer-grade and industrial-grade depth cameras (such as structured light and binocular infrared) exhibit severe depth holes, reflection distortion, or penetration on highly reflective planes or transparent surfaces, leading to unreliable depth data and consequently causing drift or even failure of iteratively optimized pose results. For highly reflective and transparent objects, existing technologies have attempted depth completion (such as ClearGrasp and TranspareNet), monocular depth estimation (such as DepthAnything), and multimodal fusion and synthetic data training (such as BlenderProc and Keypose). However, these methods generally suffer from drawbacks such as slow inference speed, poor generalization ability, high sensitivity to training data distribution, and difficulty in simulating complex real-world lighting environments, and cannot yet meet the high precision and robustness requirements of real-time robotic operations and other practical applications.

[0003] As mentioned above, how to provide a 6-DOF pose estimation method and system for reflective and transparent objects that can improve the reliability and accuracy of pose inference has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] The purpose of this invention is to provide a 6-DOF pose estimation method and system for facing reflective and transparent objects, in order to solve the above-mentioned problems existing in the prior art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a 6-DOF pose estimation method for facing reflective and transparent objects, including: Acquire the RGB image of the target object and the original binocular disparity image pair, and perform image correction processing on the original binocular disparity image pair to obtain the binocular disparity image pair of the target object; A preset depth feature extraction model is obtained. The binocular disparity image pair of the target object is used as input to the depth feature extraction model. The high-frequency contour features and semantic context features of the binocular disparity image pair are extracted by the depth feature extraction model. The high-frequency contour features and the semantic context features are concatenated to output the depth image fusion features of the target object. A matching cost body is constructed based on the depth image fusion features of the target object. Based on the attention-based hybrid filtering mechanism, the matching cost volume is filtered, and the disparity of the filtered matching cost volume is adjusted cyclically using a recurrent neural network to obtain the target depth map of the target object. The RGB image of the target object and the target depth map of the target object are aligned to obtain the RGB depth fusion feature map of the target object. A three-dimensional structural model of the target object is obtained. The three-dimensional structural model of the target object is uniformly sampled and rendered to obtain model binocular disparity image pairs from multiple perspectives. The model binocular disparity image pairs are similar to the RGB depth fusion feature map of the target object to filter out multiple candidate coarse poses. Each candidate coarse pose is used to simultaneously crop the model binocular disparity image pairs and the RGB depth fusion feature map to obtain fine model binocular disparity image pairs and fine target depth map. The model pose feature map and the actual pose feature map are extracted from the fine model binocular disparity image pair and the fine target depth map, respectively. The pose residual between the model pose feature map and the actual pose feature map is calculated. Each candidate coarse pose is updated with the pose residual until the pose residual converges, and the 6-DOF pose of the target object is output.

[0006] In one possible design, an RGB image of the target object and a pair of original binocular disparity images are acquired, and image correction processing is performed on the original binocular disparity image pair to obtain a pair of binocular disparity images of the target object, including: The RGB image of the target object is acquired in real time by an RGB camera, and the original binocular parallax image pair of the target object is acquired in real time by a binocular depth camera. The RGB image and the original binocular parallax image pair are then time-synchronized to obtain the synchronized RGB image and the original binocular parallax image pair. A preset epipolar correction model is obtained, and the original binocular disparity image pair after synchronous processing is used as input to the epipolar correction model so as to perform epipolar correction on the original binocular disparity image pair to obtain the binocular disparity image pair of the target object.

[0007] In one possible design, the deep feature extraction model includes an input layer, a first branch feature processing layer, a second branch feature processing layer, a feature fusion layer, and an output layer, wherein: The input layer is used to receive binocular parallax image pairs as input, and to pass the binocular parallax image pairs to the first branch feature processing layer and the second branch feature processing layer respectively. The first branch feature processing layer is used to identify edge region information, corner region information and texture change region information of the target object from the binocular parallax image pair, so as to complete the extraction of geometric structure feature information of the target object and output the high-frequency contour features of the target object. The second branch feature processing layer is used to identify the object category information, local and global relationship information and occlusion and occlusion relationship information of the target object from the binocular parallax image pair, so as to complete the extraction of the target object association feature information and output the semantic context features of the target object. The feature fusion layer is used to receive the high-frequency contour features output by the first branch feature processing layer and the semantic context features output by the second branch feature processing layer, and to perform feature concatenation on the high-frequency contour features and the semantic context features in the feature dimension to obtain the depth image fusion features. The output layer is used to receive the depth image fusion features output by the feature fusion layer and output the depth image fusion features.

[0008] In one possible design, the binocular disparity image pair of the target object is used as input to the depth feature extraction model. The depth feature extraction model extracts high-frequency contour features and semantic context features of the binocular disparity image pair. The high-frequency contour features and the semantic context features are concatenated to output the depth image fusion features of the target object. A matching cost body is constructed based on the depth image fusion features of the target object, including: The binocular disparity image pairs of the target object are used as inputs and fed into the input layer of the depth feature extraction model. The binocular disparity image pair is obtained from the input layer of the deep feature extraction model through the first branch feature processing layer of the deep feature extraction model, and the binocular disparity image pair is convolved using the first branch feature processing layer of the deep feature extraction model to extract the high-frequency contour features of the target object from the binocular disparity image pair. The binocular disparity image pair is obtained from the input layer of the deep feature extraction model through the second branch feature processing layer of the deep feature extraction model, so as to perform semantic recognition on the binocular disparity image pair using the second branch feature processing layer of the deep feature extraction model, and to extract the semantic context features of the target object from the binocular disparity image pair. The high-frequency contour features are extracted from the first branch feature processing layer of the deep feature extraction model through the feature fusion layer of the deep feature extraction model, and the semantic context features are extracted from the second branch feature processing layer of the deep feature extraction model. The semantic context features are linearly upsampled, and the upsampled semantic context features and the high-frequency contour features are concatenated to obtain the depth image fusion features of the target object. The depth image fusion features are extracted from the feature fusion layer of the depth feature extraction model through the output layer of the depth feature extraction model, and the depth image fusion features of the target object are used as the output of the depth feature extraction model for output. Obtain a preset desired depth range, calculate the maximum disparity using the depth image fusion features of the target object and the desired depth range, generate a corresponding disparity search range based on the maximum disparity, and search for each pixel of the binocular disparity image pair within the disparity search range to form multiple disparities; Based on the depth image fusion features of the target object, binocular feature similarity is calculated for each disparity to obtain the fusion feature similarity corresponding to each pixel position; The fused feature similarity of each pixel location is used as the matching similarity of each pixel location between the binocular disparity image pairs. A corresponding two-dimensional similarity map is constructed for each pixel location, and the two-dimensional similarity maps of each pixel location are stacked in the depth direction to construct a matching cost body. Each element in the matching cost body represents the degree of disparity matching between the binocular disparity image pairs.

[0009] In one possible design, the matching cost volume is filtered based on an attention-based hybrid filtering mechanism, and a recurrent neural network is used to perform disparity cyclic adjustment on the filtered matching cost volume to obtain a target depth map of the target object. The RGB image and target depth map of the target object are then aligned to obtain an RGB depth fusion feature map of the target object, including: The matching cost volume is subjected to planar convolution filtering and axial convolution filtering to obtain planar convolutional three-dimensional feature tensors and axial convolutional three-dimensional feature tensors. The planar convolutional 3D feature tensor and the axial convolutional 3D feature tensor are subjected to feature concatenation and dimensionality reduction to obtain a filtered matching cost body as the current matching cost body. A preset recurrent neural network is obtained, and the current matching cost body is input into the recurrent neural network to generate a corresponding disparity correction amount for the current matching cost body using the recurrent neural network. The disparity correction amount is then used to update the disparity of each pixel position in the current matching cost body to obtain the updated matching cost body. The updated matching cost body is used as the current matching cost body, and the recurrent neural network is used to update the current matching cost body in multiple rounds until the disparity correction amount converges to obtain the optimal matching cost body. The final disparity value of each pixel position is extracted using the optimal matching cost body, and a target depth map of the target object is generated based on the final disparity value of each pixel position. Obtain the preset extrinsic transformation matrix, perform three-dimensional back projection on each depth pixel in the target depth map of the target object, obtain the three-dimensional coordinates of each depth pixel in the depth camera coordinate system, and use the extrinsic transformation matrix to transform the three-dimensional coordinates of each depth pixel in the depth camera coordinate system to the RGB camera coordinate system to obtain the three-dimensional coordinates of each depth pixel in the RGB camera coordinate system. Obtain the intrinsic parameter matrix of the RGB camera. Using the intrinsic parameter matrix of the RGB camera, reproject the three-dimensional coordinates of each depth pixel in the RGB camera coordinate system onto the RGB image plane to obtain the RGB image plane pixel coordinates corresponding to each three-dimensional coordinate. Match each pixel in the RGB image of the target object with the RGB image plane pixel coordinates corresponding to each three-dimensional coordinate reprojected onto the RGB image plane to complete the alignment of the target depth map of the target object with the RGB image, and obtain the RGB depth fusion feature map of the target object.

[0010] In one possible design, a 3D structural model of the target object is acquired. This model is then uniformly sampled and rendered to obtain binocular disparity image pairs from multiple viewpoints. The binocular disparity image pairs are then matched with the RGB depth fusion feature map of the target object to filter out multiple candidate coarse poses. Each candidate coarse pose is then used to simultaneously crop the binocular disparity image pairs and the RGB depth fusion feature map, resulting in refined binocular disparity image pairs and a refined target depth map, including: A three-dimensional structural model of the target object is obtained, and the three-dimensional structural model of the target object is uniformly sampled and rendered to generate a two-dimensional rendering image from multiple perspectives. The two-dimensional rendering images are then integrated to form a binocular parallax image pair of the three-dimensional structural model. The depth feature extraction model is used to extract the corresponding model depth image fusion feature map from the model binocular parallax image pair; A preset feature similarity threshold is obtained, and feature similarity is calculated between the model depth image fusion feature map and each observed feature in the RGB depth fusion feature map of the target object, so as to match each observed feature with a feature similarity higher than the feature similarity threshold from the RGB depth fusion feature map as a candidate coarse pose. Based on the candidate coarse pose, the three-dimensional structure model is rendered a second time to set the virtual observation view of the three-dimensional structure model to the view corresponding to the candidate coarse pose. The model's binocular disparity image pair is then cropped using the virtual observation view after the second rendering to obtain a fine model binocular disparity image pair. Finally, the RGB depth fusion feature map is cropped using the actual observation view corresponding to the candidate coarse pose to obtain a fine target depth map.

[0011] In one possible design, model pose feature maps and actual pose feature maps are extracted from the fine model binocular disparity image pair and the fine target depth map, respectively. The pose residual between the model pose feature map and the actual pose feature map is calculated. Each candidate coarse pose is updated using the pose residual until the pose residual converges, and the 6-DOF pose of the target object is output, including: A preset cross-attention feature extraction model is obtained. The fine model binocular disparity image pair and the fine target depth map are flattened in the spatial dimension, respectively. The flattened fine model binocular disparity image pair and the flattened fine target depth map are input into the cross-attention feature extraction model. The cross-attention feature extraction model is used to extract features from the flattened fine model binocular disparity image pair and the flattened fine target depth map to obtain the model observation feature map and the actual observation feature map. Using the actual observed feature map as the query and the model observed feature map as the key and value, the observation attention weight matrix is ​​calculated. Based on the observation attention weight matrix, the actual observed feature map and the model observed feature map are concatenated into a difference feature tensor, and linear regression is performed on the difference feature tensor to obtain the pose residual between the model pose feature map and the actual pose feature map. The pose residual is subjected to exponential mapping to obtain the pose transformation matrix. The pose transformation matrix is ​​then used to update each candidate coarse pose to obtain the updated candidate coarse pose. The updated candidate coarse pose is used as the current candidate coarse pose. Based on the current candidate coarse pose, the model binocular disparity image pair and the target depth map are simultaneously cropped to obtain the current fine model binocular disparity image pair and the current fine target depth map. The current model pose feature map and the current actual pose feature map are extracted from the current fine model binocular disparity image pair and the current fine target depth map, respectively. The pose residual between the current model pose feature map and the current actual pose feature map is calculated as the current pose residual, and the current candidate coarse pose is updated using the current pose residual. Obtain the preset pose residual convergence condition, perform multiple rounds of iterative updates on the current candidate coarse pose until the pose residual convergence condition is met, then consider the pose residual to be converged, and obtain the current candidate coarse pose updated in the last round as the 6-DOF pose of the target object for output.

[0012] Secondly, the present invention provides a 6-DOF pose estimation system for facing reflective and transparent objects, comprising: The actual image acquisition unit is used to acquire the RGB image of the target object and the original binocular parallax image pair, and to perform image correction processing on the original binocular parallax image pair to obtain the binocular parallax image pair of the target object. The cost body construction unit is used to obtain a preset depth feature extraction model, take the binocular disparity image pair of the target object as input, input it into the depth feature extraction model, extract the high-frequency contour features and semantic context features of the binocular disparity image pair through the depth feature extraction model, concatenate the high-frequency contour features and the semantic context features to output the depth image fusion features of the target object, and construct a matching cost body based on the depth image fusion features of the target object. The depth map generation unit is used to filter the matching cost body based on the attention hybrid filtering mechanism, and to perform disparity cyclic adjustment on the filtered matching cost body using a recurrent neural network to obtain the target depth map of the target object. The RGB image of the target object and the target depth map of the target object are aligned to obtain the RGB depth fusion feature map of the target object. The 3D model rendering unit is used to acquire the 3D structural model of the target object, uniformly sample and render the 3D structural model of the target object to obtain model binocular disparity image pairs from multiple perspectives, perform similarity matching between the model binocular disparity image pairs and the RGB depth fusion feature map of the target object to filter out multiple candidate coarse poses, and use each candidate coarse pose to simultaneously crop the model binocular disparity image pairs and the target depth map to obtain fine model binocular disparity image pairs and fine target depth maps. The target pose generation unit is used to extract the model pose feature map and the actual pose feature map from the fine model binocular disparity image pair and the fine target depth map, respectively, calculate the pose residual between the model pose feature map and the actual pose feature map, update each candidate coarse pose with the pose residual until the pose residual converges, and output the 6-DOF pose of the target object.

[0013] Thirdly, the present invention provides an electronic device comprising a memory, a processor, and a transceiver connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the 6-DOF pose estimation method for facing reflective and transparent objects as described in the first aspect or any possible design of the first aspect.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the 6-DOF pose estimation method for facing reflective and transparent objects as described in the first aspect or any possible design of the first aspect.

[0015] Fifthly, the present invention provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform a 6-DOF pose estimation method for facing reflective and transparent objects as described in the first aspect or any possible design of the first aspect.

[0016] Beneficial Effects: This invention provides a 6-DOF pose estimation method and system for reflective and transparent objects. It extracts high-frequency contour features and semantic context features using a depth feature extraction model to generate depth image fusion features, thereby constructing a corresponding matching cost volume. Then, attention-based hybrid filtering and a recurrent neural network are used to iteratively fine-tune the disparity, obtaining a highly reliable target depth map. Finally, RGB images are aligned to obtain an RGB depth fusion feature map, avoiding depth holes and distortion on highly reflective and transparent object surfaces. Furthermore, a closed-loop iterative architecture of rendering clipping and iterative updates is employed. Candidate coarse poses are quickly obtained through uniform sampling and matching with a 3D structural model. Then, for each candidate coarse pose, the model is clipped and rendered. The pose residual between the model pose feature map and the actual pose feature map is calculated, and the pose is iteratively updated until convergence. This effectively eliminates local texture ambiguity caused by high reflectivity, achieving high-precision pose estimation for weakly textured and transparent objects. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the 6-DOF pose estimation method for facing reflective and transparent objects provided in an embodiment of the present invention. Figure 2A schematic diagram of the model structure of the deep feature extraction model provided in the embodiments of the present invention; Figure 3 This is a functional structure diagram of a 6-DOF pose estimation system for facing reflective and transparent objects provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0019] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0020] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0021] Example: like Figure 1 As shown, the first aspect of this embodiment provides a 6-DOF pose estimation method for facing reflective and transparent objects, which may include, but is not limited to, the following steps: S1. Acquire the RGB image of the target object and the original binocular disparity image pair, and perform image correction processing on the original binocular disparity image pair to obtain the binocular disparity image pair of the target object; In one possible implementation, step S1 involves acquiring an RGB image of the target object and an original binocular disparity image pair, and performing image correction processing on the original binocular disparity image pair to obtain a binocular disparity image pair of the target object. This step can be decomposed into, but is not limited to, the following steps S11-S12, specifically including: S11. Acquire RGB images of the target object in real time using an RGB camera, acquire original binocular parallax image pairs of the target object in real time using a binocular depth camera, and perform time synchronization processing on the RGB images and original binocular parallax image pairs to obtain synchronized RGB images and original binocular parallax image pairs. S12. Obtain a preset epipolar correction model, and input the original binocular disparity image pair after synchronous processing into the epipolar correction model to perform epipolar correction on the original binocular disparity image pair to obtain a binocular disparity image pair of the target object.

[0022] In specific application scenarios, intelligent robots can use independent RGB sensors as RGB cameras, binocular infrared sensors as binocular depth cameras, or directly use integrated intelligent depth cameras (such as Intel RealSense D455) as visual acquisition devices. When performing time synchronization processing on the RGB image and the original binocular parallax image pair, it is necessary to achieve image frame synchronization between the RGB camera and the binocular depth camera through hardware triggering or software API (such as Intel RealSense SDK) to ensure that a high-resolution RGB image (resolution, for example, 1920×1080 for acquiring the color appearance information of the target object) and a pair of original binocular parallax image pairs (left eye image and right eye image, resolution, for example, 1280×720, acquired by the left and right infrared sensors respectively, for subsequent depth calculation) are acquired at the same time.

[0023] The epipolar correction model is actually a pre-defined epipolar correction algorithm (such as Semi-Global Matching (SGM) or a deep learning-based epipolar correction network). Through this epipolar correction algorithm, the original binocular disparity image pairs can be epipolar corrected according to the intrinsic parameters (focal length, principal point coordinates) and extrinsic parameters (rotation matrix, translation vector, i.e., baseline distance) calibrated by the binocular depth camera at the factory. This ensures that the same spatial point in the left and right target images is strictly aligned on the image row coordinates, satisfying the epipolar constraint conditions and completing the epipolar correction.

[0024] S2. Obtain a preset depth feature extraction model, take the binocular disparity image pair of the target object as the input, input it into the depth feature extraction model, extract the high-frequency contour features and semantic context features of the binocular disparity image pair through the depth feature extraction model, concatenate the high-frequency contour features and the semantic context features to output the depth image fusion features of the target object, and construct a matching cost body based on the depth image fusion features of the target object; like Figure 2 As shown, in one possible implementation, in step S2, the deep feature extraction model includes an input layer, a first branch feature processing layer, a second branch feature processing layer, a feature fusion layer, and an output layer, wherein: The input layer is used to receive binocular parallax image pairs as input, and to pass the binocular parallax image pairs to the first branch feature processing layer and the second branch feature processing layer respectively. The first branch feature processing layer is used to identify edge region information, corner region information and texture change region information of the target object from the binocular parallax image pair, so as to complete the extraction of geometric structure feature information of the target object and output the high-frequency contour features of the target object. The second branch feature processing layer is used to identify the object category information, local and global relationship information and occlusion and occlusion relationship information of the target object from the binocular parallax image pair, so as to complete the extraction of the target object association feature information and output the semantic context features of the target object. The feature fusion layer is used to receive the high-frequency contour features output by the first branch feature processing layer and the semantic context features output by the second branch feature processing layer, and to perform feature concatenation on the high-frequency contour features and the semantic context features in the feature dimension to obtain the depth image fusion features. The output layer is used to receive the depth image fusion features output by the feature fusion layer and output the depth image fusion features.

[0025] In practical applications, the deep feature extraction model adopts a dual-branch parallel structure. The first branch feature processing layer (high-frequency contour feature extraction) uses a lightweight convolutional neural network (CNN, such as the first 5 layers of ResNet-18 or directly setting 3 convolutional layers). This convolutional neural network performs multiple convolution and pooling operations on the binocular parallax image pairs to extract high-frequency, detailed geometric structural features of the target object. It focuses on the edge regions, corner regions, and regions with drastic changes in object texture. Its output feature map includes rich local geometric structural information.

[0026] The second branch, the feature processing layer (semantic context feature extraction), employs a pre-trained monocular depth estimation model with frozen parameters. In selectable implementation scenarios, the open-source DepthAnything model (an Encoder-Decoder based on the Transformer architecture) is preferred. The DepthAnything model is trained on large-scale datasets with real depth labels and possesses strong scene understanding and semantic awareness capabilities. The binocular disparity image is input into the DepthAnything model without any gradient updates. This allows the DepthAnything model to output a low-resolution but highly semantic feature map. This feature map encodes contextual information such as the object category, local and global relationships, and occlusion / being-occluded relationships of the target object.

[0027] After parallel feature extraction is completed, the feature fusion layer collects corresponding high-frequency contour features and semantic context features from the two branch feature processing layers. The semantic context features are bilinearly upsampled to make their spatial resolution consistent with that of the high-frequency contour features. Then, the two are concatenated in the feature channel dimension to obtain the depth image fusion feature. The depth image fusion feature has both local geometric details and global contextual semantics, and has stronger robustness in dealing with the blurring of reflective surfaces of objects and specular reflection interference.

[0028] In one possible implementation, in step S2, the binocular disparity image pair of the target object is used as input to the depth feature extraction model. The depth feature extraction model extracts the high-frequency contour features and semantic context features of the binocular disparity image pair. The high-frequency contour features and the semantic context features are concatenated to output the depth image fusion features of the target object. A matching cost body is constructed based on the depth image fusion features of the target object. This can be decomposed into, but is not limited to, the following steps S21-S28, specifically including: S21. The binocular disparity image pairs of the target object are used as inputs and input to the input layer of the depth feature extraction model; S22. The binocular disparity image pair is obtained from the input layer of the deep feature extraction model through the first branch feature processing layer of the deep feature extraction model, and the binocular disparity image pair is convolved using the first branch feature processing layer of the deep feature extraction model to extract the high-frequency contour features of the binocular disparity image from the binocular disparity image pair. S23. The binocular disparity image pair is obtained from the input layer of the deep feature extraction model through the second branch feature processing layer of the deep feature extraction model, so as to perform semantic recognition on the binocular disparity image pair using the second branch feature processing layer of the deep feature extraction model, and to extract the semantic context features of the binocular disparity image from the binocular disparity image pair. S24. Through the feature fusion layer of the deep feature extraction model, the high-frequency contour features are extracted from the first branch feature processing layer of the deep feature extraction model, and the semantic context features are extracted from the second branch feature processing layer of the deep feature extraction model. The semantic context features are linearly upsampled, and the upsampled semantic context features and the high-frequency contour features are concatenated to obtain the deep image fusion features of the binocular parallax image. S25. Extract the depth image fusion features from the feature fusion layer of the depth feature extraction model through the output layer of the depth feature extraction model, and output the depth image fusion features of the binocular parallax image as the output of the depth feature extraction model; S26. Obtain a preset desired depth range, calculate the maximum disparity using the depth image fusion features of the target object and the desired depth range, generate a corresponding disparity search range based on the maximum disparity, and search for each pixel of the binocular disparity image pair within the disparity search range to form multiple disparities; S27. Based on the depth image fusion features of the target object, perform binocular feature similarity calculation for each disparity to obtain the fusion feature similarity corresponding to each pixel position; S28. The fusion feature similarity of each pixel location is used as the matching similarity of each pixel location between the binocular disparity image pairs. A corresponding two-dimensional similarity map is constructed for each pixel location, and the two-dimensional similarity maps of each pixel location are stacked in the depth direction to construct a matching cost body. Each element in the matching cost body represents the degree of disparity matching between the binocular disparity image pairs.

[0029] It should be noted that in the 6-DOF pose estimation method for reflective and transparent objects provided in this embodiment, the matching cost volume is constructed based on the depth image fusion features of the binocular disparity images. This requires calculating the fusion feature similarity per disparity. For each pixel location, a two-dimensional similarity map is constructed. In the depth image fusion features corresponding to the left eye image, the corresponding left feature vector is extracted for each pixel location. Correspondingly, in the depth image fusion features corresponding to the right eye image, due to epipolar constraints, corresponding points are located in the same row, with the horizontal coordinate offset by the disparity value. The corresponding right feature vector is extracted for each pixel location, and then the left feature vector is calculated. The fusion feature similarity between the feature vector and the right feature vector (calculated using dot product and inner product) is calculated. The larger this fusion feature similarity is, the more similar the binocular images are and the higher the matching degree. Finally, all the two-dimensional similarity maps are stacked in the depth direction (i.e., the disparity dimension) to form a three-dimensional feature tensor (the three-dimensional feature tensor is V, and its dimensions are H×W×D (where H is the image height, W is the image width, and D is the maximum disparity layer). Each element V(u,v,d) in the three-dimensional feature tensor V represents the matching degree of the spatial pixel (u,v) when the disparity is d). This three-dimensional feature tensor is the matching cost body.

[0030] For each pixel in the left-eye image, within a preset desired depth range on the same row of the right-eye image, pixel matching and cost aggregation are performed based on the pixel grayscale value to find a disparity value that minimizes the matching cost. To generate the disparity value that minimizes the matching cost for each pixel, the following formula is then used:

[0031] Calculate the depth value corresponding to each pixel. ,in, The focal length of the binocular depth camera. The baseline distance (i.e., the optical center distance between the binocular depth cameras) is used to generate a feature map with the same resolution as the binocular parallax image pair as the depth image fusion feature.

[0032] S3. Based on the attention hybrid filtering mechanism, the matching cost body is filtered, and the disparity of the filtered matching cost body is adjusted cyclically using a recurrent neural network to obtain the target depth map of the target object. The RGB image of the target object and the target depth map of the target object are aligned to obtain the RGB depth fusion feature map of the target object. In one possible implementation, in step S3, the matching cost volume is filtered based on an attention-based hybrid filtering mechanism, and a recurrent neural network is used to perform disparity cyclic adjustment on the filtered matching cost volume to obtain a target depth map of the target object. The RGB image and target depth map of the target object are then aligned to obtain an RGB depth fusion feature map of the target object. This can be, but is not limited to, decomposed into the following steps S31-S37, specifically including: S31. Perform planar convolution filtering and axial convolution filtering on the matching cost body to obtain the planar convolution three-dimensional feature tensor and the axial convolution three-dimensional feature tensor; S32. Perform feature concatenation and dimensionality reduction on the planar convolutional three-dimensional feature tensor and the axial convolutional three-dimensional feature tensor to obtain the filtered matching cost body as the current matching cost body; S33. Obtain a preset recurrent neural network, input the current matching cost body into the recurrent neural network, use the recurrent neural network to generate a corresponding disparity correction amount for the current matching cost body, and use the disparity correction amount to update the disparity of each pixel position in the current matching cost body to obtain the updated matching cost body. S34. The updated matching cost body is used as the current matching cost body, and the recurrent neural network is used to update the current matching cost body in multiple rounds until the disparity correction amount converges to obtain the optimal matching cost body; S35. Using the optimal matching cost body, extract the final disparity value of each pixel position, and generate a target depth map of the target object based on the final disparity value of each pixel position; S36. Obtain a preset extrinsic transformation matrix, perform three-dimensional back projection on each depth pixel in the target depth map of the target object to obtain the three-dimensional coordinates of each depth pixel in the depth camera coordinate system, and use the extrinsic transformation matrix to transform the three-dimensional coordinates of each depth pixel in the depth camera coordinate system to the RGB camera coordinate system to obtain the three-dimensional coordinates of each depth pixel in the RGB camera coordinate system. S37. Obtain the intrinsic parameter matrix of the RGB camera. Using the intrinsic parameter matrix of the RGB camera, reproject the three-dimensional coordinates of each depth pixel in the RGB camera coordinate system onto the RGB image plane to obtain the RGB image plane pixel coordinates corresponding to each three-dimensional coordinate. Match each pixel in the RGB image of the target object with the RGB image plane pixel coordinates corresponding to each three-dimensional coordinate reprojected onto the RGB image plane to complete the alignment of the target depth map of the target object with the RGB image, and obtain the RGB depth fusion feature map of the target object.

[0033] In specific application scenarios, the matching cost volume is filtered by the Disparity Transformer, and the current matching cost volume is sliced ​​(slicing the matching cost volume refers to two different cross sections of the three-dimensional feature tensor; one cross section can be understood as a two-dimensional similarity map of a pair of disparity images under disparity d, and the other cross section represents the similarity of a column of the matching cost volume under different disparities). At the same time, the matching cost volume is subjected to planar convolution filtering and axial convolution filtering.

[0034] Among them, the planar convolutional filtering process (i.e., spatial smoothing filtering) requires performing two-dimensional convolution on the matching cost volume in the spatial dimensions (H and W). For each disparity layer d (fixed), a spatial convolution kernel (e.g., 3×3 size) is used to slide on the two-dimensional image plane, which can be represented as:

[0035] in, This is the planar convolutional 3D feature tensor output after planar convolution processing. It can represent the spatial coordinates of the pixel position of the current operation. For the current parallax, The weight matrix is ​​a preset planar two-dimensional convolution kernel. Represents the matching cost body. The spatial radius of the predefined planar convolution kernel (e.g., for a 3×3 kernel, its...). (1) This represents the local coordinate offset of the planar 2D convolution kernel on the 2D image plane. This is the bias term for planar two-dimensional convolution.

[0036] Correspondingly, axial convolution filtering (i.e., disparity dimension filtering) requires performing a one-dimensional convolution on the matching cost volume along the disparity dimension D for each pixel position. Extracting a vector of length maximum disparity along the disparity direction and performing sliding convolution with a one-dimensional convolution kernel (e.g., length 3) can be represented as:

[0037] in, This is the axially convolutional 3D feature tensor output after axial convolution processing. The weight vector is a predefined one-dimensional convolution kernel along the axis. The receptive field radius in the parallax dimension of the preset axial convolution kernel (e.g., when the kernel size is 3, (1) This represents the local coordinate offset along the parallax axis (D-axis). This is the bias term for axial one-dimensional convolution.

[0038] Furthermore, by concatenating the planar convolutional 3D feature tensor output by the planar 2D convolution with the axial convolutional 3D feature tensor output by the axial 1D convolution in the channel dimension, and then performing dimensionality reduction through a 1×1 convolutional layer, the filtered matching cost body (current matching cost body) can be obtained. This hybrid filtering method of planar 2D filtering and axial 1D filtering makes the peak of the probability distribution sharper and suppresses multi-peak noise. After the two filters are executed together, the results are fused to remove the noise of the matching cost body. The matching cost body after this filtering process obviously has a more reliable matching metric.

[0039] It should be noted that the recurrent neural network actually employs a recurrent neural network (RNN) based on a convolutionally gated recurrent unit (ConvGRU) to iteratively optimize the filtered matching cost body, progressively fine-tuning the disparity value at each pixel position. This RNN is pre-trained on a synthetic dataset (such as various disparity map sequences rendered using Blender), with the training loss being the L2 loss between the predicted disparity and the true disparity. The input to the RNN is the filtered matching cost body, and the hidden state is initialized to a zero tensor. During the iterative update process, let's assume that for the nth iteration, the current matching cost body and the hidden state from the previous time step are compared... The state is input into a recurrent neural network. The recurrent neural network outputs the current disparity correction (the correction value at each pixel position, which can be positive or negative, used to adjust the current disparity estimate) and the updated hidden state. The disparity correction is used to update the current disparity estimate, resulting in an updated current disparity estimate. A differentiable projection operation is then performed on the updated current disparity estimate to update the sampling position of the current cost body (i.e., recalculate the matching cost), forming a new input for the next step. When the average disparity change over two consecutive steps is less than a preset convergence threshold (e.g., 0.05 pixels) or the maximum number of iterations is reached, the loop terminates, and the optimal matching cost body is obtained.

[0040] Furthermore, due to the fixed physical offset between the stereo depth camera and the RGB camera (their optical centers are not on the same axis), the coordinate system of the target depth map is inconsistent with the coordinate system of the RGB image. To obtain the true depth value corresponding to each RGB pixel, it is necessary to perform homogeneous coordinate transformation and reprojection on each depth pixel of the target depth map to align the depth of the target object with the color space. Specifically, firstly, each depth pixel of the target depth map is back-projected onto the three-dimensional coordinate space of the stereo depth camera. Using the intrinsic parameter matrix of the stereo depth camera, each depth pixel and its corresponding depth value in the target depth map are back-projected onto the three-dimensional coordinates in the depth camera coordinate system. Then, using the pre-calibrated extrinsic parameter transformation matrix from the stereo depth camera to the RGB camera, the three-dimensional coordinates in the depth camera coordinate system are transformed into the RGB camera coordinate system. Finally, using the intrinsic parameter matrix of the RGB camera (including focal length and principal point), the three-dimensional coordinates in the RGB camera coordinate system are projected onto the RGB image plane to obtain the corresponding RGB image plane pixel coordinates and generate the aligned target depth map. For each integer coordinate (u, v) in the RGB image, the corresponding depth value Z_rgb is found through the above projection mapping. Since the mapping result may be non-integer coordinates, bilinear interpolation or nearest neighbor interpolation is used to fill the coordinates, and finally an RGB image plane pixel with the same resolution as the RGB image is generated, and the depth value of each pixel is strictly aligned with the actual distance of the object surface corresponding to the pixel, thus completing the alignment process with the RGB image.

[0041] S4. Obtain the three-dimensional structural model of the target object, uniformly sample and render the three-dimensional structural model of the target object to obtain model binocular disparity image pairs from multiple perspectives, perform similarity matching between the model binocular disparity image pairs and the RGB depth fusion feature map of the target object to filter out multiple candidate coarse poses, and use each candidate coarse pose to simultaneously crop the model binocular disparity image pairs and the RGB depth fusion feature map to obtain fine model binocular disparity image pairs and fine target depth map; In one possible implementation, in step S4, a three-dimensional structural model of the target object is acquired, and the three-dimensional structural model of the target object is uniformly sampled and rendered to obtain model binocular disparity image pairs from multiple viewpoints. The model binocular disparity image pairs are then matched with the RGB depth fusion feature map of the target object for similarity to filter out multiple candidate coarse poses. Each candidate coarse pose is then used to simultaneously crop the model binocular disparity image pairs and the RGB depth fusion feature map to obtain fine model binocular disparity image pairs and a fine target depth map. This can be, but is not limited to, decomposed into the following steps S41-S44, specifically including: S41. Obtain the three-dimensional structural model of the target object, uniformly sample and render the three-dimensional structural model of the target object to generate a two-dimensional rendering image from multiple perspectives, and integrate the two-dimensional rendering images to form a binocular parallax image pair of the three-dimensional structural model. S42. Using the depth feature extraction model, extract the corresponding model depth image fusion features from the model binocular disparity image pair; S43. Obtain a preset feature similarity threshold, calculate the feature similarity between the model depth image fusion feature and each observation feature in the target depth map, and match each observation feature with a feature similarity higher than the feature similarity threshold from the target depth map as a candidate coarse pose; S44. Based on the candidate coarse pose, the three-dimensional structure model is rendered a second time to set the virtual observation view of the three-dimensional structure model to the view corresponding to the candidate coarse pose. The model binocular disparity image pair is cropped using the virtual observation view after the second rendering to obtain a fine model binocular disparity image pair. The target depth map is then cropped using the actual observation view corresponding to the candidate coarse pose to obtain a fine target depth map.

[0042] The 3D structural model of the target object can be a CAD model. The 3D structural model is sampled from various surrounding angles. A graphics engine is used to generate 2D renderings of the CAD model from different angles. On the graphics engine, the geometric center is calculated based on the bounding box of the CAD model. The geometric center is set as the virtual center of a sphere, and sampling points are uniformly generated on the sphere. A virtual camera is placed on the sampling points to sample using the virtual camera, resulting in several rendered RGB images and depth images, forming a model binocular disparity image pair. From the rendered model binocular disparity image pair, the depth feature extraction model is used to extract the corresponding model depth image fusion features from the model binocular disparity image pair. Actual observation features are extracted from the target depth map of the target object. By comparing with several virtual observation features, virtual observation features similar to actual observation features are selected. The sampling poses corresponding to these virtual observation features are the candidate coarse poses for preliminary perception.

[0043] After obtaining the candidate coarse pose, a more refined rendering is performed (in the graphics engine, the intrinsic parameter matrix of the virtual camera is strictly aligned with the intrinsic parameter of the real camera (to eliminate geometric errors caused by perspective distortion), and the viewing angle of the virtual camera is precisely set to the candidate coarse pose selected in the previous step, so as to output a high-fidelity image that is highly consistent with the real observation conditions). The model binocular parallax image pair and the target depth map are cropped according to the center point and size of the target object to remove the interference of environmental factors such as background.

[0044] Specifically, in the existing 3D structural model of the target object, obtain the set of its corresponding bounding-box corner points. Using candidate coarse poses, these corner points are transformed from the object coordinate system to the camera coordinate system, denoted as... :

[0045] in, This is the target object rotation increment matrix extracted from the candidate coarse pose. This is the translation increment vector of the geometric center of the target object extracted from the candidate coarse pose; the three-dimensional point set in the camera coordinate system... Projecting onto a two-dimensional image plane yields a set of two-dimensional pixel coordinates. By projecting, the coordinates of each corner point of the bounding box in the two-dimensional image plane can be obtained. Cropping can then be performed according to the two-dimensional bounding box. The same coordinates are used for cropping images from real cameras.

[0046] S5. Extract the model pose feature map and the actual pose feature map from the fine model binocular disparity image pair and the fine target depth map respectively, calculate the pose residual between the model pose feature map and the actual pose feature map, update each candidate coarse pose with the pose residual until the pose residual converges, and output the 6-DOF pose of the target object.

[0047] In one possible implementation, step S5 involves extracting the model pose feature map and the actual pose feature map from the fine model binocular disparity image pair and the fine target depth map, respectively. The pose residual between the model pose feature map and the actual pose feature map is calculated, and each candidate coarse pose is updated using the pose residual until the pose residual converges. The 6-DOF pose of the target object is then output. This can be decomposed, but is not limited to, steps S51-S56, specifically including: S51. Obtain a preset cross-attention feature extraction model, flatten the fine model binocular disparity image pair and the fine target depth map in the spatial dimension respectively, and input the flattened fine model binocular disparity image pair and the flattened fine target depth map into the cross-attention feature extraction model, so as to use the cross-attention feature extraction model to extract features from the flattened fine model binocular disparity image pair and the flattened fine target depth map to obtain the model observation feature map and the actual observation feature map; S52. Using the actual observed feature map as the query and the model observed feature map as the key and value, calculate the observation attention weight matrix. Based on the observation attention weight matrix, concatenate the actual observed feature map and the model observed feature map into a difference feature tensor, and perform linear regression calculation on the difference feature tensor to obtain the pose residual between the model pose feature map and the actual pose feature map. S53. Perform exponential mapping on the pose residual to obtain the pose transformation matrix, and use the pose transformation matrix to update each candidate coarse pose to obtain the updated candidate coarse pose. S54. The updated candidate coarse pose is used as the current candidate coarse pose. Based on the current candidate coarse pose, the model binocular disparity image pair and the target depth map are simultaneously cropped to obtain the current fine model binocular disparity image pair and the current fine target depth map. S55. Extract the current model pose feature map and the current actual pose feature map from the current fine model binocular disparity image pair and the current fine target depth map respectively, calculate the pose residual between the current model pose feature map and the current actual pose feature map as the current pose residual, and update the current candidate coarse pose using the current pose residual; S56. Obtain the preset pose residual convergence condition, perform multiple rounds of iterative updates on the current candidate coarse pose until the pose residual convergence condition is met, consider the pose residual to be converged, and obtain the current candidate coarse pose updated in the last round as the 6-DOF pose of the target object for output.

[0048] Specifically, the pose residual is calculated by introducing a cross-attention mechanism to calculate spatial correlation. The extracted fine model binocular disparity image pair and the fine target depth map are flattened in the spatial dimension and input into the cross-attention feature extraction model. The similarity matrix between the two image patches is densely calculated to perceive pixel-level spatial misalignment and extract a high-dimensional difference feature tensor including deep misalignment information. The difference feature tensor is input into a fully connected regression network (MLP) to directly linearly regress a continuous three-dimensional spatial pose residual vector (expressed in the Lie algebra se(3) format). The pose residual clearly indicates the translation increment and rotation increment required by the virtual camera. Subsequently, the residual is transformed into a standard pose transformation matrix using exponential mapping and directly applied to the current candidate coarse pose to complete the pose alignment in physical space. To achieve the highest accuracy, the updated pose needs to be cropped and rendered and the pose residual is calculated in multiple rounds to achieve multiple iterations until the regressed pose residual approaches zero, and finally outputs a high-precision 6-DOF pose.

[0049] like Figure 3As shown, the second aspect of this embodiment provides a hardware system for implementing the 6-DOF pose estimation method for reflective and transparent objects described in the first aspect of the embodiment, including: The actual image acquisition unit is used to acquire the RGB image of the target object and the original binocular parallax image pair, and to perform image correction processing on the original binocular parallax image pair to obtain the binocular parallax image pair of the target object. The cost body construction unit is used to obtain a preset depth feature extraction model, take the binocular disparity image pair of the target object as input, input it into the depth feature extraction model, extract the high-frequency contour features and semantic context features of the binocular disparity image pair through the depth feature extraction model, concatenate the high-frequency contour features and the semantic context features to output the depth image fusion features of the target object, and construct a matching cost body based on the depth image fusion features of the target object. The depth map generation unit is used to filter the matching cost body based on the attention hybrid filtering mechanism, and to perform disparity cyclic adjustment on the filtered matching cost body using a recurrent neural network to obtain the target depth map of the target object. The RGB image of the target object and the target depth map of the target object are aligned to obtain the RGB depth fusion feature map of the target object. The 3D model rendering unit is used to acquire the 3D structural model of the target object, uniformly sample and render the 3D structural model of the target object to obtain model binocular disparity image pairs from multiple perspectives, perform similarity matching between the model binocular disparity image pairs and the RGB depth fusion feature map of the target object to filter out multiple candidate coarse poses, and use each candidate coarse pose to simultaneously crop the model binocular disparity image pairs and the target depth map to obtain fine model binocular disparity image pairs and fine target depth maps. The target pose generation unit is used to extract the model pose feature map and the actual pose feature map from the fine model binocular disparity image pair and the fine target depth map, respectively, calculate the pose residual between the model pose feature map and the actual pose feature map, update each candidate coarse pose with the pose residual until the pose residual converges, and output the 6-DOF pose of the target object.

[0050] It should be noted that the 6-DOF pose estimation system for reflective and transparent objects described in this embodiment is not only specifically designed for highly reflective and transparent objects, but its depth acquisition and pose iteration framework is also applicable to 6-DOF pose estimation for general textured objects, demonstrating good generalization performance and practical value. Furthermore, during the inference phase, this system requires only a single forward propagation plus a small number of iterations, resulting in high computational efficiency. This meets the real-time operation requirements of intelligent robots, not only solving the technical bottleneck of handling highly reflective and transparent objects, but also significantly improving the robustness and accuracy of 6-DOF pose perception.

[0051] The working process, working details and technical effects of the system provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0052] like Figure 3 As shown, the third aspect of this embodiment provides an electronic device, including: a memory, a processor, and a transceiver that are sequentially and communicatively connected, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the 6-DOF pose estimation method for facing reflective and transparent objects as described in the first aspect of the embodiment.

[0053] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0054] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RISC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard) transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the electronic device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0055] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0056] The fourth aspect of this embodiment provides a storage medium that stores instructions containing the 6-DOF pose estimation method for facing reflective and transparent objects as described in the first aspect of the embodiment. That is, the storage medium stores instructions that, when executed on a computer, perform the 6-DOF pose estimation method for facing reflective and transparent objects as described in the first aspect of the embodiment.

[0057] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0058] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0059] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform a 6-DOF pose estimation method for facing reflective and transparent objects as described in the first aspect of this embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0060] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A 6-DOF pose estimation method for facing reflective and transparent objects, characterized in that, include: Acquire the RGB image of the target object and the original binocular disparity image pair, and perform image correction processing on the original binocular disparity image pair to obtain the binocular disparity image pair of the target object; A preset depth feature extraction model is obtained. The binocular disparity image pair of the target object is used as input to the depth feature extraction model. The high-frequency contour features and semantic context features of the binocular disparity image pair are extracted by the depth feature extraction model. The high-frequency contour features and the semantic context features are concatenated to output the depth image fusion features of the target object. A matching cost body is constructed based on the depth image fusion features of the target object. Based on the attention-based hybrid filtering mechanism, the matching cost volume is filtered, and the disparity of the filtered matching cost volume is adjusted cyclically using a recurrent neural network to obtain the target depth map of the target object. The RGB image of the target object and the target depth map of the target object are aligned to obtain the RGB depth fusion feature map of the target object. A three-dimensional structural model of the target object is obtained. The three-dimensional structural model of the target object is uniformly sampled and rendered to obtain model binocular disparity image pairs from multiple perspectives. The model binocular disparity image pairs are similar to the RGB depth fusion feature map of the target object to filter out multiple candidate coarse poses. Each candidate coarse pose is used to simultaneously crop the model binocular disparity image pairs and the RGB depth fusion feature map to obtain fine model binocular disparity image pairs and fine target depth map. The model pose feature map and the actual pose feature map are extracted from the fine model binocular disparity image pair and the fine target depth map, respectively. The pose residual between the model pose feature map and the actual pose feature map is calculated. Each candidate coarse pose is updated with the pose residual until the pose residual converges, and the 6-DOF pose of the target object is output.

2. The 6-DOF pose estimation method for facing reflective and transparent objects according to claim 1, characterized in that, Acquire an RGB image of the target object and a pair of original binocular disparity images, and perform image correction processing on the original binocular disparity image pair to obtain a pair of binocular disparity images of the target object, including: The RGB image of the target object is acquired in real time by an RGB camera, and the original binocular parallax image pair of the target object is acquired in real time by a binocular depth camera. The RGB image and the original binocular parallax image pair are then time-synchronized to obtain the synchronized RGB image and the original binocular parallax image pair. A preset epipolar correction model is obtained, and the original binocular disparity image pair after synchronous processing is used as input to the epipolar correction model so as to perform epipolar correction on the original binocular disparity image pair to obtain the binocular disparity image pair of the target object.

3. The 6-DOF pose estimation method for facing reflective and transparent objects according to claim 1, characterized in that, The deep feature extraction model includes an input layer, a first branch feature processing layer, a second branch feature processing layer, a feature fusion layer, and an output layer, wherein: The input layer is used to receive binocular parallax image pairs as input, and to pass the binocular parallax image pairs to the first branch feature processing layer and the second branch feature processing layer respectively. The first branch feature processing layer is used to identify edge region information, corner region information and texture change region information of the target object from the binocular parallax image pair, so as to complete the extraction of geometric structure feature information of the target object and output the high-frequency contour features of the target object. The second branch feature processing layer is used to identify the object category information, local and global relationship information and occlusion and occlusion relationship information of the target object from the binocular parallax image pair, so as to complete the extraction of the target object association feature information and output the semantic context features of the target object. The feature fusion layer is used to receive the high-frequency contour features output by the first branch feature processing layer and the semantic context features output by the second branch feature processing layer, and to perform feature concatenation on the high-frequency contour features and the semantic context features in the feature dimension to obtain the depth image fusion features. The output layer is used to receive the depth image fusion features output by the feature fusion layer and output the depth image fusion features.

4. The 6-DOF pose estimation method for facing reflective and transparent objects according to claim 3, characterized in that, The binocular disparity image pair of the target object is used as input to the depth feature extraction model. The model extracts high-frequency contour features and semantic context features of the binocular disparity image pair. The high-frequency contour features and semantic context features are concatenated to output the depth image fusion features of the target object. A matching cost body is constructed based on the depth image fusion features of the target object, including: The binocular disparity image pairs of the target object are used as inputs and fed into the input layer of the depth feature extraction model. The binocular disparity image pair is obtained from the input layer of the deep feature extraction model through the first branch feature processing layer of the deep feature extraction model, and the binocular disparity image pair is convolved using the first branch feature processing layer of the deep feature extraction model to extract the high-frequency contour features of the target object from the binocular disparity image pair. The binocular disparity image pair is obtained from the input layer of the deep feature extraction model through the second branch feature processing layer of the deep feature extraction model, so as to perform semantic recognition on the binocular disparity image pair using the second branch feature processing layer of the deep feature extraction model, and to extract the semantic context features of the target object from the binocular disparity image pair. The high-frequency contour features are extracted from the first branch feature processing layer of the deep feature extraction model through the feature fusion layer of the deep feature extraction model, and the semantic context features are extracted from the second branch feature processing layer of the deep feature extraction model. The semantic context features are linearly upsampled, and the upsampled semantic context features and the high-frequency contour features are concatenated to obtain the depth image fusion features of the target object. The depth image fusion features are extracted from the feature fusion layer of the depth feature extraction model through the output layer of the depth feature extraction model, and the depth image fusion features of the target object are used as the output of the depth feature extraction model for output. Obtain a preset desired depth range, calculate the maximum disparity using the depth image fusion features of the target object and the desired depth range, generate a corresponding disparity search range based on the maximum disparity, and search for each pixel of the binocular disparity image pair within the disparity search range to form multiple disparities; Based on the depth image fusion features of the target object, binocular feature similarity is calculated for each disparity to obtain the fusion feature similarity corresponding to each pixel position; The fused feature similarity of each pixel location is used as the matching similarity of each pixel location between the binocular disparity image pairs. A corresponding two-dimensional similarity map is constructed for each pixel location, and the two-dimensional similarity maps of each pixel location are stacked in the depth direction to construct a matching cost body. Each element in the matching cost body represents the degree of disparity matching between the binocular disparity image pairs.

5. The 6-DOF pose estimation method for facing reflective and transparent objects according to claim 1, characterized in that, Based on an attention-based hybrid filtering mechanism, the matching cost volume is filtered, and a recurrent neural network is used to perform disparity cyclic adjustment on the filtered matching cost volume to obtain the target depth map of the target object. The RGB image and the target depth map of the target object are then aligned to obtain the RGB depth fusion feature map of the target object, including: The matching cost volume is subjected to planar convolution filtering and axial convolution filtering to obtain planar convolutional three-dimensional feature tensors and axial convolutional three-dimensional feature tensors. The planar convolutional 3D feature tensor and the axial convolutional 3D feature tensor are subjected to feature concatenation and dimensionality reduction to obtain a filtered matching cost body as the current matching cost body. A preset recurrent neural network is obtained, and the current matching cost body is input into the recurrent neural network to generate a corresponding disparity correction amount for the current matching cost body using the recurrent neural network. The disparity correction amount is then used to update the disparity of each pixel position in the current matching cost body to obtain the updated matching cost body. The updated matching cost body is used as the current matching cost body, and the recurrent neural network is used to update the current matching cost body in multiple rounds until the disparity correction amount converges to obtain the optimal matching cost body. The final disparity value of each pixel position is extracted using the optimal matching cost body, and a target depth map of the target object is generated based on the final disparity value of each pixel position. Obtain the preset extrinsic transformation matrix, perform three-dimensional back projection on each depth pixel in the target depth map of the target object, obtain the three-dimensional coordinates of each depth pixel in the depth camera coordinate system, and use the extrinsic transformation matrix to transform the three-dimensional coordinates of each depth pixel in the depth camera coordinate system to the RGB camera coordinate system to obtain the three-dimensional coordinates of each depth pixel in the RGB camera coordinate system. Obtain the intrinsic parameter matrix of the RGB camera. Using the intrinsic parameter matrix of the RGB camera, reproject the three-dimensional coordinates of each depth pixel in the RGB camera coordinate system onto the RGB image plane to obtain the RGB image plane pixel coordinates corresponding to each three-dimensional coordinate. Match each pixel in the RGB image of the target object with the RGB image plane pixel coordinates corresponding to each three-dimensional coordinate reprojected onto the RGB image plane to complete the alignment of the target depth map of the target object with the RGB image, and obtain the RGB depth fusion feature map of the target object.

6. The 6-DOF pose estimation method for facing reflective and transparent objects according to claim 1, characterized in that, A 3D structural model of the target object is obtained. This model is then uniformly sampled and rendered to obtain binocular disparity image pairs from multiple viewpoints. Similarity matching is performed between these binocular disparity image pairs and the RGB depth fusion feature map of the target object to filter out multiple candidate coarse poses. Each candidate coarse pose is then used to simultaneously crop both the binocular disparity image pairs and the RGB depth fusion feature map, resulting in refined binocular disparity image pairs and a refined target depth map, including: A three-dimensional structural model of the target object is obtained, and the three-dimensional structural model of the target object is uniformly sampled and rendered to generate a two-dimensional rendering image from multiple perspectives. The two-dimensional rendering images are then integrated to form a binocular parallax image pair of the three-dimensional structural model. The depth feature extraction model is used to extract the corresponding model depth image fusion feature map from the model binocular parallax image pair; A preset feature similarity threshold is obtained, and feature similarity is calculated between the model depth image fusion feature map and each observed feature in the RGB depth fusion feature map of the target object, so as to match each observed feature with a feature similarity higher than the feature similarity threshold from the RGB depth fusion feature map as a candidate coarse pose. Based on the candidate coarse pose, the three-dimensional structure model is rendered a second time to set the virtual observation view of the three-dimensional structure model to the view corresponding to the candidate coarse pose. The model's binocular disparity image pair is then cropped using the virtual observation view after the second rendering to obtain a fine model binocular disparity image pair. Finally, the RGB depth fusion feature map is cropped using the actual observation view corresponding to the candidate coarse pose to obtain a fine target depth map.

7. The 6-DOF pose estimation method for facing reflective and transparent objects according to claim 1, characterized in that, The model pose feature map and the actual pose feature map are extracted from the fine model binocular disparity image pair and the fine target depth map, respectively. The pose residual between the model pose feature map and the actual pose feature map is calculated. Each candidate coarse pose is updated with the pose residual until the pose residual converges. The 6-DOF pose of the target object is then output, including: A preset cross-attention feature extraction model is obtained. The fine model binocular disparity image pair and the fine target depth map are flattened in the spatial dimension, respectively. The flattened fine model binocular disparity image pair and the flattened fine target depth map are input into the cross-attention feature extraction model. The cross-attention feature extraction model is used to extract features from the flattened fine model binocular disparity image pair and the flattened fine target depth map to obtain the model observation feature map and the actual observation feature map. Using the actual observed feature map as the query and the model observed feature map as the key and value, the observation attention weight matrix is ​​calculated. Based on the observation attention weight matrix, the actual observed feature map and the model observed feature map are concatenated into a difference feature tensor, and linear regression is performed on the difference feature tensor to obtain the pose residual between the model pose feature map and the actual pose feature map. The pose residual is subjected to exponential mapping to obtain the pose transformation matrix. The pose transformation matrix is ​​then used to update each candidate coarse pose to obtain the updated candidate coarse pose. The updated candidate coarse pose is used as the current candidate coarse pose. Based on the current candidate coarse pose, the model binocular disparity image pair and the target depth map are simultaneously cropped to obtain the current fine model binocular disparity image pair and the current fine target depth map. The current model pose feature map and the current actual pose feature map are extracted from the current fine model binocular disparity image pair and the current fine target depth map, respectively. The pose residual between the current model pose feature map and the current actual pose feature map is calculated as the current pose residual, and the current candidate coarse pose is updated using the current pose residual. Obtain the preset pose residual convergence condition, perform multiple rounds of iterative updates on the current candidate coarse pose until the pose residual convergence condition is met, then consider the pose residual to be converged, and obtain the current candidate coarse pose updated in the last round as the 6-DOF pose of the target object for output.

8. A 6-DOF pose estimation system for facing reflective and transparent objects, characterized in that, The 6-DOF pose estimation method for facing reflective and transparent objects as described in any one of claims 1 to 7 includes: The actual image acquisition unit is used to acquire the RGB image of the target object and the original binocular parallax image pair, and to perform image correction processing on the original binocular parallax image pair to obtain the binocular parallax image pair of the target object. The cost body construction unit is used to obtain a preset depth feature extraction model, take the binocular disparity image pair of the target object as input, input it into the depth feature extraction model, extract the high-frequency contour features and semantic context features of the binocular disparity image pair through the depth feature extraction model, concatenate the high-frequency contour features and the semantic context features to output the depth image fusion features of the target object, and construct a matching cost body based on the depth image fusion features of the target object. The depth map generation unit is used to filter the matching cost body based on the attention hybrid filtering mechanism, and to perform disparity cyclic adjustment on the filtered matching cost body using a recurrent neural network to obtain the target depth map of the target object. The RGB image of the target object and the target depth map of the target object are aligned to obtain the RGB depth fusion feature map of the target object. The 3D model rendering unit is used to acquire the 3D structural model of the target object, uniformly sample and render the 3D structural model of the target object to obtain model binocular disparity image pairs from multiple perspectives, perform similarity matching between the model binocular disparity image pairs and the RGB depth fusion feature map of the target object to filter out multiple candidate coarse poses, and use each candidate coarse pose to simultaneously crop the model binocular disparity image pairs and the target depth map to obtain fine model binocular disparity image pairs and fine target depth maps. The target pose generation unit is used to extract the model pose feature map and the actual pose feature map from the fine model binocular disparity image pair and the fine target depth map, respectively, calculate the pose residual between the model pose feature map and the actual pose feature map, update each candidate coarse pose with the pose residual until the pose residual converges, and output the 6-DOF pose of the target object.

9. An electronic device, characterized in that, The device includes a memory, a processor, and a transceiver that are sequentially and communicatively connected. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the 6-DOF pose estimation method for facing reflective and transparent objects as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the 6-DOF pose estimation method for facing reflective and transparent objects as described in any one of claims 1 to 7.

Images