A single-view pose estimation method and system based on multi-modal input and attention mechanism

Abstract

The application discloses a single-view pose estimation method based on multi-modal input and attention mechanism, constructs a single-view pose estimation system including a prediction module and a pose regression module, combines multi-modal input and attention feature enhancement technology, learns various intermediate representation features of an object from a two-dimensional image, and then regresses the 6D pose of the object, and comprises the following steps: the prediction module adopts ResNet-18 as a backbone network, the module adds a channel attention mechanism to estimate various intermediate representations of the 6D pose of the object, including key points, edge vectors between the key points and symmetric corresponding relationships between pixel points; the pose regression module uses an EPnP algorithm and singular value decomposition to regress the 6D pose of the object from the intermediate representation result. The application provides a technology for accurately, conveniently and quickly estimating the 6D pose of an object from a single view.

Description

Technical Field

[0001] This invention belongs to the technical field of estimating the 6D pose of an object from a single view, and relates to a single-view pose estimation method and system based on multimodal input and attention mechanism. Background Technology

[0002] Estimating the 6D pose of objects from RGB images is a crucial task in applications such as robotic grasping and augmented reality. While introducing depth images can significantly improve this task, these images are not always readily available. For example, most mobile phones, tablets, and industrial cameras do not provide depth data. Therefore, much research has focused on estimating the 6D pose of known objects using only RGB images. Traditional methods match RGB image features with the object's 3D model to address this problem. These methods rely on manually labeled features and lack robustness to changes in lighting, background clutter, or objects with low texture. The development of deep learning has accelerated research into estimating the 6D pose of objects from RGB images. Currently popular keypoint methods utilize keypoints as intermediate supervision signals for model training. Then, the 2D keypoints predicted by the neural network are combined with the PnP algorithm to estimate the 6D pose of the object. Examples include PvNet [Peng S et al., Pixel-wise voting network for 6dofpose estimation IEEE / CVF 2019] and Pix2pose [Park K et al., Pixel-wise coordinateregression of objects for 6d pose estimation. IEEE / CVF 2019].

[0003] However, the performance of the keypoint method relies on the following two assumptions: 1) the deep learning model can accurately predict the location of 2D keypoints; 2) the predicted 2D keypoints provide sufficient constraints to regress the 6D pose of the object. However, factors such as partial occlusion of the object can lead to inaccurate keypoint predictions, so these two assumptions are not easily valid in many real-world environments. Summary of the Invention

[0004] The purpose of this invention is to overcome the limitations of existing single-view... Figure 6 To address the shortcomings of existing 6D pose estimation methods, this paper proposes a single-view pose estimation method and system based on multimodal input and attention mechanisms. By combining multimodal input and attention feature enhancement techniques, multiple intermediate representation features of the object are learned from the 2D image, and then the 6D pose of the object is estimated from a single RGB image, which has higher robustness and pose estimation accuracy.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution.

[0006] This invention discloses a single-view pose estimation method based on multimodal input and attention mechanisms. It constructs a single-view pose estimation system including a prediction module and a pose regression module. Combining multimodal input and attention feature enhancement techniques, it learns multiple intermediate representation features of the object from a two-dimensional image, and then regresses the object's 6D pose. The method includes the following steps:

[0007] Step 1: The prediction module uses multiple intermediate representations to express the geometric information in the RGB image, and introduces an attention mechanism to improve the network training efficiency; the multiple intermediate representations include the key points κ, edge vectors ε, and dense pixel-wise correspondences S of the RGB image;

[0008] The prediction module includes a first prediction network. Second prediction network Third prediction network A fully connected network is used, and channel attention modules are embedded between each downsampling module of the three prediction networks; Using PVNet as the backbone, it is a keypoint-based pose estimation network that predicts k visible and invisible keypoints using a voting method; the prediction network... Used to optimize object pose;

[0009] Step 2: The pose regression module obtains the intermediate representation results obtained by the prediction module, combines the key points, edge vectors and dense pixel-by-pixel correspondence information, and regresses the 6D pose of the object from the intermediate representation results through EPnP calculation and singular value decomposition.

[0010] The pose regression module uses the intermediate representations {κ,ε,S} predicted by each network of the prediction module as input, and outputs the 6D pose of object I: R I ∈SO(3),

[0011] Specifically, step 1 includes:

[0012] Construct a fully connected graph ε using key points as nodes: It is a network that predicts the vectors along the graph edges, using ResNet-18 as the backbone network; ε explicitly expresses the displacement between each pair of keypoints, and |ε| represents the number of edges in the predefined graph, therefore

[0013] A third intermediate representation S, which reflects the pixel symmetry correspondence, is predicted and generated to reflect the latent reflection symmetry of the object. Extend the FlowNet network architecture by fusing the dense pixel stream predicted by FlowNet with the mask map predicted by PVNet to predict the symmetrical correspondence of each pixel within the mask region;

[0014] The loss l1, The loss l2, The loss l3 is smoothed using Fast R-CNN. The loss is used for training; to reflect the importance of different intermediate representations to the pose estimation network performance, the losses of the three intermediate representations are weighted using three parameters x, y, and z, where x + y + z = 1. Therefore, the total loss is:

[0015] L=xl1+yl2+zl3 (1)

[0016] The channel attention module will and Each residual block outputs a dimension of... The tensor F is used as input to the attention module and undergoes average pooling. The channel attention module generates a dimension of by performing a one-dimensional convolution with a kernel size of n=5. The channel weights are obtained; the weights are processed by the activation function and the dimension is reduced to obtain the result F′, which is then used as the input to the next residual block.

[0017] Specifically, step 2 includes:

[0018] The true coordinates of the 3D key points in the standard coordinate system are represented as follows: The truth value of the edge vector is represented as The keypoint coordinates output by the prediction module are represented as follows: The edge vector is represented as Symmetric correspondence is represented as Homogeneous coordinates are used for ease of calculation. as well as Corresponding to p k ,ν e q s,1 and q s,2 These homogeneous coordinates are normalized using known camera intrinsic parameters;

[0019] The EPnP algorithm is used, combined with the constraints of the intermediate representation, to calculate the 6D pose of the object; firstly, the following difference vectors are introduced for the three prediction elements:

[0020]

[0021]

[0022]

[0023] Where e s and e t It is the endpoint of edge e. It is the normal to the plane of reflection symmetry in the base coordinate system;

[0024] Secondly, equation (2) is transformed into the form A1x, A2x, A3x, and similarly equation (3) is transformed into A4x, A5x, A6x, and equation (4) is transformed into A7x; A1, A2, A3, A4, A5, and A6 are merged into A; to describe the relationship between the predicted value and the true value, a linear system of the form Ax = 0 is introduced, where A is a matrix of dimension (3|κ|+3|ε|+|S|)×12; x is a vector in affine space containing the rotation matrix R and the translation vector t parameters;

[0025] Next, the EPNP algorithm is used to calculate:

[0026]

[0027] (5) where v i It is the right singular vector corresponding to the i-th smallest singular value of A; ideally, when the predicted elements are noise-free, N=1, x=v i This is the optimal solution; choose N=4, the same as EPnP; to compute the optimal x, use the following objective function to optimize the latent variable λ during the alternating optimization process. i And rotation matrix R:

[0028]

[0029] (6) In the formula Including v i The first 9 elements; to obtain the optimal λ i Then, SVD decomposition is used to... Projecting onto SO(3), we obtain the rotation matrix R = Udiag(1,1,1)V. T Finally, using Ax = 0, we obtain the corresponding translation vector t:

[0030]

[0031] (7) In the formula, A1=A [:,1:9] A2 = A [:,10:12] , Obtained by flattening R.

[0032] The present invention provides a single-view pose estimation system based on multimodal input and attention mechanism, comprising:

[0033] The prediction module utilizes multiple intermediate representations to express the geometric information in RGB images, and introduces an attention mechanism to improve network training efficiency; it includes a first prediction network. Second prediction network Third prediction network A fully connected network is used, and channel attention modules are embedded between each downsampling module of the three prediction networks; Using PVNet as the backbone, it is a keypoint-based pose estimation network that predicts k visible and invisible keypoints using a voting method; the prediction network... Used to optimize object pose;

[0034] The pose regression module acquires the intermediate representation results obtained by the prediction module, combines keypoints, edge vectors, and dense pixel-by-pixel correspondence information, and regresses the 6D pose of the object from the intermediate representation results through EPnP calculation and singular value decomposition. The pose regression module uses the intermediate representations {κ,ε,S} predicted by each network of the prediction module as input and outputs the 6D pose of object I: R I ∈SO(3),

[0035] Furthermore, in the prediction module:

[0036] Using PVNet as the backbone, it is a keypoint-based pose estimation network that constructs a fully connected graph with keypoints as nodes. It is a network that predicts the vectors along the edges of the graph, using ResNet-18 as the backbone network; ε explicitly expresses the displacement between each pair of keypoints, and |ε| represents the number of edges in the predefined graph; ε is a fully connected graph, therefore

[0037] Predict and generate a third intermediate representation S that reflects the pixel symmetry correspondence and reflects the latent reflection symmetry of the object; The network architecture of FlowNet is extended by fusing the dense pixel stream predicted by FlowNet with the mask map predicted by PVNet to predict the symmetrical correspondence of each pixel within the mask region.

[0038] loss l1, loss l2, The loss l3 is smoothed using Fast R-CNN. The loss is used for training; to reflect the importance of different intermediate representations to the pose estimation network performance, the losses for the three intermediate representations are weighted using three parameters x, y, and z, where x + y + z = 1; therefore, the total loss is:

[0039] L=xl1+yl2+zl3 (1)

[0040] The channel attention module will and Each residual block outputs a dimension of... The tensor F is used as input to the attention module and undergoes average pooling. The channel attention module generates a dimension of by performing a one-dimensional convolution with a kernel size of n=5. The channel weights are obtained; the weights are processed by the activation function and the dimension is reduced to obtain the result F′, which is then used as the input to the next residual block.

[0041] Furthermore, the pose regression module:

[0042] The true coordinates of the 3D key points in the standard coordinate system are represented as follows: The truth value of the edge vector is represented as The keypoint coordinates output by the prediction module are represented as follows: The edge vector is represented as Symmetric correspondence is represented as Homogeneous coordinates are used for ease of calculation. as well as Corresponding to p k ,ν e q s,1 and q s,2 These homogeneous coordinates are normalized using known camera intrinsic parameters;

[0043] The EPnP algorithm is used, combined with the constraints of the intermediate representation, to calculate the 6D pose of the object; firstly, the following difference vectors are introduced for the three prediction elements:

[0044]

[0045]

[0046]

[0047] Where e s and e t It is the endpoint of edge e. It is the normal to the plane of reflection symmetry in the base coordinate system;

[0048] Secondly, equation (2) is transformed into the form A1x, A2x, A3x, and similarly equation (3) is transformed into A4x, A5x, A6x, and equation (4) is transformed into A7x; A1, A2, A3, A4, A5, A6 are merged into A; to describe the relationship between the predicted value and the true value, a linear system of the form Ax = 0 is introduced, where A is a matrix of dimension (3|κ|+3|ε|+|S|)×12; x is a vector in affine space containing the rotation matrix R and the translation vector t parameters;

[0049] Next, the EPNP algorithm is used to calculate x:

[0050]

[0051] (5) where v i It is the right singular vector corresponding to the i-th smallest singular value of A; ideally, when the predicted elements are noise-free, N=1, x=v i This is the optimal solution; choose N=4, the same as EPnP; to compute the optimal x, use the following objective function to optimize the latent variable λ during the alternating optimization process. i And rotation matrix R:

[0052]

[0053] (6) In the formula Including v i The first 9 elements; to obtain the optimal λ i Then, SVD decomposition is used to... Projecting onto SO(3), we obtain the rotation matrix R = Udiag(1,1,1)V. T Finally, using Ax = 0, we obtain the corresponding translation vector t:

[0054]

[0055] (7) In the formula, A1=A [:,1:9] A2 = A [:,10:12] , Obtained by flattening R.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] 1. This invention proposes a novel 6D pose estimation method, which mainly includes a 6D pose estimation network based on channel attention. Convolutional Neural Networks (CNNs) suffer from limitations in predicting 6D poses due to their kernel size and feature pooling. When the target region image in the network input occupies a small proportion of the overall input image, background information generates significant noise during multiple convolution iterations, affecting feature extraction of the target region. To overcome the limitations of CNNs and improve network training efficiency and accuracy, this invention introduces a channel attention module, which can better extract detailed features from the image, resulting in more accurate pose estimation results.

[0058] 2. This invention utilizes multiple intermediate representations—keypoints, edge vectors between keypoints, and dense pixel-by-pixel correspondences—to express different geometric information in the input image, maximizing the accuracy of 6D pose estimation for the generated object. Adding symmetric correspondence constraints to the keypoints improves the model's performance in estimating the rotation matrix; adding edge vectors to both keypoints and symmetric correspondences provides more constraints on translation and rotation. Edge vectors offer more translational constraints than keypoints because they represent the displacement of adjacent keypoints and provide gradient information for regression. Unlike symmetric correspondences, edge vectors constrain three degrees of freedom of the rotation parameters, further improving the performance of estimating the rotation matrix.

[0059] 3. Even in complex environments and under occlusion conditions, the method of this invention can predict the 6D pose of objects with considerable accuracy. The performance of the 6D pose estimation network of this invention was evaluated on two popular benchmark datasets for 6D pose estimation: Linemod and Occlusion Linemod. Compared with current state-of-the-art 6D pose estimation algorithms (such as Pix2Pose, PVNet, PoseCNN, CDPN, etc.), the average ADD(-S) accuracy of this invention is 92.8 on the Linemod dataset and 48.0 on the Occlusion Linemod dataset. In handling occlusion problems, the method of this invention achieves a 16% improvement over Pix2Pose, clearly demonstrating the advantage of this invention in predicting the 6D pose of occluded objects. Attached Figure Description

[0060] Figure 1 This is a system block diagram of one embodiment of the present invention.

[0061] Figure 2 This is a channel attention module diagram according to an embodiment of the present invention.

[0062] Figure 3 This is a diagram showing the intermediate representation results and pose estimation results of different objects in the experiment of this invention.

[0063] Figure 4 This is a diagram showing the pose estimation results under different occlusion ratios in the experiment of this invention.

[0064] Figure 5 This is a comparison chart of network training loss before and after adding the attention module to the model in the experiment of this invention.

[0065] Figure 6 This is a comparison chart of network training loss before and after adding an attention module to the cat model in the experiment of this invention. Detailed Implementation

[0066] This invention discloses a single-view pose estimation method and system based on multimodal input and attention mechanisms. The constructed system includes a prediction module and a pose regression module. Combining multimodal input and attention feature enhancement techniques, it learns multiple intermediate representation features of the object from a two-dimensional image, and then regresses the object's 6D pose. The steps include: the prediction module uses ResNet-18 as the backbone network, and this module incorporates a channel attention mechanism to estimate multiple intermediate representations of the object's 6D pose, including keypoints, edge vectors between keypoints, and symmetric correspondences between pixels; the pose regression module uses the EPnP algorithm and singular value decomposition to regress the object's 6D pose from the intermediate representation results. This invention provides an accurate, convenient, and fast technique for estimating the 6D pose of an object from a single view.

[0067] The present invention will now be described in further detail with reference to the accompanying drawings.

[0068] Figure 1 This is a system block diagram of one embodiment of the present invention, which also embodies the technical principle of the present invention: aiming to estimate the 6D pose of an object from a single RGB image, the method of the present invention includes:

[0069] Step 1: The prediction module uses multiple intermediate representations to express the geometric information in the RGB image, and introduces an attention mechanism to improve the network training efficiency.

[0070] Step 2: The pose regression module obtains the intermediate representation results from the prediction module, combines the key points, edge vectors and dense pixel-by-pixel correspondence information, and regresses the 6D pose of the object from the intermediate representation results through EPnP calculation and singular value decomposition.

[0071] Its prediction module includes three prediction networks: the first prediction network Second prediction network and the third prediction network It also includes a fully connected network, with channel attention modules embedded between each downsampling module of the three prediction networks.

[0072] PVNet, a keypoint-based pose estimation network, is used as the backbone. It employs a voting method to predict k visible and k invisible keypoints. To address the issue of large pose errors when keypoint prediction is inaccurate, two additional prediction networks are introduced. To optimize object pose.

[0073] This invention constructs a fully connected graph using key points as nodes. This is a network that predicts graph edge vectors along the graph, using ResNet-18 as its backbone. ε explicitly represents the displacement between each pair of keypoints, and |ε| represents the number of edges in the predefined graph. In this invention, ε is a fully connected graph, therefore...

[0074] The prediction generates a third intermediate representation S that reflects the pixel symmetry correspondences, which reflect the latent reflection symmetry of the object. The FlowNet network architecture is extended by fusing the dense pixel stream predicted by FlowNet with the mask map predicted by PVNet to predict the symmetric correspondence of each pixel within the mask region. Compared with the previous two intermediate representation methods, the symmetric correspondence has a larger data volume and stronger constraints on the pose estimation results. In particular, it provides rich constraints for occluded objects and can better solve the occlusion problem.

[0075] In this invention The loss l1, The loss l2, The loss l3 is smoothed using Fast R-CNN. The loss is used for training. To reflect the importance of different intermediate representations to the pose estimation network's performance, this invention uses three parameters, x, y, and z, to weight the losses of the three intermediate representations, where x + y + z = 1. Therefore, the total loss is:

[0076] L=xl1+yl2+zl3 (1)

[0077] like Figure 2 As shown, the channel attention module will... and Each residual block outputs a dimension of... The tensor F is used as input to the attention module and undergoes average pooling. The channel attention module generates a dimension of by performing a one-dimensional convolution with a kernel size of n=5. The channel weights are then processed by an activation function and dimension reduction to obtain the result F′, which is used as the input to the next residual block.

[0078] Pose Regression Module: Takes the intermediate representation {κ,ε,S} predicted by the network as input and outputs the 6D pose (R) of object I. I ∈SO(3), ).

[0079] This invention represents the true coordinates of three-dimensional key points in the standard coordinate system as follows: The truth value of the edge vector is represented as The keypoint coordinates output by the prediction module are represented as follows: The edge vector is represented as Symmetric correspondence is represented as Homogeneous coordinates are used in this invention for ease of calculation. as well as Corresponding to p k ,ν e q s,1 and q s,2 These homogeneous coordinates are normalized using known camera intrinsic parameters.

[0080] This invention uses the EPnP algorithm and combines it with intermediate representation constraints to calculate the 6D pose of the object. Firstly, this invention introduces the following difference vectors for the three prediction elements:

[0081]

[0082]

[0083]

[0084] Where e s and e t It is the endpoint of edge e. It is the normal to the plane of reflection symmetry in the base coordinate system.

[0085] Secondly, equation (2) is transformed into the form A1x, A2x, A3x, and similarly, equation (3) is transformed into A4x, A5x, A6x, and equation (4) is transformed into A7x. A1, A2, A3, A4, A5, and A6 are then combined into A. To describe the relationship between the predicted and actual values, this invention introduces a linear system of the form Ax = 0, where A is a matrix of dimension (3|κ| + 3|ε| + |S|) × 12. x is a vector in affine space containing the rotation matrix R and the translation vector t.

[0086] Next, the EPNP algorithm is used to calculate x:

[0087]

[0088] (5) where v iIt is the right singular vector corresponding to the i-th smallest singular value of A. Ideally, when the predicted elements are noise-free, N=1, x=v i This is the optimal solution. However, in real-world situations, the performance of this method is poor. This invention chooses the same N=4 as EPNP. To calculate the optimal x, this invention uses the following objective function to optimize the latent variable λ during the alternating optimization process. i And rotation matrix R:

[0089]

[0090] (6) In the formula Including v i The first 9 elements. To obtain the optimal λ. i Then, SVD decomposition is used to... Projecting onto SO(3), we obtain the rotation matrix R = Udiag(1,1,1)V. T Finally, using Ax = 0, we obtain the corresponding translation vector t:

[0091]

[0092] (7) In the formula, A1=A [:,1:9] A2 = A [:,10:12] , Obtained by flattening R.

[0093] Experimental verification of the present invention:

[0094] Figure 3 The following are three intermediate representations of the pose estimation results of the present invention and the 6D pose estimation results of a single target object: (a) represents the input image; (b), (c), and (d) represent the predicted key points, edge vectors, and corresponding symmetrical images, respectively; (e) represents the 6D pose calculated by regression, where the green box represents the true value of the 6D pose of the object, and the blue box represents the 6D pose of the object predicted by the present invention. Figure 4 The results of 6D pose estimation of the object according to the present invention under different proportions of occlusion are presented. It can be seen that even under complex environments and occlusion conditions, the present invention can predict the 6D pose of the object with considerable accuracy. The performance of the 6D pose estimation network of the present invention is evaluated on two popular benchmark datasets in 6D pose estimation testing: Linemod and Occlusion Linemod, and compared with current state-of-the-art 6D pose estimation algorithms (such as Pix2Pose, PVNet, PoseCNN, CDPN, etc.). As shown in Tables 1 and 2, the average ADD(-S) accuracy of the present invention is 92.8 on the Linemod dataset; and the average ADD(-S) accuracy is 48.0 on the Occlusion Linemod dataset.

[0095] from Figure 3 , Figure 4 As shown in Tables 1 and 2, this invention achieves accurate pose estimation, outperforming most state-of-the-art methods in tests on Linemod and Occlusion Linemod. This invention outperforms the backbone model PVNet used for keypoint prediction, showing performance improvements across all object classes, demonstrating that using multiple intermediate representations is significantly superior to methods using only keypoints. In handling occlusion issues, this invention's method achieves a 16% improvement over Pix2Pose, clearly demonstrating the advantages of multiple intermediate representations in predicting the 6D pose of occluded objects.

[0096] Table 1 Comparison of ADD(-S) performance on the Linemod dataset.

[0097]

[0098] Table 2 Comparison of ADD(-S) performance on the Octclusion Linemod dataset

[0099]

[0100] To verify the performance of adding the attention mechanism module, this invention tests the training effects before and after adding the attention module on the same dataset. Figure 5 It shows the combined loss curves before and after the improvement.

[0101] Depend on Figure 5 It is evident that adding an attention module to the backbone network optimizes the final training results of object pose estimation, and the loss convergence speed is faster and the loss is smaller during training. Experimental results show that, compared with the previous version, using the improved ResNet-18 as the base neural network improves the ADD(-S) metric by an average of 3.1% on the Linemod dataset and by an average of 1.4% on the Occlusion Linemod dataset, demonstrating that the present invention can estimate the 6D pose of objects more accurately.

[0102] In summary, this invention proposes a single-view pose estimation method and system based on multimodal input and an attention mechanism, comprising two modules. The first module utilizes multiple intermediate representations to express the geometric information in RGB images, while introducing an attention mechanism to improve network training efficiency. This module predicts not only keypoints but also edge vectors between adjacent keypoints. Furthermore, this invention utilizes predicted dense pixel-wise correspondences to reflect the basic symmetry between pixels. This pose estimation method integrating multiple intermediate representations provides more constraints compared to methods using only keypoint representations, achieving accurate pose prediction even in the presence of occlusion and shadows. The second module acquires the intermediate representation results obtained from the previous module, combining keypoint, edge vector, and dense pixel-wise correspondence information, and regresses the 6D pose of the object from the intermediate representation results through EPnP calculation and singular value decomposition. This technology is applicable to multiple fields such as integrated ship support, automated sorting, autonomous driving, medical, virtual reality, augmented reality, and industrial manufacturing. It can accurately extract the 6D pose of objects from images, providing technical support for the intelligent and information-based development of industrial manufacturing, and has broad market prospects.

Claims

1. A single-view pose estimation method based on multi-modal input and attention mechanism, characterized in that, A single-view pose estimation system including a prediction module and a pose regression module is constructed, multi-modal input and attention feature enhancement technology are combined, various intermediate representation features of an object are learned from a two-dimensional image, and then a 6D pose of the object is regressed, including the following steps: In step 1, the prediction module uses various intermediate representations to express geometric information in an RGB image, and an attention mechanism is introduced to improve network training efficiency; the various intermediate representations include key points κ, edge vectors ε and dense pixel-by-pixel correspondence S of the RGB image; The prediction module comprises a first prediction network A second prediction network A third prediction network And a full connection network, a channel attention module is embedded between each down-sampling module of the three prediction networks; PVNet is used as the backbone network, which is a key point-based pose estimation network, and adopts a voting method to predict k key points that can be seen and cannot be seen; the prediction network To optimize the object pose; In step 2, the pose regression module obtains the intermediate representation results obtained by the prediction module, combines the key points, edge vectors and dense pixel-by-pixel correspondence information, and regresses the 6D pose of the object from the intermediate representation results through EPnP calculation and singular value decomposition; The pose regression module outputs the 6D pose of the object I: R I ∈ SO(3), Step 2 includes: The three-dimensional ground truth coordinates of the key points in the standard coordinate system are denoted as 1≤k≤|κ|, the edge vector ground truth is denoted as 1≤e≤|ε|, the key point coordinates output by the prediction module are denoted as 1≤k≤|κ|; the edge vector is denoted as 1≤e≤|ε|; the symmetric correspondence is denoted as 1≤s≤|S|; for the convenience of calculation, homogeneous coordinates are used and corresponding to p k , v e , q s,1 and q s,2 , these homogeneous coordinates are normalized by known camera intrinsic parameters; The EPnP algorithm is used, and the 6D pose of the object is calculated in combination with the constraints of the intermediate representation; first, the following difference vectors are introduced for the three predicted elements: where e s and e t is an end point of the edge e, is a normal to the plane of reflection symmetry in the base coordinate system; Secondly, formula (2) is converted into the form of A1x, A2x and A3x, formula (3) is converted into A4x, A5x and A6x in the same way, and formula (4) is converted into A7x; A1, A2, A3, A4, A5, A6 are combined into A; in order to describe the relationship between the predicted value and the true value, a linear system in the form of Ax=0 is introduced, where A is a matrix with a dimension of (3|κ|+3|ε|+|S|)×12; x is a vector containing a rotation matrix R and a translation vector t parameter in an affine space; Then, the EPnP algorithm is used to calculate x: (5) where v i is the ith smallest singular value of A; ideally, N = 1, x = v i is the optimal solution; select the same N = 4 as EPnP; to compute the optimal x, use the following objective function to optimize the hidden variable λ in an alternating optimization procedure i and the rotation matrix R: (6) where including v i The first 9 elements of U; after obtaining the optimal i , use SVD decomposition to project to SO(3), i.e., get the rotation matrix R = Udiag(1, 1, 1)V T ; finally get the corresponding translation vector t using Ax = 0: (7) where A1= A [:,1:9] , A2= A [:,10:12] , is flattened by R.

2. The single-view pose estimation method based on multi-modal input and attention mechanism according to claim 1, characterized in that, Step 1 includes: A fully connected graph ε is constructed with key points as nodes: is a network that predicts the graph edge vector along the graph, using ResNet-18 as the backbone network; ε explicitly represents the displacement between each pair of key points, and |ε| represents the number of edges in the predefined graph, so predicting a third intermediate representation S reflecting the symmetric correspondence of pixels, to reflect the potential reflection symmetry of the object; extending the network architecture of FlowNet, fusing the dense pixel flow predicted by FlowNet and the mask map predicted by PVNet, and predicting the symmetric correspondence of each pixel in the mask region. loss l1 of the loss l1, loss l2 of the loss l2, loss l3 of the loss l3 are trained using the smooth l1 loss in Fast RCNN; to reflect the importance of different intermediate representations on the pose estimation network effect, the losses of the three intermediate representations are weighted using x, y, z parameters, where x+y+z=1, so the total loss is: L=xl1+yl2+zl3 (1) Channel attention module, will and Each residual block output dimension is The tensor F is input into the attention module, an average pooling operation is performed, the channel attention module generates channel weights with a dimension of by performing one-dimensional convolution with a kernel size of n=5; the obtained weights are processed by an activation function and dimension reduction to obtain the result F', which is input into the next residual block.

3. A single-view pose estimation system based on multi-modal input and attention mechanism, characterized in that, It includes: The prediction module uses various intermediate representations to express geometric information in an RGB image, and an attention mechanism is introduced to improve network training efficiency; The first prediction network The second prediction network The third prediction network And a fully connected network, a channel attention module is embedded between each down-sampling module of the three prediction networks; Using PVNet as the backbone network, it is a key point-based pose estimation network that predicts k visible and invisible key points using the voting method; the prediction network To optimize the object pose; The pose regression module obtains the intermediate representation result obtained by the prediction module, combines the key points, edge vectors and dense pixel-by-pixel correspondence relationship information, and regresses the 6D pose of the object from the intermediate representation result through EPnP calculation and singular value decomposition. The pose regression module takes the intermediate representation {κ,ε,S} predicted by each network of the prediction module as input, and outputs the 6D pose R of the object I: I ∈SO(3), The pose regression module: The three-dimensional ground truth coordinates of the key points in the standard coordinate system are denoted as 1≤k≤|κ|, the edge vector ground truth is denoted as 1≤e≤|ε|, the key point coordinates output by the prediction module are denoted as 1≤k1≤k≤|κ|; the edge vector is denoted as 1≤e≤|ε|; the symmetric correspondence is denoted as 1≤s≤|S|; for the convenience of calculation, the homogeneous coordinates are used and corresponding to p k , v e , q s,1 and q s,2 , these homogeneous coordinates are normalized by known camera intrinsic parameters; The EPnP algorithm is used, and the 6D pose of the object is calculated in combination with the constraints of the intermediate representation; first, the following difference vectors are introduced for the three predicted elements: where e s and e t is an end point of the edge e, is a normal to the plane of reflection symmetry in the base coordinate system; Secondly, formula (2) is converted into the form of A1x, A2x and A3x, formula (3) is converted into A4x, A5x and A6x in the same way, and formula (4) is converted into A7x; A1, A2, A3, A4, A5, A6 are combined into A; in order to describe the relationship between the predicted value and the true value, a linear system in the form of Ax=0 is introduced, where A is a matrix with a dimension of (3|κ|+3|ε|+|S|)×12; x is a vector containing a rotation matrix R and a translation vector t parameter in an affine space; Then, the EPnP algorithm is used to calculate x: (5) where v i is the i-th smallest singular value of A; ideally, N = 1, x = v i is the optimal solution; select the same N = 4 as EPnP; to compute the optimal x, use the following objective function to optimize the hidden variable λ in an alternating optimization procedure i and the rotation matrix R: (6) wherein including v i the first 9 elements; After obtaining the optimal λ i , we project it to SO(3) using SVD decomposition, i.e. we obtain the rotation matrix R = U diag(1, 1, 1) V T ; finally we obtain the corresponding translation vector t using Ax = 0.​ (7) where A1= A [:,1:9] , A2= A [:,10:12] , is flattened by R.

4. The single-view pose estimation system based on multi-modal input and attention mechanism of claim 3, wherein, In the prediction module: PVNet is used as the backbone network, which is a key point-based pose estimation network. A fully connected graph is constructed by taking key points as nodes, is a network that predicts the edge vector along the graph. The structure adopts ResNet-18 as the backbone network; ε explicitly represents the displacement between each pair of key points, and |ε| represents the number of edges in the predefined graph; ε is a fully connected graph, so The third intermediate representation S reflecting the symmetric correspondence relationship of the pixels is predicted, reflecting the potential reflection symmetry of the object; The network architecture of FlowNet is extended, and the dense pixel flow predicted by FlowNet is fused with the mask graph predicted by PVNet to predict the symmetric correspondence relationship of each pixel in the mask area. loss l1 of the loss l1, loss l2 of the loss l2, loss l3 of the loss l3 are trained using the smooth l1 loss in Fast RCNN; to reflect the importance of different intermediate representations on the pose estimation network effect, the losses of the three intermediate representations are weighted using x, y, z parameters, where x+y+z=1; therefore the total loss is: L=xl1+yl2+zl3 (1) Channel attention module, will and Each residual block output dimension is The tensor F is input into the attention module, an average pooling operation is performed, the channel attention module generates channel weights with a dimension of by performing one-dimensional convolution with a kernel size of n=5; the obtained weights are processed by an activation function and dimension reduction to obtain the result F', which is input into the next residual block.

Images