End-to-end object 6D attitude estimation method based on beam adjustment method

Abstract

The invention discloses an end-to-end object 6D attitude estimation method based on a beam adjustment method, and relates to the technical field of two-stage object 6D attitude estimation. Comprises: data acquisition; preprocessing the data; establishing a model; training the model; and testing the model. According to the invention, three-dimensional translation and three-dimensional rotation of the target object can be detected in a complex scene and in the presence of shielding.

Description

Technical Field

[0001] The present invention relates to the technical field of two-stage object 6D posture estimation, and in particular to an end-to-end object 6D posture estimation method based on bundle adjustment method. Background Art

[0002] At present, 6D pose estimation of targets plays a critical role in industrial environments such as industrial automation, augmented reality, and unmanned driving. However, in practical applications, 6D pose estimation of objects is often affected by various noise factors such as background clutter, mutual occlusion of objects, and lighting changes.

[0003] According to the latest domestic and international research surveys, there are three main method paradigms for solving the problem of 6D pose estimation of objects: 6D pose estimation based on feature correspondence, 6D pose estimation based on pixel voting, and 6D pose estimation based on regression. The feature correspondence-based method corresponds the 2D pixel points of the input RGB image to the 3D coordinates of its 3D model when the complete 3D model of the target is known, and then recovers the pose of the target from the 2D-3D correspondence relationship; the pixel voting-based method uses the feature points corresponding to each pixel or 3D point to vote, and then obtains the 2D-3D correspondence relationship; the regression-based 6D pose estimation directly estimates the 6D pose of the target through a neural network.

[0004] The two-stage 6D pose estimation method benefits from the relationship between corresponding feature points to obtain 6D pose, has good 6D pose estimation results, and is widely used in practice. Although this method based on feature point correspondence is effective, this method usually uses a proxy target for corresponding regression for training, which does not necessarily reflect the actual situation of the optimized 6D pose error; this indirect method is not differentiable for 6D pose estimation, which limits the learning ability of the network; using PnP / RANSAC iterations when processing dense feature point correspondence is relatively time-consuming.

[0005] Therefore, it is an urgent problem for those skilled in the art to propose an end-to-end object 6D posture estimation method based on bundle adjustment to solve the difficulties existing in the prior art. Summary of the invention

[0006] In view of this, the present invention provides an end-to-end object 6D posture estimation method based on bundle adjustment method, which can detect the three-dimensional translation and three-dimensional rotation of the target object in complex scenes and in the presence of occlusion.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A method for estimating 6D pose of an end-to-end object based on bundle adjustment method comprises the following steps:

[0009] S1. Data acquisition: We use the public dataset LINEMOD for experiments, including two versions: LINEMOD and OcclusionLINEMOD.

[0010] S2. Data preprocessing: The training sets of LINEMOD and Occlusion LINEMOD are combined as the total training set, and the test set of LINEMOD is used for performance evaluation;

[0011] S3. Model building: Based on the semantic segmentation network structure of multi-scale convolutional feature fusion and the end-to-end PnP network structure based on the bundle adjustment method, the camera pose is adjusted by minimizing the reprojection error of the observed image feature points, and an end-to-end object 6D pose estimation model based on the bundle adjustment method is established;

[0012] S4, training model: in the first stage, the semantic segmentation network is trained separately, and in the second stage, the PnP network based on the bundle adjustment method is trained separately. Finally, the two networks are trained together to obtain a trained end-to-end object 6D pose estimation model based on the bundle adjustment method.

[0013] S5. Test model: In the target 6D pose estimation task, the 2D projection metric and the average 3D distance of the model points are calculated as performance indicators to evaluate the object 6D pose estimation model.

[0014] The above method is optional. LINEMOD in S1 consists of 13 low-texture objects in 13 videos, including 19,000 rendered training pictures, of which 1,313 pictures are for each object. The Occlusion LINEMOD dataset is a re-annotation of the LINEMOD dataset. Occlusion and illumination changes are added to the LINEMOD dataset. Each picture in the dataset contains the occlusion between objects. The dataset contains a total of 1,400 pictures and 8 objects. All categories in LINEMOD are: 'ape', 'bench vice', 'camera', 'can', 'cat', 'driller', 'duck', 'eggbox', 'glue', 'hole punch', 'iron', 'lamp', 'phone'.

[0015] The above method is optional. In S3, the semantic segmentation network structure based on multi-scale convolutional feature fusion uses a symmetrical encoder-decoder structure - UNet network as the basic architecture. The first half of the semantic segmentation network structure is an encoder structure for feature extraction, which consists of convolution operations and downsampling operations. The convolution structure used is unified as a 3×3 convolution kernel, and the second half is a decoder structure that restores the resolution by upsampling.

[0016] In the above method, optionally, the semantic segmentation network is composed of an encoder and three decoder structures, the encoder network includes a multi-scale convolution feature fusion module and a pooling layer, and after each downsampling, the number of extracted feature channels increases and the size is reduced by half; the decoder network is composed of a multi-scale convolution feature fusion module and upsampling by bilinear interpolation, and the bilinear interpolation performs linear interpolation once in the horizontal and vertical directions;

[0017] The three decoder parts of the semantic segmentation network finally output U and V channels and ID masks respectively. The U and V channels are combined to form a dense 2D-3D corresponding mapping relationship. The data set preprocessing part has mapped the model to a two-dimensional UV map: given a three-dimensional surface S(u,v), use parameters u and v to represent points on the surface, find an inverse mapping P from the three-dimensional surface to the two-dimensional plane: S(u,v)→(u,v), where P is a mapping from a three-dimensional surface to a two-dimensional UV plane; by defining an energy function E, it reflects the degree of various distortions in the mapping process, minimizes these distortion energies, and finds the optimal UV mapping:

[0018] E=∫ S (λ1(u,v)+λ2(u,v))dA

[0019] Where λ1 and λ2 represent the maximum and minimum eigenvalues ​​when unfolded on the UV plane, respectively, and dA is the differential area element on the three-dimensional surface.

[0020] Each point on the model corresponds to a unique UV value. The UV value of each pixel is predicted and the corresponding 3D point on the model is found.

[0021] The UV value decoder outputs a 256×256×3 dimensional prediction result. The prediction value of each pixel in each channel is the probability value of the pixel corresponding to the channel number. The channel with the largest probability value is taken as the prediction result of the pixel. Finally, the 3D model point corresponding to each pixel in the image is obtained based on the UV value prediction result and combined with the category information.

[0022] The above method,optionally, an end-to-end PnP network structure based on bundle adjustment in S3,considers n three-dimensional space points P in the target coordinate system and their two-dimensional projection p on the image,and calculates the 6D pose R,t relative to the camera, and the Lie algebra representation of the transformation matrix is ​​ξ;

[0023] The coordinates of n three-dimensional space points are P i =[X i ,Y i ,Z i ] T , the pixel coordinate of its two-dimensional projection is p i =[u i,v i ] T , according to three-dimensional transformation and Lie algebra, we get the following formula:

[0024]

[0025] Its matrix form is: s i p i =Kexp(ξ ∧ ) i ;

[0026] There is a reprojection error on both sides of the equation, which describes the difference between the projection position of the three-dimensional space point on the image plane and the actual observed position of the image feature point.

[0027] The above method optionally randomly initializes an initial 6D pose exp(ξ ∧ ), summing the reprojection errors to construct the least squares problem, as shown in the following formula:

[0028]

[0029] Finally, the best 6D pose is found through optimization algorithm;

[0030] Following the traditional principle of bundle adjustment, the LM optimization algorithm is used to optimize the error equation, and a method is designed to soften this if-else decision, giving a differentiable LM module: by inputting the current target value into the MLP network to predict μ, the LM module is iteratively optimized by interpreting the intermediate variables as network nodes.

[0031] In the above method, optionally, calculating the update Δξ of the solution space from the 2D-3D feature correspondence F and the current solution space ξ includes:

[0032] Calculate the reprojection error E(ξ) for all 2D-3D feature correspondences;

[0033]

[0034] Calculate the Jacobian matrix J(ξ), Hessian matrix J(ξ) T J(ξ) and its diagonal matrix D(ξ);

[0035] Use global average pooling to integrate the absolute values ​​of E(ξ) over all pixels to obtain a 128-dimensional feature vector, which is input into an MLP subnetwork consisting of four fully connected layers to predict μ;

[0036] Finally, the update Δξ of the current solution space is calculated according to the formula:

[0037] Δξ=-(J(ξ)T J(ξ)+μD(ξ)) -1 J(ξ) T E(ξ).

[0038] In the above method, optionally, μ is regarded as an intermediate variable, and each step of the LM optimization algorithm is represented as a function g, which acts on the 2D-3D feature correspondence C and the solution ξ obtained in the previous iteration, that is, Δξ=g(ξ; F);

[0039] The solution after the kth iteration can be expressed as follows:

[0040]

[0041] in represents a parameter update, which for the camera pose of SE(3) represents an exponential mapping that is differentiable with respect to the 2D-3D feature correspondence C, allowing backpropagation through the entire feature learning network.

[0042] The above method, optionally, tests the model in S5: In the target 6D pose estimation task, the calculation of two performance indicators, 2D projection metric and average 3D distance of model points (ADD(-S)):

[0043] The calculation formula for the 2D projection metric is:

[0044]

[0045] Where M represents the set of 3D model points, m is the number of points in the set, R and T are the rotation and translation corresponding to the true pose, To estimate the rotation and translation of the pose, K is the camera intrinsic parameter;

[0046] The calculation formula for the ADD metric is:

[0047]

[0048] For symmetric objects, the ADD-S metric is used, that is, the average distance is calculated based on the distance between the nearest points:

[0049]

[0050] Where M represents the set of 3D model points, m is the number of points in the set, R and T are the rotation and translation corresponding to the true pose, and is the rotation and translation of the estimated pose.

[0051] It can be seen from the above technical solution that, compared with the prior art, the present invention provides an end-to-end object 6D posture estimation method based on bundle adjustment method, which has the following beneficial effects:

[0052] 1) Through iterative optimization, the projection error is minimized, which can provide higher estimation accuracy. It takes into account the reprojection error of all points and optimizes it globally to reduce the propagation and amplification of the error;

[0053] 2) Through global optimization, this method is more robust to noise and mismatching. Even in the case of partial mismatching, this method can suppress the impact of these errors through global optimization;

[0054] 3) In scenes with object occlusion or dynamic changes, this method can handle partially failed feature point matching through global consistency, thereby still maintaining a high accuracy of pose estimation;

[0055] 4) After the present invention improves the traditional PnP algorithm, in the LINEMOD data set, the ADD(-S) metric is increased from 96.2% to 97.7%, achieving a high accuracy rate; the present invention can effectively calculate the 6D posture from the 2D-3D feature correspondence. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0057] Figure 1 A flow chart of an end-to-end object 6D posture estimation method based on bundle adjustment provided by the present invention;

[0058] Figure 2 A schematic diagram of an end-to-end object 6D posture estimation method based on bundle adjustment provided by the present invention;

[0059] Figure 3 A schematic diagram of the network structure of a semantic segmentation module of an end-to-end object 6D posture estimation method based on bundle adjustment provided by the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0062] Reference Figure 1 and Figure 2 As shown, the present invention discloses an end-to-end object 6D posture estimation method based on bundle adjustment method, comprising the following steps:

[0063] S1. Data acquisition: We use the public dataset LINEMOD for experiments, including two versions: LINEMOD and OcclusionLINEMOD.

[0064] S2. Data preprocessing: The training sets of LINEMOD and Occlusion LINEMOD are combined as the total training set, and the test set of LINEMOD is used for performance evaluation;

[0065] S3. Model building: Based on the semantic segmentation network structure of multi-scale convolutional feature fusion and the end-to-end PnP network structure based on the bundle adjustment method, the camera pose is adjusted by minimizing the reprojection error of the observed image feature points, and an end-to-end object 6D pose estimation model based on the bundle adjustment method is established;

[0066] S4, training model: in the first stage, the semantic segmentation network is trained separately, and in the second stage, the PnP network based on the bundle adjustment method is trained separately. Finally, the two networks are trained together to obtain a trained end-to-end object 6D pose estimation model based on the bundle adjustment method.

[0067] S5. Test model: In the target 6D pose estimation task, the 2D projection metric and the average 3D distance of the model points are calculated as performance indicators to evaluate the object 6D pose estimation model.

[0068] Furthermore, LINEMOD in S1 consists of 13 low-texture objects in 13 videos, including 19,000 rendered training images, of which 1,313 images are for each object; the Occlusion LINEMOD dataset is a re-annotation of the LINEMOD dataset, with occlusion and illumination changes added to the LINEMOD dataset. Each image in the dataset contains the occlusion between objects. The dataset contains a total of 1,400 images and 8 objects. In LINEMOD, all categories are: 'ape', 'bench vice', 'camera', 'can', 'cat', 'driller', 'duck', 'eggbox', 'glue', 'holepunch', 'iron', 'lamp', and 'phone'.

[0069] Further, see Figure 3 As shown in Figure 3, the semantic segmentation network structure based on multi-scale convolutional feature fusion uses a symmetrical encoder-decoder structure, the UNet network, as the basic architecture. The first half of the semantic segmentation network structure is an encoder structure for feature extraction, which consists of convolution operations and downsampling operations. The convolution structure used is unified as a 3×3 convolution kernel. The second half is a decoder structure that restores the resolution by upsampling.

[0070] Furthermore, the semantic segmentation module in the overall network architecture mainly realizes the 2D-3D corresponding mapping.

[0071] Furthermore, the semantic segmentation network consists of an encoder and three decoder structures. The encoder network includes a multi-scale convolutional feature fusion module and a pooling layer. After each downsampling, the number of extracted feature channels increases and the size is reduced by half. The decoder network consists of a multi-scale convolutional feature fusion module and upsampling through bilinear interpolation. Bilinear interpolation performs linear interpolation once in the horizontal and vertical directions.

[0072] The three decoder parts of the semantic segmentation network finally output U and V channels and ID masks respectively. The U and V channels are combined to form a dense 2D-3D corresponding mapping relationship. The data set preprocessing part has mapped the model to a two-dimensional UV map: given a three-dimensional surface S(u,v), use parameters u and v to represent points on the surface, find an inverse mapping P from the three-dimensional surface to the two-dimensional plane: S(u,v)→(u,v), where P is a mapping from a three-dimensional surface to a two-dimensional UV plane; by defining an energy function E, it reflects the degree of various distortions in the mapping process, such as angle distortion, area distortion, etc., minimizes these distortion energies, and finds the optimal UV mapping:

[0073] E=∫ S(λ1(u,v)+λ2(u,v))dA

[0074] Where λ1 and λ2 represent the maximum and minimum eigenvalues ​​when unfolded on the UV plane, respectively, and dA is the differential area element on the three-dimensional surface.

[0075] Each point on the model corresponds to a unique UV value. The UV value of each pixel is predicted and the corresponding 3D point on the model is found.

[0076] The UV value decoder outputs a 256×256×3 dimensional prediction result. The prediction value of each pixel in each channel is the probability value of the pixel corresponding to the channel number. The channel with the largest probability value is taken as the prediction result of the pixel. Finally, the 3D model point corresponding to each pixel in the image is obtained based on the UV value prediction result and combined with the category information.

[0077] Furthermore, in S2, an end-to-end PnP network structure based on the bundle adjustment method is proposed. In the PnP algorithm, the core idea of ​​the bundle adjustment method is to adjust the camera pose by minimizing the reprojection error of the observed image feature points.

[0078] Furthermore, considering n 3D space points P in the target coordinate system and their 2D projections p on the image, we calculate the 6D pose R,t relative to the camera. The Lie algebraic representation of the transformation matrix is ​​ξ;

[0079] The coordinates of n three-dimensional space points are P i =[X i ,Y i ,Z i ] T , the pixel coordinate of its two-dimensional projection is p i =[u i ,v i ] T , according to three-dimensional transformation and Lie algebra, we get the following formula:

[0080]

[0081] Its matrix form is: s i p i =Kexp(ξ ∧ ) i ;

[0082] There is a reprojection error on both sides of the equation, which describes the difference between the projection position of the three-dimensional space point on the image plane and the actual observed position of the image feature point.

[0083] Furthermore, we randomly initialize an initial 6D pose exp(ξ ∧ ), summing the reprojection errors to construct the least squares problem, as shown in the following formula:

[0084]

[0085] Finally, the best 6D pose is found through optimization algorithm;

[0086] Following the traditional principle of bundle adjustment, the LM optimization algorithm is used to optimize the error equation, and a method is designed to soften this if-else decision, giving a differentiable LM optimization algorithm: by inputting the current target value into the MLP network to predict μ, a single iteration of LM optimization is optimized by interpreting the intermediate variables as network nodes.

[0087] Furthermore, the update Δξ of the solution space is calculated from the 2D-3D feature correspondence F and the current solution space ξ, including:

[0088] Calculate the reprojection error E(ξ) for all 2D-3D feature correspondences;

[0089]

[0090] Calculate the Jacobian matrix J(ξ), Hessian matrix J(ξ) T J(ξ) and its diagonal matrix D(ξ);

[0091] Use global average pooling to integrate the absolute values ​​of E(ξ) over all pixels to obtain a 128-dimensional feature vector, which is input into an MLP subnetwork consisting of four fully connected layers to predict μ;

[0092] Finally, the update Δξ of the current solution space is calculated according to the formula:

[0093] Δξ=-(J(ξ) T J(ξ)+μD(ξ)) -1 J(ξ) T E(ξ).

[0094] Furthermore, μ is considered as an intermediate variable, and each step of the LM optimization algorithm is represented as a function g, which acts on the 2D-3D feature correspondence C and the solution ξ obtained in the previous iteration, that is, Δξ = g(ξ; F);

[0095] The solution after the kth iteration can be expressed as follows:

[0096]

[0097] in represents a parameter update, which for the camera pose of SE(3) represents an exponential mapping that is differentiable with respect to the 2D-3D feature correspondence C, allowing backpropagation through the entire feature learning network.

[0098] Furthermore, the model is tested in S5: In the target 6D pose estimation task, the 2D projection metric and the average 3D distance of the model points (ADD(-S)) metric are calculated for two performance indicators:

[0099] The calculation formula for the 2D projection metric is:

[0100]

[0101] Where M represents the set of 3D model points, m is the number of points in the set, R and T are the rotation and translation corresponding to the true pose, To estimate the rotation and translation of the pose, K is the camera intrinsic parameter;

[0102] The calculation formula for the ADD metric is:

[0103]

[0104] For symmetric objects, the ADD-S metric is used, that is, the average distance is calculated based on the distance between the nearest points:

[0105]

[0106] Where M represents the set of 3D model points, m is the number of points in the set, R and T are the rotation and translation corresponding to the true pose, and is the rotation and translation of the estimated pose.

[0107] In a specific embodiment, it relates to an end-to-end object 6D pose estimation algorithm based on bundle adjustment method, which aims to estimate the pose of an object in three-dimensional space from a monocular image.

[0108] 1) Technical solution overview

[0109] This example uses a trained convolutional neural network to predict the 3D position and posture of an object. The input of the network is an RGB image, and the output is the 6D posture parameters of the object. The posture estimation is further optimized by projecting the 3D model of the object onto the image plane and minimizing the projection error.

[0110] 2) Specific implementation process

[0111] 1. Data preparation

[0112] Model construction: Use synthetic data generation tools or labeled image datasets in real scenes to generate training samples including object pose information (position and rotation). The images in the training set contain target objects from different perspectives and are annotated with their corresponding 6D pose information.

[0113] Data augmentation: Use data augmentation techniques (such as rotation, scaling, noise addition, etc.) to expand the dataset to improve the generalization ability of the algorithm.

[0114] 2. Network Design

[0115] Network structure: In the embodiment, a multi-scale convolutional feature fusion network with UNet network as the basic architecture is used as the feature extraction network based on it. The output feature map is finally outputted through three decoder parts to U and V channels and ID mask respectively. The U and V channels are combined to form a dense 2D-3D corresponding mapping relationship, and then the 6D posture of the object is optimized from the 2D-3D corresponding mapping relationship through an end-to-end PnP network based on the bundle adjustment method.

[0116] Loss function: Use L1 loss to calculate the error between the network's predicted U and V channels and ID mask and the true label. The specific loss function includes two parts of error.

[0117]

[0118] Among them, α and β are weight parameters.

[0119] 3. Training process

[0120] Initialization: Use a pre-trained model (such as a model trained on ImageNet) to initialize the weights of the convolutional layers.

[0121] Training details: The Adam optimizer is used, and the learning rate is initially set to 0.001 and decays every 10 epochs. During the training process, the error of the validation set is monitored to prevent overfitting.

[0122] Training iterations: After 300 epochs of training, the model with the smallest error in the validation set is selected as the final pose estimation model.

[0123] 4. Reasoning process

[0124] Image input: The target image is read from the database and input into the trained 6D pose estimation model.

[0125] Posture prediction: The model outputs the 3D position and posture angle of the object. The output results are combined with the camera intrinsic parameters to obtain the precise position and posture of the object in the camera coordinate system.

[0126] 5. Post-processing

[0127] Posture correction: For multi-object scenes, redundant posture estimates are filtered out through the non-maximum suppression algorithm, and the prediction results with the highest confidence are retained.

[0128] 3D reconstruction and visualization: Using the estimated 6D pose, the 3D model of the object is reconstructed and projected onto the image plane for visualization.

[0129] 3) Experimental verification

[0130] The effectiveness of the 6D pose estimation algorithm proposed in this embodiment is verified through experiments on the standard LINEMOD dataset. The experimental results show that the algorithm is superior to the existing methods in terms of position accuracy and pose estimation accuracy.

[0131] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0132] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for end-to-end object 6D posture estimation based on bundle adjustment, characterized in that: The following steps are involved: S1. Data acquisition: We use the public dataset LINEMOD for experiments, including two versions: LINEMOD and Occlusion LINEMOD. S2. Data preprocessing: The training sets of LINEMOD and Occlusion LINEMOD are combined as the total training set, and the test set of LINEMOD is used for performance evaluation; S3. Model building: Based on the semantic segmentation network structure of multi-scale convolutional feature fusion and the end-to-end PnP network structure based on the bundle adjustment method, the camera pose is adjusted by minimizing the reprojection error of the observed image feature points, and an end-to-end object 6D pose estimation model based on the bundle adjustment method is established; S4, training model: in the first stage, the semantic segmentation network is trained separately, and in the second stage, the PnP network based on the bundle adjustment method is trained separately. Finally, the two networks are trained together to obtain a trained end-to-end object 6D pose estimation model based on the bundle adjustment method. S5. Test model: In the target 6D pose estimation task, the 2D projection metric and the average 3D distance of the model points are calculated as performance indicators to evaluate the object 6D pose estimation model.

2. The end-to-end object 6D posture estimation method based on bundle adjustment according to claim 1, characterized in that: LINEMOD in S1 consists of 13 low-texture objects in 13 videos and contains 19,000 rendered training images, including 1,313 images of each object. The Occlusion LINEMOD dataset is a re-annotation of the LINEMOD dataset. Occlusion and illumination changes are added to the LINEMOD dataset. Each image in the dataset contains the occlusion between objects. The dataset contains a total of 1,400 images and 8 objects. In LINEMOD, all categories are: 'ape', 'benchvice', 'camera', 'can', 'cat', 'driller', 'duck', 'eggbox', 'glue', 'hole punch', 'iron', 'lamp', and 'phone'.

3. The end-to-end object 6D posture estimation method based on bundle adjustment method according to claim 1, characterized in that: In S3, the semantic segmentation network structure based on multi-scale convolutional feature fusion uses a symmetrical encoder-decoder structure - UNet network as the basic architecture. The first half of the semantic segmentation network structure is the encoder structure for feature extraction, which consists of convolution operations and downsampling operations. The convolution structure used is unified as a 3×3 convolution kernel. The second half is a decoder structure that restores the resolution through upsampling.

4. The end-to-end object 6D posture estimation method based on bundle adjustment method according to claim 3 is characterized in that: The semantic segmentation network consists of an encoder and three decoder structures. The encoder network includes a multi-scale convolutional feature fusion module and a pooling layer. After each downsampling, the number of extracted feature channels increases and the size is reduced by half. The decoder network consists of a multi-scale convolutional feature fusion module and upsampling through bilinear interpolation. Bilinear interpolation performs linear interpolation once in the horizontal and vertical directions. The three decoder parts of the semantic segmentation network finally output U and V channels and ID masks respectively. The U and V channels are combined to form a dense 2D-3D corresponding mapping relationship. The data set preprocessing part has mapped the model to a two-dimensional UV map: given a three-dimensional surface S(u,v), use parameters u and v to represent points on the surface, find an inverse mapping P from the three-dimensional surface to the two-dimensional plane: S(u,v)→(u,v), where P is a mapping from a three-dimensional surface to a two-dimensional UV plane; by defining an energy function E, it reflects the degree of various distortions in the mapping process, minimizes these distortion energies, and finds the optimal UV mapping: E=∫ S (λ1(u,v)+λ2(u,v))dA Where λ1 and λ2 represent the maximum and minimum eigenvalues ​​when unfolded on the UV plane, respectively, and dA is the differential area element on the three-dimensional surface; Each point on the model corresponds to a unique UV value. The UV value of each pixel is predicted and the corresponding 3D point on the model is found. The UV value decoder outputs a 256×256×3 dimensional prediction result. The prediction value of each pixel in each channel is the probability value of the pixel corresponding to the channel number. The channel with the largest probability value is taken as the prediction result of the pixel. Finally, the 3D model point corresponding to each pixel in the image is obtained based on the UV value prediction result and combined with the category information.

5. The end-to-end object 6D posture estimation method based on bundle adjustment method according to claim 1, characterized in that: The end-to-end PnP network structure based on bundle adjustment in S3 considers n 3D space points P in the target coordinate system and their 2D projection p on the image, and calculates the 6D pose R,t relative to the camera. The Lie algebraic representation of the transformation matrix is ​​ξ; The coordinates of n three-dimensional space points are P i =[X i ,Y i ,Z i ] T , the pixel coordinate of its two-dimensional projection is p i =[u i ,v i ] T , according to three-dimensional transformation and Lie algebra, we get the following formula: Its matrix form is: s i p i =Kexp(ξ ∧ ) i ; There is a reprojection error on both sides of the equation, which describes the difference between the projection position of the three-dimensional space point on the image plane and the actual observed position of the image feature point.

6. The end-to-end object 6D posture estimation method based on bundle adjustment method according to claim 5, characterized in that: Randomly initialize an initial 6D posture exp(ξ ∧ ), summing the reprojection errors to construct the least squares problem, as shown in the following formula: Finally, the best 6D pose is found through the optimization algorithm.

7. The end-to-end object 6D posture estimation method based on bundle adjustment method according to claim 6, characterized in that: Computing the update Δξ of the solution space from the 2D-3D feature correspondence F and the current solution space ξ includes: Calculate the reprojection error E(ξ) for all 2D-3D feature correspondences; Calculate the Jacobian matrix J(ξ), Hessian matrix J(ξ) T J(ξ) and its diagonal matrix D(ξ); Use global average pooling to integrate the absolute values ​​of E(ξ) over all pixels to obtain a 128-dimensional feature vector, which is input into an MLP subnetwork consisting of four fully connected layers to predict μ; Finally, the update Δξ of the current solution space is calculated according to the formula: Δξ=-(J(ξ) T J(ξ)+μD(ξ)) -1 J(ξ) T E(ξ).

8. The end-to-end object 6D posture estimation method based on bundle adjustment method according to claim 7, characterized in that: Consider μ as an intermediate variable and represent each step of the LM optimization algorithm as a function g acting on the 2D-3D feature correspondence C and the solution ξ obtained in the previous iteration, i.e., Δξ=g(ξ;F); The solution after the kth iteration can be expressed as follows: where ° denotes a parameter update, which for the camera pose represents an exponential mapping that is differentiable with respect to the 2D-3D feature correspondence C, allowing backpropagation through the entire feature learning network.

9. The end-to-end object 6D posture estimation method based on bundle adjustment method according to claim 1, characterized in that: Test model in S5: In the target 6D pose estimation task, the calculation of two performance indicators is measured by 2D projection metric and average 3D distance of model points (ADD(-S)): The calculation formula for the 2D projection metric is: Among them, M represents the set of 3D model points, m is the number of points in the set, R and T are the rotation and translation corresponding to the real posture, To estimate the rotation and translation of the pose, K is the camera intrinsic parameter; The calculation formula for the ADD metric is: For symmetric objects, the ADD-S metric is used, that is, the average distance is calculated based on the distance between the nearest points: Where M represents the set of 3D model points, m is the number of points in the set, R and T are the rotation and translation corresponding to the true pose, and is the rotation and translation of the estimated pose.

Images