Object 6D attitude estimation method based on 2D-3D correspondence

Abstract

The invention discloses an object 6D attitude estimation method based on 2D-3D correspondence, and relates to the technical field of two-stage object 6D attitude estimation. Comprising the following steps: S1, data acquisition; s2, a data preprocessing step; s3, a model building step; s4, a model training step; and S5, a model test step. According to the invention, three-dimensional translation and three-dimensional rotation of the target object can be detected in a complex scene and in the presence of shielding, and the detection effect and the detection precision are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of two-stage object 6D posture estimation, and in particular to a method for object 6D posture estimation based on 2D-3D correspondence. Background Art

[0002] At present, in industrial environments such as industrial automation, augmented reality, and unmanned driving, the 6D pose estimation of the target plays a critical role. Many 6D pose estimation algorithms based on deep learning rely on the perspective projection transformation algorithm (PnP) and the random sampling consensus algorithm (RANSAC), which makes the network training and prediction stages relatively time-consuming. Therefore, how to accurately estimate the 6D pose of the object under different lighting environments with changes in brightness and color, different background noise, occlusion, and other problems has become a research direction that urgently needs to be solved and has attracted much attention.

[0003] According to the latest domestic and international research surveys, the three main method paradigms for solving the problem of 6D pose estimation of objects are: 6D pose estimation based on feature correspondence, 6D pose estimation based on pixel voting, and 6D pose estimation based on regression. The feature correspondence-based method is to correspond the 2D pixel points of the input RGB image to the 3D coordinates of its 3D model when the complete 3D model of the target is known, and then restore the pose of the target from the 2D-3D correspondence relationship. The pixel voting-based method uses the feature points corresponding to each pixel or 3D point to vote, and then obtains the 2D-3D correspondence relationship. The regression-based 6D pose estimation directly estimates the 6D pose of the target through a neural network.

[0004] Therefore, it is an urgent problem for those skilled in the art to propose a 6D posture estimation method for an object based on 2D-3D correspondence to solve the difficulties existing in the prior art. Summary of the invention

[0005] In view of this, the present invention provides a 6D posture estimation method for objects based on 2D-3D correspondence, which can detect the three-dimensional translation and three-dimensional rotation of target objects in complex scenes and with occlusion, and effectively improve the detection effect and detection accuracy.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for estimating 6D posture of an object based on 2D-3D correspondence comprises the following steps:

[0008] S1. Data acquisition steps: We selected the public dataset LINEMOD for experiments, including two versions: LINEMOD and Occlusion LINEMOD.

[0009] S2, data preprocessing step: use the Occlusion LINEMOD dataset to re-annotate the LINEMOD dataset, add occlusion conditions and lighting changes on the basis of the LINEMOD dataset, and obtain a training dataset containing occlusion conditions between objects;

[0010] S3, model building step: based on the training data set including the occlusion between objects obtained in S2, a 6D pose estimation model of the object based on 2D-3D correspondence is established;

[0011] S4, model training step: training the object 6D posture estimation model based on 2D-3D correspondence obtained in S3 to obtain a trained object 6D posture estimation model;

[0012] S5, model testing step: Based on the trained object 6D pose estimation model obtained in S4, a simulation test is performed, and the object 6D pose estimation model is evaluated using 2D projection metrics and the average 3D distance of model points as performance indicators.

[0013] In the above method, optionally, the specific contents in S3 include:

[0014] S301, extracting features from the training data set obtained in S2 based on backbone to obtain feature information of the training data set;

[0015] S302, constructing the true value of the 3D coordinate graph based on the feature information of the data set obtained in S301 and establishing a 2D-3D correspondence relationship;

[0016] S303, setting a scale-invariant translation estimation for directly predicting translation from an image;

[0017] S304, setting a dual-branch network structure based on decoupling 3D rotation and 3D translation to perform feature extraction; wherein, one branch is a rotation prediction branch RotNet, which is used to estimate the three-dimensional rotation R; and the other is a translation prediction branch TransNet, which is used to estimate the three-dimensional translation T.

[0018] In the above method, optionally, the feature extraction content includes:

[0019] Use the object detector to perform preliminary detection on the target object to obtain the position of the target object in the image, that is, the region of interest. The obtained region of interest is expanded into a square, and then the corresponding expanded region of interest is scaled and sent to the backbone to extract the features of the region of interest.

[0020] In the above method, optionally, the specific steps in S302 include:

[0021] S3021: Projecting the three-dimensional object model onto the image plane using the true value of the rotation matrix and the translation matrix to create the true value of the 3D coordinate map;

[0022] S3022: Establish a 2D-3D correspondence relationship and map the pixels in the coordinate map to the RGB image.

[0023] Furthermore, the specific steps in S303 include:

[0024] S3031 calculates information of the local image block obtained by sampling in the global image, including: position and size;

[0025] S3032 introduces an additional translation estimation network into the backbone network to predict scale-invariant translation.

[0026] First, calculate the information T of the local image block obtained by sampling in the global image G (including position C x,y Then, an additional translation estimation network is introduced in the backbone network to predict the scale-invariant translation T S =(Δ x ,Δ y ,t z ). Δ x and Δ y Represents the offset from the center of the bounding box to the center of the object. The training network regresses not the absolute offset, but the relative offset, as shown in the following formula:

[0027]

[0028] where t z is the scaled depth, and this relative offset is constant for sampling (i.e., the scale does not change). Finally, we can combine T S With T G To solve the translation T = (T x ,T y ,T z ), as shown in the following formula;

[0029]

[0030] In the above method, optionally, the RotNet branch in S304 maps the feature map to a size of 4×64×64 as the RotNet branch after three upsampling operations;

[0031] The output of RotNet consists of two parts; the first part is a prediction mask of size 1×64×64, which is used to penalize incorrect detection points when calculating the loss; the second part is a predicted 3D coordinate map of size 3×64×64, where each pixel of the coordinate map corresponds to the 3D coordinate point of the target object; after obtaining the predicted 3D coordinate map and establishing a 2D-3D correspondence, it will be used as the input of the PnP algorithm, which parses the 2D-3D correspondence to obtain the rotation matrix R of the target object.

[0032] In the above method, optionally, in the TransNet branch, after the feature map extracted by the backbone passes through three convolutional layers and three fully connected layers, the offset of the center of the target object relative to the optical center is output to obtain the translation matrix T.

[0033] The above method is optional. The loss of the final detection head network includes three parts, namely:

[0034] L1 loss of the 3D coordinate map output by RotNet:

[0035]

[0036] L1 loss of confidence map output by RotNet:

[0037]

[0038] The L2 loss of the target object center offset relative to the optical center output by TransNet:

[0039]

[0040] In the above method, optionally, the specific contents of S4 include:

[0041] The 6D pose estimation model is trained in three stages. The first stage is to train the RotNet branch alone, the second stage is to train the TransNet branch alone, and the third stage is to train the RotNet branch and the TransNet branch together to obtain a 6D pose estimation model of objects based on 2D-3D correspondence.

[0042] It can be seen from the above technical solution that, compared with the prior art, the object 6D posture estimation method based on 2D-3D correspondence of the present invention has the following beneficial effects:

[0043] 1) Methods based on 2D-3D correspondence can usually effectively estimate the 6D pose of objects in complex scenes, especially when the 3D model is known and the features are clear. This method is efficient and accurate.

[0044] 2) Since the 3D model contains the global information of the object, even if part of the 2D image is occluded, the 6D pose of the object can still be well estimated through the 2D-3D correspondence of other visible parts.

[0045] 3) Based on the existing 2D-3D correspondence method, deep learning technology can be combined to further improve the accuracy and robustness of posture estimation and adapt to more diverse scenes and object types. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0047] Figure 1 The present invention provides a method for estimating 6D posture of an object based on 2D-3D correspondence. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0050] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or devices, etc.

[0051] Reference Figure 1 As shown, the present invention discloses a method for estimating 6D posture of an object based on 2D-3D correspondence, comprising the following steps:

[0052] S1. Data acquisition steps: We selected the public dataset LINEMOD for experiments, including two versions: LINEMOD and Occlusion LINEMOD.

[0053] S2, data preprocessing step: use the Occlusion LINEMOD dataset to re-annotate the LINEMOD dataset, add occlusion conditions and lighting changes on the basis of the LINEMOD dataset, and obtain a training dataset containing occlusion conditions between objects;

[0054] S3, model building step: based on the training data set including the occlusion between objects obtained in S2, a 6D pose estimation model of the object based on 2D-3D correspondence is established;

[0055] S4, model training step: training the object 6D posture estimation model based on 2D-3D correspondence obtained in S3 to obtain a trained object 6D posture estimation model;

[0056] S5, model testing step: Based on the trained object 6D pose estimation model obtained in S4, a simulation test is performed, and the object 6D pose estimation model is evaluated using 2D projection metrics and the average 3D distance of model points as performance indicators.

[0057] In a specific embodiment, it relates to an object 6D pose estimation algorithm based on 2D-3D correspondence, which aims to estimate the pose of the object in three-dimensional space from a monocular image.

[0058] 1) Technical solution overview

[0059] This example uses a trained convolutional neural network to predict the 3D position and posture of an object. The input of the network is an RGB image, and the output is the 6D posture parameters of the object. The posture estimation is further optimized by projecting the 3D model of the object onto the image plane and minimizing the projection error.

[0060] 2) Specific implementation process

[0061] 1. Data preparation

[0062] Model construction: Use synthetic data generation tools or labeled image datasets in real scenes to generate training samples including object pose information (position and rotation). The images in the training set contain target objects from different perspectives and are annotated with their corresponding 6D pose information.

[0063] Data augmentation: Use data augmentation techniques (such as rotation, scaling, noise addition, etc.) to expand the dataset to improve the generalization ability of the algorithm.

[0064] 2. Network Design

[0065] Network structure: ResNet50 is used as the basic feature extraction network in the embodiment. The output feature map is finally output as a dense 2D-3D corresponding mapping relationship through the rotating head and the translating head part. Then, the 6D posture of the object is optimized from the 2D-3D corresponding mapping relationship through the PnP algorithm.

[0066] Loss function: L1 loss is used to calculate the error between the 3D coordinate map predicted by the network and the mask and the true label, and L2 loss is used to calculate the offset loss of the target object center predicted by the network relative to the optical center. The specific loss function includes three parts of error.

[0067]

[0068] Among them, α, β and γ are weight parameters.

[0069] 3. Training process

[0070] Initialization: Use a pre-trained model (such as a model trained on ImageNet) to initialize the weights of the convolutional layers.

[0071] Training details: The Adam optimizer is used, and the learning rate is initially set to 0.001 and decays every 10 epochs. During the training process, the error of the validation set is monitored to prevent overfitting.

[0072] Training iterations: After 300 epochs of training, the model with the smallest error in the validation set is selected as the final pose estimation model.

[0073] 4. Reasoning process

[0074] Image input: The target image is read from the database and input into the trained 6D pose estimation model.

[0075] Posture prediction: The model outputs the 3D position and posture angle of the object. The output results are combined with the camera intrinsic parameters to obtain the precise position and posture of the object in the camera coordinate system.

[0076] 5. Post-processing

[0077] Posture correction: For multi-object scenes, redundant posture estimates are filtered out through the non-maximum suppression algorithm, and the prediction results with the highest confidence are retained.

[0078] 3D reconstruction and visualization: Using the estimated 6D pose, the 3D model of the object is reconstructed and projected onto the image plane for visualization.

[0079] 3) Experimental verification

[0080] The effectiveness of the 6D pose estimation algorithm proposed in this embodiment is verified through experiments on the standard LINEMOD dataset. The experimental results show that the algorithm is superior to the existing methods in terms of position accuracy and pose estimation accuracy.

[0081] Specifically, data preprocessing: the public dataset LINEMOD is used for experiments, including two versions: LINEMOD and Occlusion LINEMOD. LINEMOD consists of 13 low-texture objects in 13 videos. It contains 19,000 rendered training images, of which 1,313 images are for each object. The Occlusion LINEMOD dataset is a re-annotation of the LINEMOD dataset. Occlusion and illumination changes are added to the LINEMOD dataset. Each image in the dataset contains occlusion between objects. The dataset contains a total of 1,400 images and 8 objects. The training sets of LINEMOD and Occlusion LINEMOD are combined as the total training set. The test set of LINEMOD is used for performance evaluation. All categories in LINEMOD are: ('ape', 'bench vice', 'camera', 'can', 'cat', 'driller', 'duck', 'eggbox', 'glue', 'hole punch', 'iron', 'lamp', 'phone').

[0082] Furthermore, the specific contents of S3 include:

[0083] S301, extracting features from the training data set obtained in S2 based on backbone to obtain feature information of the training data set;

[0084] S302, constructing the true value of the 3D coordinate graph based on the feature information of the data set obtained in S301 and establishing a 2D-3D correspondence relationship;

[0085] S303, setting a scale-invariant translation estimation for directly predicting translation from an image;

[0086] S304, setting a dual-branch network structure based on decoupling 3D rotation and 3D translation to perform feature extraction; wherein, one branch is a rotation prediction branch RotNet, which is used to estimate the three-dimensional rotation R; and the other is a translation prediction branch TransNet, which is used to estimate the three-dimensional translation T.

[0087] Furthermore, feature extraction includes:

[0088] Use the object detector to perform preliminary detection on the target object to obtain the position of the target object in the image, that is, the region of interest. The obtained region of interest is expanded into a square, and then the corresponding expanded region of interest is scaled and sent to the backbone to extract the features of the region of interest.

[0089] Specifically, based on the dual-branch network structure that decouples 3D rotation and 3D translation, one branch is the rotation prediction branch RotNet, and the other is the translation prediction branch TransNet. The rotation prediction branch RotNet is used to estimate the three-dimensional rotation R, and the translation prediction branch TransNet is used to estimate the three-dimensional translation T. The two branches share a feature extraction network. The process of the feature extraction network is as follows:

[0090] Use an off-the-shelf object detector to perform preliminary detection of the target object to obtain the location of the target object in the image, that is, the region of interest (RoI). The corresponding RoI is expanded into a square. Then this square RoI is scaled to a size of 256×256×3 and sent to the backbone to extract the features of the RoI. Using ResNet34 as the backbone, perform preliminary feature extraction on the RoI of size 256×256×3 output by the object detector to obtain a feature map of size 512×8×8. After backbone feature extraction, the obtained feature map is input into the two branches of RotNet and TransNet.

[0091] Furthermore, the specific steps in S302 include:

[0092] S3021: Projecting the three-dimensional object model onto the image plane using the true value of the rotation matrix and the translation matrix to create the true value of the 3D coordinate map;

[0093] S3022: Establish a 2D-3D correspondence relationship and map the pixels in the coordinate map to the RGB image.

[0094] Specifically, the true value of the rotation matrix and the translation matrix are used to project the 3D object model onto the image plane to create the true value of the 3D coordinate map. First, the points of the 3D object model are converted from the world coordinate system to the camera coordinate system, which can be calculated according to the following formula

[0095] P C =P W R gt +T gt

[0096] Where Pw is the point of the 3D object model in the camera coordinate system, Pc is the point of the 3D object model in the world coordinate system, Rgt and Tgt are the true values ​​of the rotation matrix and translation matrix respectively. Then the point of the 3D object model in the camera coordinate system is mapped to the image plane. The mapping relationship is shown in the following formula

[0097]

[0098] Where (i, j) is the pixel coordinate of the image plane, f x , f y , c x , c y is the internal parameter of the camera. The true value of the 3D coordinate graph can be obtained by f(i,j)=P W To express;

[0099] The second step is to establish the 2D-3D correspondence, which requires mapping the pixels in the coordinate map to the RGB image without losing accuracy. The center and size of the object in the RGB image are specified as (c u ,c v )and The center and size of the object in the coordinate diagram are (c i ,c j ) and (S x ,S y ). For a pixel (i, j) in the coordinate map, the corresponding pixel (u, v) in the RGB image can be calculated according to the following formula;

[0100]

[0101] Furthermore, the specific steps in S303 include:

[0102] S3031 calculates information of the local image block obtained by sampling in the global image, including: position and size;

[0103] S3032 introduces an additional translation estimation network into the backbone network to predict scale-invariant translation.

[0104] First, calculate the information T of the local image block obtained by sampling in the global image G (including position C x,y Then, an additional translation estimation network is introduced in the backbone network to predict the scale-invariant translation T S =(Δ x ,Δ y ,t z ). Δ x and Δ yRepresents the offset from the center of the bounding box to the center of the object. The training network regresses not the absolute offset, but the relative offset, as shown in the following formula:

[0105]

[0106] where t z is the scaled depth, and this relative offset is constant for sampling (i.e., the scale does not change). Finally, we can combine T S With T G To solve the translation T = (T x ,T y ,T z ), as shown in the following formula;

[0107]

[0108] Furthermore, after three upsampling operations, the RotNet branch in S304 maps the feature map to a size of 4×64×64 as the RotNet branch;

[0109] The output of RotNet consists of two parts; the first part is a prediction mask of size 1×64×64, which is used to penalize incorrect detection points when calculating the loss; the second part is a predicted 3D coordinate map of size 3×64×64, where each pixel of the coordinate map corresponds to the 3D coordinate point of the target object; after obtaining the predicted 3D coordinate map and establishing a 2D-3D correspondence, it will be used as the input of the PnP algorithm, which parses the 2D-3D correspondence to obtain the rotation matrix R of the target object.

[0110] Furthermore, in the TransNet branch, after the feature map extracted by the backbone passes through three convolutional layers and three fully connected layers, the offset of the center of the target object relative to the optical center is output to obtain the translation matrix T.

[0111] Furthermore, the loss of the final detection head network consists of three parts:

[0112] L1 loss for 3D coordinate graphs:

[0113]

[0114] L1 loss of confidence map:

[0115]

[0116] The L2 loss of the target object center relative to the optical center:

[0117]

[0118] Furthermore, the specific contents of S4 include:

[0119] The 6D pose estimation model is trained in three stages. The first stage is to train the RotNet branch alone, the second stage is to train the TransNet branch alone, and the third stage is to train the RotNet branch and the TransNet branch together to obtain a 6D pose estimation model of objects based on 2D-3D correspondence.

[0120] Specifically, the test model: In the target 6D pose estimation task, the 2D projection metric and the average 3D distance (ADD(-S)) metric of the model points are two important performance indicators. The 2D projection metric is to calculate the average distance between the estimated pose and the actual pose information of the object on the projection of the 3D model points. If the 2D reprojection error is lower than a certain threshold, it is concluded that the estimated pose is correct. The calculation formula of the 2D projection metric is:

[0121]

[0122] Where M represents the set of 3D model points, m is the number of points in the set. R and T are the rotation and translation corresponding to the real pose. To estimate the rotation and translation of the pose, K is the camera intrinsic parameter.

[0123] The ADD metric is the average distance between the estimated pose information and the 3D transformation model points of the object's true position and direction information. It is aimed at the average distance between the 3D points of the model. The threshold is set to 10% of the model diameter. If the average distance is less than the threshold, the pose estimation is considered correct. The calculation formula of the ADD metric is:

[0124]

[0125] At the same time, the ADD-S metric is used to calculate the symmetric object as shown in the following formula, that is, the average distance is calculated based on the distance between the nearest points.

[0126]

[0127] Where M represents the set of 3D model points, m is the number of points in the set. R and T are the rotation and translation corresponding to the real pose. and is the rotation and translation of the estimated pose.

[0128] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0129] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, computer software, or a combination of both.

[0130] In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0131] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for estimating 6D posture of an object based on 2D-3D correspondence, characterized in that: The following steps are involved: S1. Data acquisition steps: We selected the public dataset LINEMOD for experiments, including two versions: LINEMOD and OcclusionLINEMOD. S2, data preprocessing step: use the Occlusion LINEMOD dataset to re-annotate the LINEMOD dataset, add occlusion conditions and lighting changes on the basis of the LINEMOD dataset, and obtain a training dataset containing occlusion conditions between objects; S3, model building step: based on the training data set including the occlusion between objects obtained in S2, a 6D pose estimation model of the object based on 2D-3D correspondence is established; S4, model training step: training the object 6D posture estimation model based on 2D-3D correspondence obtained in S3 to obtain a trained object 6D posture estimation model; S5, model testing step: Based on the trained object 6D pose estimation model obtained in S4, a simulation test is performed, and the object 6D pose estimation model is evaluated using 2D projection metrics and the average 3D distance of model points as performance indicators.

2. The method for estimating 6D posture of an object based on 2D-3D correspondence according to claim 1, characterized in that: The specific contents of S3 include: S301, extracting features from the training data set obtained in S2 based on backbone to obtain feature information of the training data set; S302, constructing the true value of the 3D coordinate graph based on the feature information of the data set obtained in S301 and establishing a 2D-3D correspondence relationship; S303, setting a scale-invariant translation estimation for directly predicting translation from an image; S304, setting a dual-branch network structure based on decoupling 3D rotation and 3D translation to perform feature extraction; wherein, one branch is a rotation prediction branch RotNet, which is used to estimate the three-dimensional rotation R; and the other is a translation prediction branch TransNet, which is used to estimate the three-dimensional translation T.

3. The method for estimating 6D posture of an object based on 2D-3D correspondence according to claim 2, characterized in that: Feature extraction includes: Use the object detector to perform preliminary detection on the target object to obtain the position of the target object in the image, that is, the region of interest. The obtained region of interest is expanded into a square, and then the corresponding expanded region of interest is scaled and sent to the backbone to extract the features of the region of interest.

4. The method for estimating 6D posture of an object based on 2D-3D correspondence according to claim 2, characterized in that: The specific steps in S302 include: S3021: Projecting the three-dimensional object model onto the image plane using the true value of the rotation matrix and the translation matrix to create the true value of the 3D coordinate map; S3022: Establish a 2D-3D correspondence relationship and map the pixels in the coordinate map to the RGB image.

5. The method for estimating 6D posture of an object based on 2D-3D correspondence according to claim 4, characterized in that: The specific steps in S303 include: S3031 calculates information of the local image block obtained by sampling in the global image, including: position and size; S3032 introduces an additional translation estimation network into the backbone network to predict scale-invariant translation.

6. The method for estimating 6D posture of an object based on 2D-3D correspondence according to claim 5, characterized in that: In the RotNet branch in S304, after three upsampling operations, the feature map is mapped to a size of 4×64×64 as the RotNet branch; The output of RotNet consists of two parts; the first part is a prediction mask of size 1×64×64, which is used to penalize incorrect detection points when calculating the loss; The second part is a predicted 3D coordinate map of size 3×64×64, where each pixel corresponds to the 3D coordinate point of the target object. After the predicted 3D coordinate map is obtained and the 2D-3D correspondence is established, it will be used as the input of the PnP algorithm, which parses the 2D-3D correspondence to obtain the rotation matrix R of the target object.

7. The method for estimating 6D posture of an object based on 2D-3D correspondence according to claim 5, characterized in that: In the TransNet branch, after the feature map extracted by the backbone passes through three convolutional layers and three fully connected layers, the offset of the center of the target object relative to the optical center is output to obtain the translation matrix T.

8. The method for estimating 6D posture of an object based on 2D-3D correspondence according to claim 7, characterized in that: The final loss of the detection head network consists of three parts: L1 loss for 3D coordinate graphs: L1 loss of confidence map: The L2 loss of the target object center relative to the optical center:

9. The method for estimating 6D posture of an object based on 2D-3D correspondence according to claim 8, characterized in that: The specific contents of S4 include: The 6D pose estimation model is trained in three stages. The first stage is to train the RotNet branch alone, the second stage is to train the TransNet branch alone, and the third stage is to train the RotNet branch and the TransNet branch together to obtain a 6D pose estimation model of objects based on 2D-3D correspondence.

Images