Object pose estimation method based on deep learning and synthetic data and related equipment

Abstract

The invention discloses an object pose estimation method based on deep learning and synthetic data and related equipment, and belongs to the technical field of computer vision. The method comprises the following steps: generating an instance segmentation and pose estimation synthesis data set by utilizing simulation modeling software and an automatic script; a YOLO11-seg model is trained, and a data set used for training a domain adaptation model is constructed in an auxiliary mode; training a simulation-to-reality domain adaptation model based on the improved CycleGAN and using the model to enhance the fidelity of the RGB image in the pose estimation synthesis data set; and training a pose estimation model based on multi-modal feature fusion and an attention mechanism by using an enhanced synthetic data set, extracting real scene target foreground RoI data by using YOLO11-seg, inputting the pose estimation model, and outputting a 6D pose of an object. According to the method, the object pose in the disordered scene can be accurately estimated, the grabbing success rate of the robot is improved, the inter-domain difference between synthetic data and real data can be effectively reduced, the data labeling cost is reduced, and the method can be suitable for industrial scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for estimating an object pose based on deep learning and synthetic data and related equipment. Background Art

[0002] With the rapid development of digital technology, information technology, and artificial intelligence, intelligent manufacturing has become a core driving force for the transformation and upgrading of the manufacturing industry. Industrial robots play a key role in this process. Their autonomous operation capabilities significantly improve production efficiency and quality while reducing reliance on human labor. However, traditional grasping robots typically rely on pre-programmed, fixed-path grasping movements, making them difficult to adapt to the chaotic placement of objects. To address this issue, object pose estimation methods have become crucial.

[0003] Traditional object pose estimation methods lack adaptability in complex and dynamic environments, and their versatility and scalability are limited. With the advancement of deep learning and sensor technology, methods based on deep learning and RGB-D data have gradually become mainstream, but they still face challenges such as object occlusion, insufficient accuracy under weak texture conditions, limited real-time performance, and difficulties in multimodal data fusion.

[0004] Furthermore, deep learning models typically rely on large amounts of labeled data for training, and the high cost of manual labeling limits their application in industrial scenarios. While using synthetic data to generate annotations is a viable approach, the discrepancy between synthetic and real data leads to reduced model generalization performance. Therefore, effectively narrowing the gap between synthetic and real data is a key issue in improving the practicality of object pose estimation methods. Summary of the Invention

[0005] In order to at least solve one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide an object pose estimation method based on deep learning and synthetic data and related equipment.

[0006] The first technical solution adopted by the present invention is:

[0007] A method for estimating object pose based on deep learning and synthetic data, comprising the following steps:

[0008] Use simulation modeling software to render the 3D model of the object to obtain synthetic data, and combine it with automated scripts to generate instance segmentation synthetic datasets and pose estimation synthetic datasets;

[0009] The YOLO11-seg model is trained using the synthetic instance segmentation dataset. Real RGB images are collected and the trained YOLO11-seg model is used to extract object masks and bounding boxes from the real RGB images. For the real RGB images and the synthetic RGB images, the target foreground RoI is extracted using their respective masks and bounding boxes, and a dataset is constructed for training the domain adaptation model.

[0010] Build and train a simulation-to-reality domain adaptation model, and use the trained domain adaptation model to enhance the realism of RGB images in a synthetic pose estimation dataset.

[0011] Build a pose estimation model based on multimodal feature fusion and attention mechanism, and use the enhanced pose estimation synthetic dataset to train the model;

[0012] The RGB image of the real scene is input into the trained YOLO11-seg model to output the bounding box and mask of the object in the image; the normal vector image is obtained from the depth image of the real scene, and the target foreground RoI of the RGB image, depth image, normal vector image and mask is extracted based on the bounding box and mask, and the depth image RoI is converted into a local point cloud; the RGB image RoI, normal vector image RoI, mask RoI and local point cloud are input into the trained pose estimation model to output the 6D pose of the object.

[0013] Furthermore, the rendering of the three-dimensional model of the object to obtain synthetic data, and combining with an automated script to generate an instance segmentation synthetic dataset and a pose estimation synthetic dataset, include:

[0014] Use Blender software to build a virtual scene containing a camera and a placement platform, import the PLY model of the object, set the object as an active rigid body, and set the placement platform as a passive rigid body;

[0015] By baking an animation, we simulate the free fall of objects onto a platform to form a stacked scene. At the end frame of the animation, we render the RGB image, depth image, and mask of the scene, and calculate the 6D pose and bounding box of each object. We integrate them to obtain a synthetic dataset of instance segmentation in the COCO format and a synthetic dataset of pose estimation in the LineMOD format.

[0016] Furthermore, the simulation-to-real domain adaptation model is a simulation-to-real domain adaptation model based on an improved CycleGAN, which takes a real RGB image and a target foreground RoI of a synthetic RGB image as model input;

[0017] The model uses the U-Net network as the generator and adds a CBAM module after the first four downsampling of the U-Net network; based on the original loss of CycleGAN, content consistency loss and perceptual loss are introduced.

[0018] Furthermore, the introduction of content consistency loss and perception loss includes:

[0019] The content consistency loss based on the pairwise mean squared error is calculated between the input image and the generated image of the model, and the calculation formula is:

[0020]

[0021] In the formula, x represents the input synthetic image, G(x) represents the fake real image generated by the generator G, y represents the input real image, F(y) represents the fake synthetic image generated by the generator F, k represents the number of pixels in the image, and x i 、y i 、G(x) i and F(y) i Represents the value of the i-th pixel of each image, L C (G,X) represents the content consistency loss between the synthetic image and the fake real image, L C (F,Y) represents the content consistency loss between the real image and the fake synthetic image, L C Indicates the eventual content consistency loss; represents the expectation of the true sample distribution of the X domain, Represents the expectation of the true sample distribution of the Y domain;

[0022] The pre-trained VGG-19 network is used to extract the high-level conv4_4 feature maps of the input image and the reconstructed image, and the L2 loss between the feature maps is calculated as the perceptual loss. The calculation formula is:

[0023]

[0024] Where F(G(x)) represents the reconstructed image corresponding to the synthetic image x, G(F(y)) represents the reconstructed image corresponding to the real image y, φ represents the conv4_4 layer feature extraction function of the VGG-19 network, and ||·||2 represents the L2 norm.

[0025] Furthermore, the pose estimation model based on multimodal feature fusion and attention mechanism takes RGB image RoI, normal vector image RoI, mask RoI and local point cloud as input; the model works as follows:

[0026] The same CNN network without shared weights is used to extract the image features of each image RoI, and the PointNet network is used to extract the point cloud features. Each image feature is input into the intra-modal feature weighting module based on the attention mechanism for enhancement.

[0027] The enhanced RGB image features are input into the 2D inter-modal feature weighting module based on the attention mechanism together with the enhanced mask features and normal vector image features to obtain two image fusion features. The two fusion features are then fused with the point cloud features at the pixel level to obtain the initial dense fusion features.

[0028] The initial dense fusion features are input into the one-dimensional inter-modal feature weighting module based on the attention mechanism to obtain the final dense fusion features.

[0029] Furthermore, the intra-modality feature weighting module based on the attention mechanism works as follows:

[0030] The input image feature F is obtained by 1×1 convolution conv ; F conv It is divided into two parallel branches. The first branch passes through the shared multi-layer perceptron with maximum pooling and a scaling factor of 4. The second branch passes through the shared multi-layer perceptron with average pooling and a scaling factor of 4. The results of the two branches are added together and then passed through the Sigmoid activation function to obtain the channel feature weight W c ; Using W c F conv Weighted to obtain the intermediate feature F c ; F c After 5×5 depth-wise separable convolution, we get F d1 ; F d1 It is divided into three parallel branches. The first branch undergoes 1×7 and 7×1 depthwise separable convolutions to obtain F d2 The second branch is obtained by 1×11 and 11×1 depth-separable convolution to obtain F d3 , the third branch is successively subjected to 1×21 and 21×1 depthwise separable convolution to obtain F d4 , F d1 、F d2 、F d3 and F d4 Merge by element-wise addition to get the spatial attention weight W s ; Using W s F c The final output feature F is obtained by weighting and undergoing 1×1 two-dimensional convolution out .

[0031] Furthermore, the two-dimensional inter-modal feature weighting module based on the attention mechanism works as follows:

[0032] The input image features F1 and F2 are spliced ​​in the channel dimension to obtain the initial fusion feature F concat ; F concat After 1×1 two-dimensional convolution, we get F conv2d ; F conv2d It is divided into two parallel branches. The first branch passes through a two-dimensional maximum pooling and a shared multi-layer perceptron with a scaling factor of 4. The second branch passes through a two-dimensional average pooling and a shared multi-layer perceptron with a scaling factor of 4. The results of the two branches are added together and then passed through a Sigmoid activation function to obtain the channel feature weight W. 2d ; Using W 2d F conv2d Weighted output fusion feature F fusion .

[0033] Furthermore, the one-dimensional inter-modal feature weighting module based on the attention mechanism works as follows:

[0034] The initial dense fusion features are subjected to 1×1 one-dimensional convolution to obtain F conv1d ; F conv1d It is divided into two parallel branches. The first branch passes through a one-dimensional maximum pooling and a shared multi-layer perceptron with a scaling factor of 4. The second branch passes through a one-dimensional average pooling and a shared multi-layer perceptron with a scaling factor of 4. The results of the two branches are added together and then passed through a Sigmoid activation function to obtain the channel feature weight W. 1d ; Using W 1d F conv1d The final dense fusion features are obtained by weighting.

[0035] The second technical solution adopted by the present invention is:

[0036] An electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement an object pose estimation method based on deep learning and synthetic data as described above.

[0037] The third technical solution adopted by the present invention is:

[0038] A computer-readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a method for estimating an object pose based on deep learning and synthetic data as described above.

[0039] The fourth technical solution adopted by the present invention is:

[0040] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above-mentioned method for estimating object pose based on deep learning and synthetic data.

[0041] The beneficial effects of the present invention are: the present invention can accurately estimate the pose of objects in disordered scenes, improve the success rate of robot grasping, and effectively reduce the domain difference between synthetic and real data, reduce the data annotation cost, and make the method applicable to industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1 is a flowchart of a method for estimating object pose based on deep learning and synthetic data in an embodiment of the present invention;

[0044] Figure 2 is an example diagram of a synthetic dataset for pose estimation according to an embodiment of the present invention;

[0045] Figure 3 This is an example diagram of a synthetic data set for instance segmentation according to an embodiment of the present invention;

[0046] Figure 4 2 is a structural diagram of a simulation-to-real-domain adaptation model based on an improved CycleGAN in an embodiment of the present invention;

[0047] Figure 5 : is a structural diagram of a generator based on U-Net and CBAM modules in an embodiment of the present invention;

[0048] Figure 6 is a structural diagram of a pose estimation model based on multimodal feature fusion and attention mechanism in an embodiment of the present invention;

[0049] Figure 7 is a structural diagram of an intra-modal feature weighting module based on an attention mechanism in an embodiment of the present invention;

[0050] Figure 8 is a structural diagram of a two-dimensional inter-modal feature weighting module based on an attention mechanism in an embodiment of the present invention;

[0051] Figure 9 This is a structural diagram of a one-dimensional inter-modal feature weighting module based on the attention mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application. For the step numbers in the following embodiments, they are provided only for the convenience of explanation and are not intended to limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. The singular forms of "a", "said", and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise clearly defined, words such as setting, installing, and connecting should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0054] In the description of this application, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0055] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly specifying the number or order of the technical features indicated.

[0056] In the description of this application, "and / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0057] Explanation of terms:

[0058] YOLO11-seg model: A deep learning model that detects and segments objects in images at the pixel level. It takes a two-dimensional image as input and outputs the object category, the bounding box in the image, and the mask image.

[0059] CycleGAN: A deep learning network for image style transfer that can convert between two image styles X and Y. When an image of style X is used as input, it outputs an image of style Y. When an image of style Y is used as input, it outputs an image of style X.

[0060] Domain adaptation: When the data distributions of the source domain and the target domain are different but the tasks are similar, algorithms are used to reduce the distribution differences so that the model trained in the source domain can better adapt to the target domain.

[0061] PLY model: It is a file format for storing three-dimensional graphic data. The PLY format is mainly used to store mesh models generated by three-dimensional scanners, such as point clouds and polygonal patches.

[0062] COCO format: Common Objects in Context, is a data format widely used in image recognition and segmentation tasks, especially in tasks such as object detection and semantic segmentation.

[0063] LineMOD format: is a dataset format for 6D object pose estimation, mainly used in the fields of machine vision and robotics.

[0064] Example 1

[0065] like Figure 1 As shown, this embodiment provides an object pose estimation method based on deep learning and synthetic data, which specifically includes the following steps:

[0066] S1. Use simulation modeling software to render the 3D model of the object to obtain synthetic data, and combine it with automated scripts to generate instance segmentation synthetic datasets and pose estimation synthetic datasets.

[0067] In this embodiment, step S1 specifically includes the following steps:

[0068] S11. Use Blender software to build a virtual scene containing a camera and a placement platform. Import the PLY model of the object, set the object as an active rigid body, and set the placement platform as a passive rigid body.

[0069] S12. Set the save path of the dataset and initialize the basic information of the dataset.

[0070] S13. Set the number of loop scenes to 30,000.

[0071] S14. Randomly set the initial pose of objects in the scene, the highlight of the object surface, the roughness of the object surface, the number of light sources, the position of the light source spherical coordinate system, the light source intensity, the light source color temperature and the depth image Gaussian noise.

[0072] S15. Perform physical simulation by baking animation to simulate the free fall of objects to form a stacking scene, render the RGB image, depth image and mask of each object at the end frame of the animation, and save them to the corresponding position of the dataset.

[0073] S16, such as Figure 2 As shown, the 6D pose and bounding box of each object are calculated and written into the annotation file of the pose estimation synthetic dataset. Figure 2 (a) is an RGB image. Figure 2 (b) is the depth image, Figure 2 (c) is the mask image.

[0074] S17, such as Figure 3 As shown, COCO format annotations are generated and written to the JSON file of the instance segmentation synthetic dataset. Figure 3 (a) is an RGB image. Figure 3 (b) is the instance segmentation annotation JSON file.

[0075] S18: Execute steps S14 to S17 in a loop until all preset scenes are generated.

[0076] S2. Use the instance segmentation synthetic dataset to train the YOLO11-seg model, and use the trained YOLO11-seg to obtain the object masks and bounding boxes of the real RGB images to assist in building a dataset for training the domain adaptation model.

[0077] In this embodiment, the implementation of constructing a data set for training a domain adaptation model specifically includes the following steps:

[0078] S21. Use the pose estimation to estimate the mask and bounding box of the object in the synthetic dataset to extract the foreground RoI of the synthetic RGB image, forming the synthetic domain part of the dataset.

[0079] As an implementation, the extraction of the foreground RoI of the synthesized RGB image specifically includes the following steps:

[0080] S211 , performing a logical AND operation on the values ​​of the positions corresponding to each pixel domain mask of the RGB image respectively; if the mask value is 1, the pixel value at that position is retained; if the mask value is 0, the pixel value at that position is set to 0.

[0081] S212 , cropping the result obtained in step S211 using the bounding box, and symmetrically filling black pixels to obtain a square image, and finally scaling the image up to a size of 256×256.

[0082] S22. Collect real RGB images and use the mask and bounding box obtained by YOLO11-seg to extract the foreground RoI of the real RGB image to form the real domain part of the dataset.

[0083] In this embodiment, the extraction of the foreground RoI of the real RGB image is implemented in the same manner as steps S211 to S212.

[0084] S3. Build and train a simulation-to-reality domain adaptation model based on the improved CycleGAN, and use the trained domain adaptation model to enhance the realism of RGB images in the pose estimation synthetic dataset.

[0085] In this embodiment, step S3 specifically includes the following steps:

[0086] S31. Use the domain adaptation dataset constructed in step S2 to train a simulation-to-reality domain adaptation model based on the improved CycleGAN.

[0087] The simulation-to-real-world adaptation model structure based on the improved CycleGAN is as follows: Figure 4 As shown, the specific steps include:

[0088] S311, use U-Net to replace ResNet as the generator, and add a CBAM module after the first four downsampling of the U-Net encoder to improve the quality of the generated image. The generator structure is as follows Figure 5 shown.

[0089] S312. Based on the original loss of CycleGAN, a content consistency loss based on pairwise mean square error is introduced between the input image and the generated image to improve the ability to preserve pose information during the conversion process. The calculation formula is:

[0090]

[0091] In the formula, x represents the input synthetic image, G(x) represents the fake real image generated by the generator G, y represents the input real image, F(y) represents the fake synthetic image generated by the generator F, k represents the number of pixels in the image, and x i 、y i 、G(x) i and F(y) i Represents the value of the i-th pixel of each image, L C (G,X) represents the content consistency loss between the synthetic image and the fake real image, LC (F,Y) represents the content consistency loss between the real image and the fake synthetic image, L C Represents the eventual content consistency loss.

[0092] S313. Based on step S312, the pre-trained VGG-19 network is used to extract high-level conv4_4 feature maps of the input image and the reconstructed image, and the L2 loss between the feature maps is calculated as the perceptual loss to improve the ability to retain high-level semantic information. The calculation formula is:

[0093]

[0094] Where F(G(x)) represents the reconstructed image corresponding to the synthetic image x, G(F(y)) represents the reconstructed image corresponding to the real image y, φ represents the conv4_4 layer feature extraction function of the VGG-19 network, and ||·||2 represents the L2 norm.

[0095] S32. Extract the foreground RoI of the synthetic RGB image of each sample in the pose estimation synthetic dataset respectively, input the domain adaptation model, output a more realistic foreground RoI and use the annotation information to restore it to a complete RGB image, replacing the original synthetic RGB image of the sample.

[0096] S4. Build a pose estimation model based on multimodal feature fusion and attention mechanism, use the enhanced pose estimation synthetic dataset to train the pose estimation model, use the trained YOLO11-seg to obtain the bounding box and mask of the object in the real scene image, extract the target foreground RoI data and input it into the trained pose estimation model, and output the 6D pose of the object.

[0097] In this embodiment, step S4 specifically includes the following steps:

[0098] S41. Use the pose estimation synthetic dataset enhanced in step S3 to train a pose estimation model.

[0099] S42. Input the real scene RGB image into YOLO11-seg, obtain the bounding box and mask of the object, use the depth image in the real scene to obtain the normal vector image, extract the target foreground RoI of the RGB image, depth image, normal vector image and mask based on the bounding box and mask, and convert the depth image RoI into a local point cloud.

[0100] The method of obtaining a normal vector image by using a depth image in a real scene may be specifically implemented as follows:

[0101] S421. The depth value d(u,v) of each pixel (u,v) in the depth image can be transformed into a three-dimensional point vector P(u,v)=(X,Y,Z) in the camera coordinate system. The transformation formula is as follows:

[0102]

[0103] Where, f x and f y is the focal length of the depth camera, (c x ,c y ) is the depth camera origin.

[0104] S422. Approximate the local surface at pixel (u, v) as a k×k plane and calculate the local differential vectors in the horizontal and vertical directions of the plane. The calculation formula is as follows:

[0105]

[0106] S423. Perform a cross product calculation on the two local difference vectors to obtain the unit normal vector of the plane. The calculation formula is as follows:

[0107]

[0108] S424 , linearly map each component of the unit normal vector to a range of 0 to 255 to obtain the value of each pixel point of the normal vector image.

[0109] S43, input the RGB image RoI, normal vector image RoI, mask RoI and local point cloud into the trained pose estimation model. The pose estimation model structure is as follows: Figure 6 shown.

[0110] S44. Use the same CNN network without sharing weights to extract image features of each image RoI respectively.

[0111] As an implementation method, the CNN network specifically includes the following steps:

[0112] S441. Use the pre-trained ResNet-50 as the backbone network to encode the input image and extract the underlying and deep features of the image.

[0113] S442. Perform multi-scale processing on the encoded features through the pyramid pooling module to fuse global and local information.

[0114] S443, after four upsampling modules and a 1×1 convolution, obtains high-dimensional features consistent with the input data size.

[0115] S45. Use the PointNet network to extract the features of local point clouds.

[0116] As an embodiment, the PointNet network includes the following steps:

[0117] S451. Use one-dimensional convolution operation to map the three-dimensional coordinates of each point to a high-dimensional feature space and preliminarily extract local geometric information.

[0118] S452, after continuous convolutional layer processing to increase the feature dimension of each point, further enriching the feature expression ability.

[0119] S453. Perform global symmetric aggregation on the features of all points through the average pooling operation to extract global features that reflect the overall geometric structure.

[0120] S46. Input each image feature into the intra-modal feature weighting module based on the attention mechanism for enhancement.

[0121] In some embodiments, the structure of the intra-modality feature weighting module based on the attention mechanism is as follows: Figure 7 As shown, specifically including:

[0122] The input image feature F is obtained by 1×1 convolution conv ; F conv It is divided into two parallel branches. The first branch passes through the shared multi-layer perceptron with maximum pooling and a scaling factor of 4. The second branch passes through the shared multi-layer perceptron with average pooling and a scaling factor of 4. The results of the two branches are added together and then passed through the Sigmoid activation function to obtain the channel feature weight W c ; Using W c F conv Weighted to obtain the intermediate feature F c ; F c After 5×5 depth-wise separable convolution, we get F d1 ; F d1 It is divided into three parallel branches. The first branch undergoes 1×7 and 7×1 depthwise separable convolutions to obtain F d2 The second branch is obtained by 1×11 and 11×1 depth-separable convolution to obtain F d3 , the third branch is successively subjected to 1×21 and 21×1 depthwise separable convolution to obtain F d4 , F d1 、F d2 、F d3 and F d4 Merge by element-wise addition to get the spatial attention weight W s ; Using W s F c The final output feature F is obtained by weighting and undergoing 1×1 two-dimensional convolution out .

[0123] S47. The enhanced RGB image features are respectively input into the two-dimensional inter-modal feature weighting module based on the attention mechanism together with the enhanced mask features and normal vector image features to obtain two image fusion features.

[0124] In some embodiments, the structure of the two-dimensional inter-modal feature weighting module based on the attention mechanism is as follows: Figure 8 As shown, specifically including:

[0125] The input image features F1 and F2 are spliced ​​in the channel dimension to obtain the initial fusion feature F concat ; F concat After 1×1 two-dimensional convolution, we get F conv2d ; F conv2d It is divided into two parallel branches. The first branch passes through a two-dimensional maximum pooling and a shared multi-layer perceptron with a scaling factor of 4. The second branch passes through a two-dimensional average pooling and a shared multi-layer perceptron with a scaling factor of 4. The results of the two branches are added together and then passed through a Sigmoid activation function to obtain the channel feature weight W. 2d ; Using W 2d F conv2d Weighted output fusion feature F fusion .

[0126] S48. Flatten the two image fusion features from two dimensions into one dimension, sample them using the pixel index of the point cloud, obtain two pixel-level features that match the point cloud features, and fuse them with the point cloud features at the pixel level to obtain the initial dense fusion features. Input the initial dense fusion features into the one-dimensional inter-modal feature weighting module based on the attention mechanism to obtain the final dense fusion features.

[0127] The structure of the one-dimensional inter-modal feature weighting module based on the attention mechanism is as follows: Figure 9 As shown, specifically including:

[0128] The initial dense fusion features are subjected to 1×1 one-dimensional convolution to obtain F conv1d ; F conv1d It is divided into two parallel branches. The first branch passes through a one-dimensional maximum pooling and a shared multi-layer perceptron with a scaling factor of 4. The second branch passes through a one-dimensional average pooling and a shared multi-layer perceptron with a scaling factor of 4. The results of the two branches are added together and then passed through a Sigmoid activation function to obtain the channel feature weight W. 1d ; Using W 1d F conv1d The final dense fusion features are obtained by weighting.

[0129] S49. Input the dense fusion features into the pose estimation network and the pose optimization network, and output the 6D pose of the object.

[0130] S491, the pose estimation network consists of three independent networks with four fully connected layers, which predict the translation parameters (x i ,y i ,z i ), rotation quaternion (q wi ,q xi ,q yi ,q zi ) and confidence c i , the pose predicted by the pixel with the highest confidence is selected as the initial predicted pose.

[0131] S492. When estimating an asymmetric object, the 3D model is spatially transformed according to the actual and predicted poses, and the spatial distances between corresponding sampling points are compared. The average of these distances is taken as the loss function. The calculation formula is:

[0132]

[0133] Where x j represents the jth sampling point, [R|t] represents the true posture, represents the predicted pose of the i-th pixel, and n represents the number of sampling points.

[0134] When estimating symmetrical objects, the distance between the sampling point and the corresponding point in the loss function is replaced by the distance between the sampling point and the nearest point. The calculation formula is:

[0135]

[0136] Where x k Indicates the kth point when finding the nearest point.

[0137] The final loss function will lose It is weighted with the confidence and a confidence regularization term is added to enhance the accuracy and robustness of the estimation. The calculation formula is:

[0138]

[0139] Where N represents the number of pixels, c i represents the confidence of the i-th pixel, and w represents the balance hyperparameter.

[0140] S493. Use the initial pose obtained in step S491 to transform the input point cloud as the input of the pose optimization network, iteratively correct the pose by predicting the pose error, and output the final 6D pose of the object.

[0141] Example 2

[0142] An embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the following Figure 1 A method for object pose estimation based on deep learning and synthetic data is shown.

[0143] It is understood that the memory may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created based on the use of the server, etc.

[0144] The processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect various components within the server. It executes various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, as well as accessing data stored in memory. Optionally, the processor may be implemented using at least one of the following hardware forms: digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU) and a modem. The CPU primarily processes the operating system and application programs, while the modem handles wireless communications. It is understood that the modem may not be integrated into the processor and may be implemented separately via a single chip.

[0145] Since the electronic device is an electronic device corresponding to the object pose estimation method based on deep learning and synthetic data in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0146] Example 3

[0147] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 A method for object pose estimation based on deep learning and synthetic data is shown.

[0148] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0149] Since the storage medium is a storage medium corresponding to an object pose estimation method based on deep learning and synthetic data in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0150] Example 4

[0151] In some possible implementations, various aspects of the method of the embodiments of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps of the object pose estimation method based on deep learning and synthetic data according to various exemplary embodiments of the present application described above in this specification. The executable computer program code or "code" for executing each embodiment may be written in a high-level programming language such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0152] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0153] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0154] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made based on the essence of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A method for object pose estimation based on deep learning and synthetic data, characterized in that: The following steps are involved: Render the 3D model of the object to obtain synthetic data, and use automated scripts to generate instance segmentation synthetic datasets and pose estimation synthetic datasets; Using the synthetic instance segmentation dataset to train a YOLO11-seg model, and using the trained YOLO11-seg model to extract object masks and bounding boxes from real RGB images to assist in constructing a dataset for training a domain adaptation model; Build and train a simulation-to-reality domain adaptation model, and use the trained domain adaptation model to enhance the realism of RGB images in a synthetic pose estimation dataset. Build a pose estimation model based on multimodal feature fusion and attention mechanism, and use the enhanced pose estimation synthetic dataset to train the model; The trained YOLO11-seg model is used to extract the target foreground RoI data of the real scene, and input it into the trained pose estimation model to output the 6D pose of the object.

2. The method for object pose estimation based on deep learning and synthetic data according to claim 1, characterized in that: The method of rendering the 3D model of the object to obtain synthetic data and combining it with an automated script to generate a synthetic dataset for instance segmentation and a synthetic dataset for pose estimation includes: Use Blender software to build a virtual scene containing a camera and a placement platform, import the PLY model of the object, set the object as an active rigid body, and set the placement platform as a passive rigid body; By baking an animation, we simulate the free fall of objects onto a platform to form a stacked scene. At the end frame of the animation, we render the RGB image, depth image, and mask of the scene, and calculate the 6D pose and bounding box of each object. We integrate them to obtain a synthetic dataset of instance segmentation in the COCO format and a synthetic dataset of pose estimation in the LineMOD format.

3. The object pose estimation method based on deep learning and synthetic data according to claim 1, characterized in that: The simulation-to-reality domain adaptation model is a simulation-to-reality domain adaptation model based on an improved CycleGAN, which takes the target foreground RoI of the real RGB image and the synthetic RGB image as model input; The model uses the U-Net network as the generator and adds a CBAM module after the first four downsampling of the U-Net network; based on the original loss of CycleGAN, content consistency loss and perceptual loss are introduced.

4. The object pose estimation method based on deep learning and synthetic data according to claim 3, characterized in that: The introduction of content consistency loss and perceptual loss includes: The content consistency loss based on the pairwise mean squared error is calculated between the input image and the generated image of the model, and the calculation formula is: In the formula, x represents the input synthetic image, G(x) represents the fake real image generated by the generator G, y represents the input real image, F(y) represents the fake synthetic image generated by the generator F, k represents the number of pixels in the image, and x i 、y i 、G(x) i and F(y) i Represents the value of the i-th pixel of each image, L C (G,X) represents the content consistency loss between the synthetic image and the fake real image, L C (F,Y) represents the content consistency loss between the real image and the fake synthetic image, L C Indicates the eventual content consistency loss; represents the expectation of the true sample distribution of the X domain, Represents the expectation of the true sample distribution of the Y domain; The pre-trained VGG-19 network is used to extract the high-level conv4_4 feature maps of the input image and the reconstructed image, and the L2 loss between the feature maps is calculated as the perceptual loss. The calculation formula is: Where F(G(x)) represents the reconstructed image corresponding to the synthetic image x, G(F(y)) represents the reconstructed image corresponding to the real image y, φ represents the conv4_4 layer feature extraction function of the VGG-19 network, and ||·||2 represents the L2 norm.

5. The method for object pose estimation based on deep learning and synthetic data according to claim 1, characterized in that: The pose estimation model based on multimodal feature fusion and attention mechanism takes RGB image RoI, normal image RoI, mask RoI and local point cloud as input; the model works as follows: The image features of each image RoI are extracted using a CNN network without shared weights, and the point cloud features are extracted using a PointNet network. Each image feature is then input into an intra-modal feature weighting module based on an attention mechanism for enhancement. The enhanced RGB image features are input into the 2D inter-modal feature weighting module based on the attention mechanism together with the enhanced mask features and normal vector image features to obtain two image fusion features. The two fusion features are then fused with the point cloud features at the pixel level to obtain the initial dense fusion features. The initial dense fusion features are input into the one-dimensional inter-modal feature weighting module based on the attention mechanism to obtain the final dense fusion features.

6. The method for object pose estimation based on deep learning and synthetic data according to claim 5, characterized in that: The attention-based intra-modality feature weighting module works as follows: The input image feature F is obtained by 1×1 convolution conv ; F conv It is divided into two parallel branches. The first branch undergoes maximum pooling and shared multi-layer perceptron, and the second branch undergoes average pooling and shared multi-layer perceptron. The results of the two branches are added together and then passed through the Sigmoid activation function to obtain the channel feature weight W. c ; Using W c F conv Weighted to obtain the intermediate feature F c ; F c After depth-wise separable convolution, we get F d1 ; F d1 It is divided into three parallel branches. The first branch undergoes 1×7 and 7×1 depthwise separable convolutions to obtain F d2 The second branch is obtained by 1×11 and 11×1 depth-separable convolution to obtain F d3 , the third branch is successively subjected to 1×21 and 21×1 depthwise separable convolution to obtain F d4 , F d1 、F d2 、F d3 and F d4 Merge by element-wise addition to get the spatial attention weight W s ; Using W s F c The final output feature F is obtained by weighting and undergoing 1×1 two-dimensional convolution out .

7. The method for object pose estimation based on deep learning and synthetic data according to claim 5, characterized in that: The two-dimensional inter-modal feature weighting module based on the attention mechanism works as follows: The input image features F1 and F2 are spliced ​​in the channel dimension to obtain the initial fusion feature F concat ; F concat After 1×1 two-dimensional convolution, we get F conv2d ; F conv2d It is divided into two parallel branches. The first branch undergoes two-dimensional maximum pooling and a shared multi-layer perceptron. The second branch undergoes two-dimensional average pooling and a shared multi-layer perceptron. The results of the two branches are added together and then passed through the Sigmoid activation function to obtain the channel feature weight W. 2d ; Using W 2d F conv2d Weighted output fusion feature F fusion .

8. The method for object pose estimation based on deep learning and synthetic data according to claim 5, characterized in that: The one-dimensional inter-modal feature weighting module based on the attention mechanism works as follows: The initial dense fusion features are subjected to 1×1 one-dimensional convolution to obtain F conv1d ; F conv1d It is divided into two parallel branches. The first branch passes through one-dimensional maximum pooling and shared multi-layer perceptron, and the second branch passes through one-dimensional average pooling and shared multi-layer perceptron. The results of the two branches are added together and then passed through the Sigmoid activation function to obtain the channel feature weight W 1d ; Using W 1d F conv1d The final dense fusion features are obtained by weighting.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

Images