A 6D pose estimation dataset migration method based on image content and style decoupling

Abstract

The present invention discloses a 6D pose estimation dataset migration method based on image content and style decoupling, comprising: step one: training a pseudo-paired image generation network; step two: training an independently designed image migration network. The present invention uses an encoder to decouple image content and style representation, and then reconstructs the image using a content representation space shared with source domain data and a style representation space shared with target domain data. During reconstruction, strong supervision of the object structure and refinement of the key point structure are achieved through the designed object structure feature extractor and key point attention feature extractor. The present invention can effectively bridge the domain gap between real data and synthetic data, and using a decoupled representation as input can effectively improve mode collapse, and while making full use of unlabeled target domain data, it will not increase the complexity of the 6D pose estimation algorithm.

Description

Technical Field

[0001] The present invention belongs to the field of 6D pose estimation and relates to a 6D pose estimation dataset migration method based on image content and style decoupling, which can effectively fill the domain gap between real data and synthetic data. Background Art

[0002] The 6D pose estimation task aims to estimate the 6 degrees of freedom of a given object relative to the camera, including 3D rotation and 3D translation, which is a fundamental task in computer vision. It is widely applicable to many real-world tasks such as robotic manipulation, augmented reality, and autonomous driving.

[0003] In recent years, with the development of deep neural networks, many 6D pose estimation algorithms based on convolutional neural networks have been proposed and achieved good performance. However, convolutional neural networks are extremely data-driven, so they often require a large amount of real data with 3D pose annotations for training to achieve better results. In fact, 3D pose annotations for real images are extremely difficult to obtain, but 3D poses for synthetic images are easy to obtain. However, due to the domain gap between real data and synthetic data, the performance of the 6D pose estimation network trained on synthetic datasets will be severely degraded when tested on real images. Therefore, how to narrow the domain gap between unlabeled real data and labeled synthetic data has gradually attracted the attention of more and more researchers.

[0004] Image data migration methods can be used to migrate 6D pose estimation datasets, which are divided into paired image domain migration and unpaired image domain migration. Although paired image domain migration has good effects in structure preservation and style migration, the pairing conditions are too harsh, and the real datasets and synthetic datasets of 6D pose estimation cannot meet the pairing requirements. Although unpaired image domain migration has achieved good results in the fields of target detection and classification, it cannot form image pairs and lacks strong supervision of object structure, so it performs poorly in pixel-level tasks such as 6D pose estimation. Summary of the invention

[0005] In view of the shortcomings of the above-mentioned prior arts, the present invention provides a 6D pose estimation dataset migration method based on image content and style decoupling, which can realize inter-domain migration of unpaired images while strongly supervising the object structure, and independently designs a migration network for the 6D pose estimation task, effectively bridging the domain gap between real data and synthetic data in the 6D pose estimation dataset.

[0006] The technical solution of the present invention comprises the following steps:

[0007] A 6D pose estimation dataset migration method based on image content and style decoupling, comprising:

[0008] Step 1: Train the pseudo-paired image generation network: transform the source domain image I s and the target domain image I t Send to the pseudo-paired image generation network for training;

[0009] The encoder is used to obtain the domain-invariant content space and domain-specific style space that share information across domains. Secondly, the domain-invariant content space is exchanged and the domain-specific style space is fixed. The representation space of the domain is fed into the image generator to generate a pseudo-paired image of the domain. Finally, the pseudo-paired image is decoupled and the domain-invariant content space is exchanged to obtain the reconstructed source domain image. and the reconstructed target domain image To enable training on unpaired data;

[0010] Step 2: Train the self-designed image transfer network: The image transfer network is based on the trained pseudo-paired image generation network through the object structure feature extractor H c , style feature extractor H s and keypoint attention feature extractor H k The image generator is further refined to refine the structure near the key points of the object.

[0011] Optionally, the key point attention feature extractor H k Using the source domain image I s Trained, the source domain image I s Feed it into the key point attention feature extractor H k The heat map of the key points is obtained, and the heat map of the key points is used to extract the object structure feature extractor H c The structural loss of the processed source domain image is weighted by attention, and the key point structural loss is defined as:

[0012]

[0013] L represents loss, keypoint represents key point, 2 represents L2 loss; H c represents the structural feature extractor; H k represents the key point feature extractor; I s represents the source domain image; T t represents the migration image of the target domain; the circle in the formula represents the Hadamard product; the subscript 2 under the double vertical line represents the bi-norm.

[0014] Optionally, the total loss function of the image migration network is:

[0015]

[0016] L total : total loss; λ: weight; LKL :KL loss; Domain adversarial loss; Reconstruction loss, the subscript 1 indicates that the loss is L1 loss; Structural loss, subscript 2 indicates that the loss is L2 loss); Style loss; Key point structure loss; Color loss.

[0017] Optionally, the step 2 specifically includes:

[0018] (2.1) The source domain image I s Feeding the source domain content encoder Get the source domain image content encoding The target domain image I t Feed into the target domain style encoder Get the target domain image style encoding

[0019] (2.2) Encoding the target domain image style and source domain image content encoding Feed it into the target domain image generator G t Generate the migration image T of the source domain t ;

[0020] (2.3) Using the structural feature extractor H c Extraction of structural features of the object c : Structural feature extractor H c Using the pre-trained VGG-19 network, first use the mask image M to obtain the object part T of the migration image T object , then T object Send it to the pre-trained VGG-19 network and take out the conv4_2 layer as the object structure feature f c , the structural loss of the object is defined as:

[0021]

[0022] f c It is the structural feature of the object, with the subscript T s Denotes the migration image of the source domain image, with the subscript T t Represents the migration image of the target domain image, with the subscript I t Indicates the target domain image, with the subscript I s represents the source domain image; the double vertical lines represent the norm, and the subscript 2 under the double vertical lines represents the one-norm;

[0023] (2.4) Using style feature extractor H s Extraction of structural features of the objects : Style feature extractor H s Using the same pre-trained VGG-19 network as the structural feature extractor, we first feed the migration image T into the pre-trained VGG-19 network, take out the conv1_1, conv2_1, conv3_1, conv4_1, and conv5_1 layers to calculate the Gram matrix as the style feature f s , the weights of each layer are 1, 0.8, 0.5, 0.3 and 0.1 respectively; the style loss is defined as:

[0024]

[0025] f s Indicates style characteristics, subscript T t Represents the migration image of the target domain image, with the subscript I t represents the target domain image; the double vertical lines represent the norm, and the subscript 2 under the double vertical lines represents the one-norm;

[0026] (2.5) Using the key point attention feature extractor H k Extracting heatmaps of key points of an image: Keypoint Attention Feature Extractor H k Using the source domain image I s Trained, the source domain image I s Feed it into the key point attention feature extractor H k The heat map of the key points is obtained, and the heat map of the key points is used to extract the object structure feature extractor H c The structural loss of the processed source domain image is weighted by attention, and the key point structural loss is defined as:

[0027]

[0028] L represents loss, keypoint represents key point, 2 represents L2 loss; H c represents the structural feature extractor; H k represents the key point feature extractor; I s represents the source domain image; T t represents the migration image of the target domain; the circle in the formula represents the Hadamard product; the subscript 2 under the double vertical line represents the two-norm;

[0029] (2.6) Since the difference between domains is partly caused by light, light is decoupled when defining color loss. ρ is used to represent the image from RGB color model to LAB color model. After removing the light channel, L1 loss is applied to the other two channels:

[0030]

[0031] L represents loss, color represents color loss, 1 represents L1 loss; ρ represents the image from RGB color model to LAB color model; I s represents the source domain image; M s represents the source domain mask image; T t Represents the migration image of the target domain; the double vertical lines represent the norm, and the subscript 1 under the double vertical lines represents the -norm.

[0032] Optionally, the key point attention feature extractor H is used k The heat map of the key points of the extracted image is as follows:

[0033] 2.5.1: Extract feature maps of input images using feature pyramid network and ResNet101;

[0034] 2.5.2: Input the extracted features into the key point extractor H k The network consists of 4 consecutive 3*3 convolutional layers, each followed by a Relu as an activation function. The last layer is upsampled to obtain a feature map consistent with the image size. The extracted features are used to generate a pixel-level probability distribution map h using softmax, indicating the probability that the pixel is a key point.

[0035] Optionally, the step 1 specifically includes:

[0036] (1.1) The source domain image I s Feed into the source domain style encoder and source domain content encoder Get the source domain style encoding and source domain content encoding The target domain image I t Feed into the target domain style encoder and target domain content encoder Get the target domain style encoding and target domain content encoding

[0037] (1.2) Encode the source domain content and target domain content encoding Feed it into the content discriminator D c Differentiate between the two domains’ content encoding;

[0038] (1.3) Encoding the source domain image style and the content encoding of the target domain image Feed it into the source domain image generator G s Generate pseudo paired images F of the target domain s ; Encode the target domain image style and the content encoding of the source domain image Feed it into the target domain image generator G t Generate a pseudo paired image F of the source domain t ;

[0039] (1.4) The pseudo paired image F of the source domain t Send to the target domain discriminator D t , distinguishing between real images and generated images in the target domain;

[0040] (1.5) The pseudo paired image F of the target domain s Send to the target domain discriminator D s , distinguishing between real images from the source domain and generated images;

[0041] (1.6) The pseudo paired image F of the target domain s Style encoding and the pseudo paired image F of the source domain t Content encoding Feed it into the source domain image generator G s Generate the reconstructed source domain image The pseudo paired image F in the source domain t Style encoding and the pseudo paired image F of the target domain s Content encoding Feed it into the target domain image generator G t Generate a reconstructed target domain image

[0042] Optionally, the source domain content encoder and the target domain image generator G t The last layer of shared weights;

[0043] The target domain content encoder and the source domain image generator G s The last layer of shared weights.

[0044] Optionally, step 3: test the network:

[0045] Step 3.1: Transform the source image I s Input source domain content encoder Get the source domain content encoding The target domain image I t Input target domain style encoder Get the target domain style encoding

[0046] Step 3.2: Encode the source domain content and target domain style encoding Feed it into the target domain image generator G t Get the migration image Tt .

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] 1. The present invention proposes a method for achieving inter-domain migration of unpaired images while strongly supervising the structure of objects.

[0049] 2. We independently designed a migration network for the 6D pose estimation task, which effectively bridged the domain gap between real data and synthetic data in the 6D pose estimation dataset.

[0050] 3. The present invention uses decoupled representation as input, which can effectively improve mode collapse and increase the diversity of output.

[0051] 4. The present invention solves the domain gap problem from the data generation level, while making full use of unlabeled target domain data without increasing the complexity of the 6D pose estimation algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0053] Figure 1 The figure shows an overall overview of the 6D pose estimation dataset migration method based on image content and style decoupling of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In the present invention:

[0056] The domain-invariant content space refers to the space composed of object structural features that do not change with the image domain.

[0057] Domain-specific style space means that each image domain has its own style features, and style features of different domains constitute different style spaces.

[0058] The representation space of the domain is divided into the source domain representation space and the target domain representation space. The representation space of the source domain consists of the content space of the source domain and the style space of the source domain. The content space of the source domain consists of the content features of the source domain, and the style space of the source domain consists of the style features of the source domain; the representation space of the target domain consists of the content space of the target domain and the style space of the target domain. The content space of the target domain consists of the content features of the target domain, and the style space of the target domain consists of the style features of the target domain.

[0059] The source domain image refers to the synthetic image, and the target domain image refers to the real image.

[0060] The meanings of all letters in the superscripts in the present invention are uniformly defined as follows:

[0061] Superscript: c stands for structure; s stands for style; k stands for key point;

[0062] Subscript: s represents the source domain; t represents the target domain;

[0063] H generally represents an extractor; G generally represents an image generator; E generally represents an encoder; Z generally represents encoding; I generally represents the original image; T generally represents the transferred image; M represents the mask image; L represents the loss; λ represents the weight; the above letters, together with the meaning of the upper and lower subscripts, represent the corresponding content, such as Denoted as the style encoder of the source domain.

[0064] Combination Figure 1 The 6D posture estimation data set migration method for RGB images of the present invention comprises:

[0065] Step 1: Train the pseudo-paired image generation network: Transform the source domain image I s and the target domain image I t Send to the pseudo-paired image generation network;

[0066] First, the encoder is used to obtain (the source domain image I s and the target domain image I t ) The domain-invariant content space and the domain-specific style space that share information across domains are exchanged; secondly, the domain-invariant content space is exchanged and the domain-specific style space is fixed, and the domain representation space is fed into the respective (source domain and target domain) image generators to generate pseudo-paired images of the domains. Finally, the pseudo-paired images are decoupled and the domain-invariant content space is exchanged to reconstruct the original source domain image. and the target domain image This method uses decoupled representation as input, which can effectively improve mode collapse and increase the diversity of output. This method solves the domain gap problem from the data generation level, while making full use of unlabeled target domain data without increasing the complexity of the 6D pose estimation algorithm.

[0067] Step 2: Train the self-designed image transfer network: The image transfer network is based on the trained pseudo-paired image generation network, and designs the object structure feature extractor H c , style feature extractor H s and keypoint attention feature extractor H k The image generator is further tuned. The tuning content is to refine the structure near the key points of the object through the key point structure loss in step 2. Among them, the object structure feature extractor H c and style feature extractor H s The image after processing the source domain image is the image after the source domain image is migrated.

[0068] In step 1, the pseudo-paired image generation network is trained using the input 6D pose estimation data, which includes the following steps:

[0069] Step 101: Transform the source domain image I s Feed into the source domain style encoder and content encoders Get style code and content encoding The target domain image I t Feed into the target domain style encoder and content encoders Get style code and content encoding

[0070] Step 102: Encode the source domain image style and the content encoding of the target domain image Feed it into the source domain image generator G s Generate pseudo paired images F of the target domain s ; Encode the target domain image style and the content encoding of the source domain image Feed it into the target domain image generator G t Generate a pseudo paired image F of the source domain t .

[0071] Step 103: Pseudo paired image F of the target domain s Feed into the source domain style encoder and content encoders Get style code and content encoding The pseudo paired image F in the source domain t Feed into the target domain style encoder and content encoders Get style code and content encoding

[0072] Step 104: Pseudo paired image F of the target domain s Style encoding and the pseudo paired image F of the source domain t Content encoding Feed it into the source domain image generator G s Generate the reconstructed source domain image The pseudo paired image F in the source domain t Style encoding and the pseudo paired image F of the target domain s Content encoding Feed it into the target domain image generator G t Generate a reconstructed target domain image

[0073] Training the image transfer network consists of the following steps:

[0074] Step 201: Transform the source domain image I s Feeding the source domain content encoder Get content encoding The target domain image I t Feed into the target domain style encoder Get style code

[0075] Step 202: Encode the target domain image style and the content encoding of the source domain image Feed it into the target domain image generator G t Generate the migration image T of the source domain t .

[0076] Step 203: Transform the source domain’s migration image T t , source domain image I s and its mask image M s (Because the migration image T of the source domain t The content and target domain image I s The same, so the migration image T of the source domain t The mask image is the source domain image I s The mask image M s ) is sent to the pre-trained structural feature extractor H c The image T is obtained t The structural characteristics of the object and source domain image I s The structural characteristics of the object

[0077] Step 204: Transform the source domain image T tand the target domain image I t Send it to the pre-trained style feature extractor H s Get image T t Style characteristics and the target domain image I t Style characteristics

[0078] Step 205: Transform the source domain image I s Send it to the pre-trained key point attention feature extractor H k Get the source domain image I s Heatmap

[0079] The present invention will be further described below in conjunction with the accompanying drawings.

[0080] a) is a pseudo-paired image generation network. The network training includes the following steps:

[0081] Step 301: Transform the source domain image I s Feed into the source domain style encoder and content encoders Get style code and content encoding The target domain image I t Feed into the target domain style encoder and content encoders Get style code and content encoding

[0082] Step 302: Apply KL loss to the style encoding to encourage the style representation to be as close as possible to the previous Gaussian distribution:

[0083] L KL =E[D KL ((Z s )||N(0,1))];

[0084] in:

[0085]

[0086] p represents the true sample distribution, q represents the estimated sample distribution, and D KL (p||q) represents the distance between p and q; Z s represents the style code, and Z represents the code.

[0087] Step 303: Encode the source domain content and target domain content encoding Feed it into the content discriminator D c , the content adversarial loss is:

[0088]

[0089] Step 304: Encode the source domain image style and the content encoding of the target domain image Feed it into the source domain image generator G s Generate pseudo paired images F of the target domain s ; Encode the target domain image style and the content encoding of the source domain image Feed it into the target domain image generator G t Generate a pseudo paired image F of the source domain t .

[0090] Step 305: Pseudo paired image F of the source domain t Send to the target domain discriminator D t , the target domain adversarial loss is:

[0091]

[0092] Step 306: Pseudo paired image F of the target domain s Send to the source domain discriminator D s , the source domain adversarial loss is, then the domain adversarial loss for:

[0093]

[0094] Step 307: Encode the style of the pseudo paired image in the target domain Content encoding of pseudo-paired images from the source domain Feed it into the source domain image generator G s Generate the reconstructed source domain image Encoding the style of pseudo-paired images from the source domain Content encoding of pseudo-paired images from the target domain Feed it into the target domain image generator G t Generate a reconstructed target domain image The reconstruction loss is defined as:

[0095]

[0096] Step 308: The total loss function is:

[0097]

[0098] L total : total loss; λ: weight (corresponding to the loss function with the same subscript); L KL :KL loss; Domain adversarial loss; Reconstruction loss, the subscript 1 indicates that the loss is L1 loss; Structural loss, subscript 2 indicates that the loss is L2 loss); Style loss; Key point structure loss; Color loss.

[0099] (b) is a self-designed image transfer network, based on the pseudo-paired image generation network, and the source domain content encoder trained in the pseudo-paired image generation network Style encoder for target domain and the target domain image generator G t The structural feature extractor H is designed. c , style feature extractor H s and keypoint attention feature extractor H k The specific steps are as follows:

[0100] Step 401: Transform the source domain image I s Feeding the source domain content encoder Get content encoding The target domain image I t Feed into the target domain style encoder Get style code

[0101] Step 402: Encode the target domain image style and the content encoding of the source domain image Feed it into the target domain image generator G t Generate the migration image T of the source domain t .

[0102] Step 402: Transform the source domain image T t , source domain image I s and its mask image M s Send it to the pre-trained structural feature extractor H c In. H c Using the pre-trained VGG-19 network, first use the mask image M to obtain the object part T of the migration image T object , then T object Send it to the pre-trained VGG-19 network and take out the conv4_2 layer as the object structure feature f c , the structural loss of the object is defined as:

[0103]

[0104] f is the feature, the superscript c indicates the structure, and the subscript T s Denotes the migration image of the source domain image, with the subscript Tt Represents the migration image of the target domain image, with the subscript I t Indicates the target domain image, with the subscript I s represents the source domain image; the double vertical lines represent the norm, and the subscript 2 under the double vertical lines represents the one-norm;

[0105] Step 403: Transform the source domain's migration image T t and the target domain image I t Send it to the pre-trained style feature extractor H s In. H s Using the same pre-trained VGG-19 network as the structural feature extractor, we first feed the migration image T into the pre-trained VGG-19 network, take out the conv1_1, conv2_1, conv3_1, conv4_1, and conv5_1 layers to calculate the Gram matrix as the style feature f s , the weights of each layer are 1, 0.8, 0.5, 0.3 and 0.1 respectively. The style loss is:

[0106]

[0107] f is the feature, the superscript s indicates the style, and the subscript T indicates the style. t Represents the migration image of the target domain image, with the subscript I t represents the target domain image; the double vertical lines represent the norm, and the subscript 2 under the double vertical lines represents the one-norm;

[0108] Step 404: Transform the source domain image I s Input source domain image I s Pre-trained key point attention feature extractor H k In , we get the heat map of the key points, and use the heat map to weight the structural loss of the generated image. Then the key point structural loss is:

[0109]

[0110] Where: H c represents the structural feature extractor; H k represents the key point feature extractor; I s represents the source domain image; T t represents the migration image of the target domain; the circle in the formula represents the Hadamard product, that is, the product of the corresponding elements of the matrix; the double vertical lines represent the norm, and the subscript 2 under the double vertical lines represents the binary norm.

[0111] Step 405: Since the inter-domain difference is partially caused by light, we decouple light when defining the color loss, using ρ to represent the image from the RGB color model to the LAB color model, and apply L1 loss to the other two channels after removing the light channel:

[0112]

[0113] L represents loss, color represents color loss, 1 represents L1 loss (a common loss); ρ represents the image from RGB color model to LAB color model; I s represents the source domain image; M s represents the source domain mask image; T t Represents the migration image of the target domain; the double vertical lines represent the norm, and the subscript 1 under the double vertical lines represents the -norm.

[0114] Step 406: The total loss function of the image style transfer network is:

[0115]

[0116] Ltotal: total loss; λ: weight (corresponding to the loss function with the same subscript); for example λ weight, the weight of content adversarial loss; L KL :KL loss; Domain adversarial loss; Reconstruction loss (subscript 1 indicates that the loss is L1 loss); Structural loss (subscript 1 indicates the loss is L2 loss); Style loss; Key point structure loss; Color loss.

[0117] The extraction steps in step 404 are specifically as follows:

[0118] Step 501: Use the feature pyramid network and ResNet101 to extract the feature map of the input image.

[0119] Step 502: Input the extracted features into the key point extraction network. The network includes 4 consecutive 3*3 convolutional layers, each layer is followed by a Relu as an activation function, and the last layer is upsampled to obtain a feature map consistent with the image size. The extracted features are used to generate a pixel-level probability distribution map h using softmax, indicating the probability that the pixel is a key point;

[0120] (c) is the test network. The specific steps are as follows:

[0121] Step 601: Transform the source domain image I s Input source domain content encoder Get content encoding The target domain image I t Input target domain style encoder Get style code

[0122] Step 602: Encode the content and style encoding Feed it into the target domain image generator G t Get the migration image T t .

[0123] Experimental part:

[0124] To prove the effectiveness of the method, tests were conducted on the LINEMOD real dataset and the LINEMOD-PBR synthetic dataset. First, the RGB images of the real dataset and the synthetic dataset were input into the network to obtain the RGB images of the synthetic dataset after migration; then the synthetic images, the migrated images, and the labels of the synthetic dataset were respectively input into the 6D pose estimation network to obtain the 6D pose estimation network model of the synthetic image and the 6D pose estimation network model of the migrated image; finally, the performance of the two 6D pose estimation network models was tested on the real dataset.

[0125] Since the amount of data in the LINEMOD real data set is very small, only one thousand images are trained for testing. The 6D pose estimation network uses HRNet proposed by USTC and Microsoft Research Asia in 2019 to estimate key points, followed by a PnP to calculate the object pose. The ADD index was tested on eight objects such as Cat. As shown in Table 1 below, the average ADD value of the present invention is ten percentage points higher than that of the LINEMOD-PBR synthetic data set, which shows that the method of the present invention effectively fills the domain gap between the real data and the synthetic data in the 6D pose estimation data set.

[0126] Table 1

[0127] Model PBR Our Cat 0.455 0.543 Cam 0.187 0.337 Phone 0.253 0.389 Iron 0.268 0.340 Driller 0.617 0.766 Can 0.592 0.727 Glue 0.211 0.255 Duck 0.139 0.164 Mean 0.340 0.440

[0128] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A 6D pose estimation dataset migration method based on image content and style decoupling, characterized in that: include: Step 1: Train the pseudo-paired image generation network: transform the source domain image I s and the target domain image I t Send to the pseudo-paired image generation network for training; The encoder is used to obtain the domain-invariant content space and domain-specific style space that share information across domains. Secondly, the domain-invariant content space is exchanged and the domain-specific style space is fixed. The representation space of the domain is fed into the image generator to generate a pseudo-paired image of the domain. Finally, the pseudo-paired image is decoupled and the domain-invariant content space is exchanged to obtain the reconstructed source domain image. and the reconstructed target domain image , to enable training on unpaired data; Step 2: Train the self-designed image transfer network: The image transfer network is based on the trained pseudo-paired image generation network through the object structure feature extractor H c , style feature extractor H s and keypoint attention feature extractor H k Further refine the structure near the key points of the object for the image generator; The step 1 specifically includes: (1.1) The source domain image I s Feed into the source domain style encoder and source domain content encoder Get the source domain style encoding and source domain content encoding ; The target domain image I t Feed into the target domain style encoder and target domain content encoder Get the target domain style encoding and target domain content encoding ; (1.2) Encode the source domain content and target domain content encoding Feed it into the content discriminator D c Differentiate between the two domains’ content encoding; (1.3) Encoding the source domain image style and the content encoding of the target domain image Feed it into the source domain image generator G s Generate pseudo paired images F of the target domain s ; Encode the target domain image style and the content encoding of the source domain image Feed it into the target domain image generator G t Generate a pseudo paired image F from the source domain t ; (1.4) The pseudo paired image F of the source domain t Send to the target domain discriminator D t , distinguishing between real images and generated images in the target domain; (1.5) The pseudo paired image F of the target domain s Send to the target domain discriminator D s , distinguishing between real images from the source domain and generated images; (1.6) The pseudo paired image F of the target domain s Style encoding and the pseudo paired image F of the source domain t Content encoding Feed it into the source domain image generator G s Generate the reconstructed source domain image ; The pseudo paired image F of the source domain t Style encoding and the pseudo paired image F of the target domain s Content encoding Feed it into the target domain image generator G t Generate a reconstructed target domain image ; The step 2 specifically includes: (2.1) The source domain image I s Feeding the source domain content encoder Get the source domain image content encoding ; The target domain image I t Feed into the target domain style encoder Get the target domain image style encoding ; (2.2) Encoding the target domain image style and source domain image content encoding Feed it into the target domain image generator G t Generate the migration image T of the source domain t ; (2.3) Using the structural feature extractor H c Extraction of structural features : Structural feature extractor H c Using the pre-trained VGG-19 network, first use the mask image M to obtain the object part of the migration image T , then Send it to the pre-trained VGG-19 network and take out the conv4_2 layer as the object structure feature , the structural loss of the object is defined as: ； It is the structural feature of the object, with the subscript T s Denotes the migration image of the source domain image, with the subscript T t Represents the migration image of the target domain image, with the subscript I t Indicates the target domain image, with the subscript I s represents the source domain image; the double vertical lines represent the norm, and the subscript 2 under the double vertical lines represents the one-norm; (2.4) Using style feature extractor H s Extraction of structural features : Style feature extractor H s Using the same pre-trained VGG-19 network as the structural feature extractor, first feed the migration image T into the pre-trained VGG-19 network, take out the conv1_1, conv2_1, conv3_1, conv4_1 and conv5_1 layers to calculate the Gram matrix as the style feature , the weights of each layer are 1, 0.8, 0.5, 0.3 and 0.1 respectively; the style loss is defined as: ； Indicates style characteristics, subscript T t Represents the migration image of the target domain image, with the subscript I t represents the target domain image; the double vertical lines represent the norm, and the subscript 2 under the double vertical lines represents the one-norm; (2.5) Using the key point attention feature extractor H k Extracting heatmaps of key points of an image: Keypoint Attention Feature Extractor H k Using the source domain image I s Trained, the source domain image I s Feed it into the key point attention feature extractor H k The heat map of the key points is obtained, and the heat map of the key points is used to extract the object structure feature extractor H c The structural loss of the processed source domain image is weighted by attention, and the key point structural loss is defined as: ； L represents loss, keypoint represents key point, and 2 represents L2 loss; H c represents the structural feature extractor; H k represents the key point feature extractor; I s represents the source domain image; T t represents the migration image of the target domain; the circle in the formula represents the Hadamard product; the subscript 2 under the double vertical line represents the two-norm; (2.6) Since the inter-domain difference is partly caused by light, light is decoupled when defining the color loss. ρ is used to represent the image from the RGB color model to the LAB color model. After removing the light channel, L1 loss is applied to the other two channels: ； : L represents loss, color represents color loss, 1 represents L1 loss; ρ represents the image from RGB color model to LAB color model; I s represents the source domain image; M s represents the source domain mask image; T t Represents the migration image of the target domain; the double vertical lines represent the norm, and the subscript 1 under the double vertical lines represents the -norm.

2. The 6D pose estimation dataset migration method based on image content and style decoupling according to claim 1, characterized in that: The total loss function of the image migration network is: ； L total : total loss; λ: weight; :KL loss; : Domain adversarial loss; : Reconstruction loss, the subscript 1 indicates that the loss is L1 loss; : structural loss, the subscript 2 indicates that the loss is L2 loss); : style loss; : Key point structure loss; : Color loss.

3. The 6D pose estimation dataset migration method based on image content and style decoupling according to claim 1 or 2, characterized in that: The key point attention feature extractor H k The heat map of the key points of the extracted image is as follows: 2.5.1: Extract feature maps of input images using feature pyramid network and ResNet101; 2.5.2: Input the extracted features into the key point extractor H k The network consists of 4 consecutive 3*3 convolutional layers, each followed by a Relu as an activation function. The last layer is upsampled to obtain a feature map consistent with the image size. The extracted features are used to generate a pixel-level probability distribution map h using softmax, indicating the probability that the pixel is a key point.

4. The 6D pose estimation dataset migration method based on image content and style decoupling as claimed in claim 3, characterized in that: The source domain content encoder and the target domain image generator G t The last layer of shared weights; The target domain content encoder and the source domain image generator G s The last layer of shared weights.

5. The 6D pose estimation dataset migration method based on image content and style decoupling as claimed in claim 1 or 2, characterized in that: Also includes step 3, test network: Step 3.1: Transform the source image I s Input source domain content encoder Get the source domain content encoding , the target domain image Input target domain style encoder Get the target domain style encoding ; Step 3.2: Encode the source domain content and target domain style encoding Feed it into the target domain image generator G t Get the migration image T t .

Images