A self-supervised pose estimation method based on domain adaptability and texture regularization

Abstract

The present invention proposes a self-supervised pose estimation method based on domain adaptability and texture regularization, comprising the following steps: step S1: based on the EfficientPose pose estimator, a channel enhancement module is used in each cascade operation to assign different weights to each channel, and each feature channel is adaptively enhanced; step S2: a multi-level domain adaptation module at the image level and instance level is used to learn domain-invariant features, and the H-divergence is used to measure the differences between samples with different distributions; step S3: an entropy-based alignment method is used to minimize the representation embedding distance between the source domain and the target domain to capture visual representative features; step S4: texture regularization is used to improve the model's sensitivity to pose changes, and finally complete pose estimation and recognition.

Description

Technical Field

[0001] The present invention relates to the technical fields of pattern recognition and computer vision, and in particular to a self-supervised pose estimation method based on domain adaptability and texture regularization. Background Art

[0002] Accurately understanding the target's translation and rotation in three-dimensional space is crucial for interacting with objects in space. Inferring the 6D pose parameters of an object from an image is a core task in 3D computer vision, with widespread applications in fields including augmented reality, virtual reality, autonomous driving, and robotic manipulation. With the recent rise of deep learning, much research is focused on applying this technology to 6D pose estimation.

[0003] However, directly applying deep learning methods to 6D pose estimation faces numerous challenges, one of the most significant being the lack of sufficient labeled data from real-world environments. 6D pose annotation is expensive, requires significant manpower, and requires significant time and effort, making it difficult to train high-performance deep learning models. To overcome this challenge, existing research has begun to employ methods for virtually generating synthetic data. The advantage of synthetic data is that it simplifies the process of accurately generating ground truth and is relatively inexpensive. However, these advantages are countered by the domain differences between real data (target) and synthetic data (source), which pose challenges to the generalization capabilities of the model. Therefore, how to leverage the advantages of synthetic data while reducing or eliminating this domain difference has become a hot topic and a challenge in current research. Summary of the Invention

[0004] In view of this, the object of the present invention is to provide a self-supervised pose estimation method based on domain adaptability and texture regularization.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a self-supervised pose estimation method based on domain adaptivity and texture regularization, comprising the following steps:

[0006] Step S1: Based on the EfficientPose pose estimator, a channel enhancement module is used in each cascade operation to assign different weights to each channel and adaptively enhance each feature channel;

[0007] Step S2: Use image-level and instance-level multi-level domain adaptation modules to learn domain-invariant features, and use H-divergence to measure the differences between samples in different distributions;

[0008] Step S3: An entropy-based alignment method is used to minimize the representation embedding distance between the source and target domains to capture visual representative features.

[0009] Step S4: Texture regularization is used to improve the model's sensitivity to posture changes, and finally complete posture estimation and recognition.

[0010] In a preferred embodiment, step S1 specifically includes the following steps:

[0011] Step S11: Obtain a public pose estimation training set from the Internet, and divide the training data into source training data and target training data. The source training data includes images and their annotations, and the target training data includes unlabeled images. The labeled data of the source domain is represented as S = (x S ,y S ), where x S is the source image, y S is the corresponding 6D pose label, and the unlabeled data in the target domain is represented as T = x T , where x T represents the target image without annotations;

[0012] Step S12: Using EfficientPose as the base pose estimator, a channel enhancement module is used to highlight the importance of each feature channel in the part of the EfficientPose network involving feature cascade operations. The channel enhancement module performs global average pooling on the original feature F0, and then regresses through two fully connected layers and activation functions to obtain channel weights.

[0013] Step S13: Weight the original feature F0 channel by channel to obtain the weighted feature F w , instead of the original feature F0 for network training.

[0014] In a preferred embodiment, step S2 includes the following steps:

[0015] Step S21: Use the domain classifier to perform domain classification. The domain classifier U consists of four convolutional layers and two linear layers with ReLU activation functions.

[0016] Step S22: Use H-divergence to measure the difference between two groups of samples with different distributions, define the distance between the two domains, and minimize the eigenvector of the network domain distance; d U (S, T) represents the distance between two domains under the domain classifier U, and is calculated as follows:

[0017]

[0018] Among them, err S (u(o)), err T (u(o)) are the prediction errors of the classifier on the source domain and target domain samples, respectively. u represents the function under the domain classifier U, o represents the input sample, and u(o) represents the result of the input sample under the domain classifier function.

[0019] Step S23: Use the multi-level domain adaptation domain at the image level and instance level to learn domain-invariant features and train the domain classifier by minimizing the cross entropy loss; the image-level representation is the feature map of the feature extractor, and the loss L of the image-level representation is img The calculation formula is as follows:

[0020] L img =-∑ i [D i log p i +(1-D i )log(1-p i )

[0021] Among them, D i represents the image-level representation of the i-th image, p i represents the domain classifier prediction result of the i-th image;

[0022] The instance-level representation is the feature vector of the region of interest, and the loss L of the instance-level representation is obj The calculation formula is as follows:

[0023] L obj =-∑ i,j [D i log p i,j +(1-D i )log(1-p i,j )]

[0024] Among them, p i,j The domain classifier prediction result for the jth region of interest under the i-th image;

[0025] Step S24: Use the gradient backpropagation layer to minimize the image-level and instance-level loss functions, allowing the domain classifier to distinguish different domain features and achieve multi-level domain adaptation.

[0026] In a preferred embodiment, step S3 specifically includes the following steps:

[0027] Step S31: Perform a projection transformation on the RGB image, create a mask by the dot product transformation of rotation R and translation T to represent the pose, and use a projection head composed of four convolutional layers and two ReLU activation function linear layers and a multi-layer perceptron to generate a pose embedding representation vector;

[0028] Step S32: Adopt entropy-based alignment method to minimize the embedding distance between the source and target domains; use z a represents the representation embedding of the source sample, z b Indicates that target samples from the same category are represented by embeddings;

[0029] Step S33: Create a negative sample pair, add Gaussian noise based on the transformed value; use z' a represents z after adding noise a , z' b represents z after adding noise b ; Setting of transformation values, using true values ​​in the source domain and predicted values ​​in the target domain; Using dot product operation to calculate the representation embedding similarity between each pair of samples; Entropy p of positive sample pairs po The calculation formula is as follows:

[0030]

[0031] Among them, the parameter τ represents the control centrality, k is the index of the negative sample pair, z' a,k Indicates z' a The kth negative sample pair, z' b,k Indicates z' b The kth negative sample pair, N represents the total number of sample pairs; the entropy p of the negative sample pair ne The calculation formula is as follows:

[0032]

[0033] In summary, the entropy-based alignment loss L EA The calculation formula is as follows:

[0034]

[0035] In a preferred embodiment, step S4 specifically includes the following steps:

[0036] Step S41: Establish a regularization method for synthesizing textures, using the current prediction area block Q pre The pose estimation result Y generates a synthetic block Q syn and the composite mask M syn , for the predicted mask M pre Perform an erosion operation using the fill mask M pad Perform boundary filling to fill the foreground boundary of the real image with a synthetic color to compensate for boundary inconsistencies. The calculation formula for the filling mask is as follows:

[0037] M pad =M syn (1-M pre )

[0038] Step S42: Texture regularization E feat The boundary features of the predicted and synthesized patches are aligned by minimizing the MSE error of the deep features extracted by the pre-trained VGG19 network, and the texture regularization loss L reg The calculation formula is as follows:

[0039] L reg =E feat (Q pre ,M pre Q gt +M pad Q syn )+λ e E feat (M pre Q pre +(1-M pre )Q gt ,Q gt )

[0040] Among them, λ e Indicates E feat The weight parameter, Q gt Indicates the true value of the current region block;

[0041] The total loss is calculated as follows:

[0042] Loss = L EA +L pose +λ·L reg

[0043] Among them, L pose is the loss function of the EfficientPose pose estimator, and λ is the regularization weight hyperparameter.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. Through the channel enhancement module, channel weights are learned to effectively extract and utilize the information in each feature channel, improve the representation ability of features, and enable the model to better adapt to different postures and scene changes.

[0046] 2. A multi-level domain adaptation module is introduced, which considers domain adaptation at both the image and instance levels, learns domain-invariant features, reduces the impact of domain shift, uses H-divergence to measure the differences between samples of different distributions, and evaluates the differences between domains, which helps improve the generalization ability and stability of the model.

[0047] 3. The introduction of entropy-based alignment method can better capture the differences between the source domain and the target domain, improve the performance of the model on the target domain, minimize the representation embedding distance, and help the model capture visual representative features.

[0048] 4. Texture regularization is introduced to increase the model's sensitivity to posture changes and improve the model's performance in complex scenes under noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1This is a flowchart of a method implementation according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0051] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0053] like Figure 1 As shown, the present invention provides a self-supervised pose estimation method based on domain adaptability and texture regularization, comprising the following steps:

[0054] Step S1: Based on the EfficientPose pose estimator, a channel enhancement module is used in each cascade operation to assign different weights to each channel and adaptively enhance each feature channel. Specifically, the following steps are included:

[0055] Step S11: Obtain a public pose estimation training set from the Internet and divide the training data into two parts: one is the source training data, including images and their annotations, and the other is the target training data, including unlabeled images. The labeled data of the source domain is represented as S = (x S ,y S ), where x S is the source image, y S is the corresponding 6D pose label, and the unlabeled data in the target domain is represented as T = x T , where x T represents the target image without annotations;

[0056] Step S12: Using EfficientPose as the base pose estimator, a channel enhancement module is used in the EfficientPose network to highlight the importance of each feature channel in the feature concatenation operation. The channel enhancement module performs global average pooling on the original feature F0, and then regresses the channel weights through two fully connected layers and activation functions.

[0057] Step S13: Weight the original feature F0 channel by channel to obtain the weighted feature F w , instead of the original feature F0 for network training.

[0058] Step S2: Use image-level and instance-level multi-level domain adaptation modules to learn domain-invariant features, and use H-divergence to measure the differences between samples in different distributions. Specifically, it includes the following steps:

[0059] Step S21: Use the domain classifier to perform domain classification. The domain classifier U consists of four convolutional layers and two linear layers with ReLU activation functions.

[0060] Step S22: Use H-divergence to measure the difference between two groups of samples with different distributions, define the distance between the two domains, and minimize the feature vector of the network domain distance. U (S, T) represents the distance between two domains under the domain classifier U, and is calculated as follows:

[0061]

[0062] Among them, err S (u(a)), err T (u(a)) are the prediction errors of the classifier on the source domain and target domain samples, respectively, u represents the function under the domain classifier U, o represents the input sample, and u(o) represents the result of the input sample under the domain classifier function;

[0063] Step S23: Use the multi-level domain adaptation domain at the image level and instance level to learn domain-invariant features and train the domain classifier by minimizing the cross entropy loss. The image-level representation is the feature map of the feature extractor, and the loss L of the image-level representation is img The calculation formula is as follows:

[0064] L img =-∑ i [D i log p i +(1-D i )log(1-p i )]

[0065] Among them, D i represents the image-level representation of the i-th image, p i Represents the domain classifier prediction result of the i-th image.

[0066] The instance-level representation is the feature vector of the region of interest, and the loss L of the instance-level representation is obj The calculation formula is as follows:

[0067] L obj =-∑ i,j [D ilog p i,j +(1-D i )log(1-p i,j )]

[0068] Among them, p i,j The domain classifier prediction result for the jth region of interest under the i-th image.

[0069] Step S24: Use the gradient backpropagation layer to minimize the image-level and instance-level loss functions, allowing the domain classifier to distinguish different domain features and achieve multi-level domain adaptation.

[0070] Step S3: Use an entropy-based alignment method to minimize the representation embedding distance between the source domain and the target domain to capture visual representative features. Specifically, it includes the following steps:

[0071] Step S31: Perform a projection transformation on the RGB image, create a mask by the dot product transformation of rotation R and translation T to represent the posture, and use a projection head consisting of four convolutional layers and two ReLU activation function linear layers and a multi-layer perceptron to generate a posture embedding representation vector.

[0072] Step S32: Adopt entropy-based alignment method to minimize the embedding distance between the source and target domains. a represents the representation embedding of the source sample, z b Indicates that target samples from the same category are represented by embeddings.

[0073] Step S33: Create a negative sample pair and add Gaussian noise based on the transformed value. Use z' a represents z after adding noise a , z' b represents z after adding noise b The transformation value is set to use the true value in the source domain and the predicted value in the target domain. The dot product operation is used to calculate the representation embedding similarity between each pair of samples. The entropy p of the positive sample pair po The calculation formula is as follows:

[0074]

[0075] Among them, the parameter τ represents the control centrality, k is the index of the negative sample pair, z' a,k Indicates z' a The kth negative sample pair, z' b,k Indicates z' b The kth negative sample pair, N represents the total number of sample pairs;

[0076] Entropy p of negative sample pairs ne The calculation formula is as follows:

[0077]

[0078] In summary, the entropy-based alignment loss L EA The calculation formula is as follows:

[0079]

[0080] Step S4: Use texture regularization to improve the model's sensitivity to posture changes and ultimately complete posture estimation and recognition. This specifically includes the following steps:

[0081] Step S41: Establish a regularization method for synthesizing textures, using the current prediction area block Q pre The pose estimation result Y generates a synthetic block Q syn and the composite mask M syn , for the predicted mask M pre Perform an erosion operation using the fill mask M pad Perform boundary filling to fill the foreground boundary of the real image with the synthetic color to compensate for the boundary inconsistency. The calculation formula of the filling mask is as follows:

[0082] M pad =M syn (1-M pre )

[0083] Step S42: Texture regularization E feat The boundary features of the predicted and synthesized patches are aligned by minimizing the MSE error of the deep features extracted by the pre-trained VGG19 network, and the texture regularization loss L reg The calculation formula is as follows:

[0084] L reg =E feat (Q pre ,M pre Q gt +M pad Q syn )+λ e E feat (M pre Q pre +(1-M pre )Q gt ,Q gt )

[0085] Among them, λ e Indicates E feat The weight parameter, Q gt Indicates the true value of the current region block.

[0086] The total loss is calculated as follows:

[0087] Loss = L EA +Lpose +λ·L reg

[0088] Among them, L pose is the loss function of the EfficientPose pose estimator, and λ is the regularization weight hyperparameter.

[0089] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions and effects do not exceed the scope of the technical solution of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A self-supervised pose estimation method based on domain adaptability and texture regularization, characterized in that The following steps are involved: Step S1: Based on the EfficientPose pose estimator, a channel enhancement module is used in each cascade operation to assign different weights to each channel and adaptively enhance each feature channel; Step S2: Use image-level and instance-level multi-level domain adaptation modules to learn domain-invariant features, and use H-divergence to measure the differences between samples in different distributions; Step S3: An entropy-based alignment method is used to minimize the representation embedding distance between the source and target domains to capture visual representative features. Step S4: Texture regularization is used to improve the model's sensitivity to posture changes, and finally complete posture estimation and recognition; Step S2 includes the following steps: Step S21: Use the domain classifier to perform domain classification. The domain classifier U consists of four convolutional layers and two linear layers with ReLU activation functions. Step S22: Use H-divergence to measure the difference between two groups of samples with different distributions, define the distance between the two domains, and minimize the eigenvector of the network domain distance; d U (S, T) represents the distance between two domains under the domain classifier U, and is calculated as follows: Among them, err S (u(o)), err T (u(o)) are the prediction errors of the classifier on the source domain and target domain samples, respectively. u represents the function under the domain classifier U, o represents the input sample, and u(o) represents the result of the input sample under the domain classifier function. Step S23: Use the multi-level domain adaptation domain at the image level and instance level to learn domain-invariant features and train the domain classifier by minimizing the cross entropy loss; the image-level representation is the feature map of the feature extractor, and the loss L of the image-level representation is img The calculation formula is as follows: L img =-∑ i [D i logp i +(1-D i )log(1-p i )] Among them, D i represents the image-level representation of the i-th image, p i represents the domain classifier prediction result of the i-th image; The instance-level representation is the feature vector of the region of interest, and the loss L of the instance-level representation is obj The calculation formula is as follows: L obj =-∑ i,j [D i logp i,j +(1-D i )log(1-p i,j )] Among them, p i,j The domain classifier prediction result for the jth region of interest under the i-th image; Step S24: Use the gradient backpropagation layer to minimize the image-level and instance-level loss functions, allowing the domain classifier to distinguish different domain features and achieve multi-level domain adaptation; Step S4 specifically includes the following steps: Step S41: Establish a regularization method for synthesizing textures, using the current prediction area block Q pre The pose estimation result Y generates a synthetic block Q syn and the composite mask M syn , for the predicted mask M pre Perform an erosion operation using the fill mask M pad Perform boundary filling to fill the foreground boundary of the real image with a synthetic color to compensate for boundary inconsistencies. The calculation formula for the filling mask is as follows: M pad =M syn (1-M pre ) Step S42: Texture regularization E feat The boundary features of the predicted and synthesized patches are aligned by minimizing the MSE error of the deep features extracted by the pre-trained VGG19 network, and the texture regularization loss L reg The calculation formula is as follows: 50 reg =E feat (Q pre ,M pre Q gt +M pad Q syn )+λ e From feat (M pre Q pre +(1-M pre )Q gt ,Q gt ) Among them, λ e Indicates E feat The weight parameter, Q gt Indicates the true value of the current region block; The total loss is calculated as follows: Loss=L EA +L pose +λ·L reg Among them, L pose is the loss function of the EfficientPose pose estimator, and λ is the regularization weight hyperparameter.

2. The self-supervised pose estimation method based on domain adaptability and texture regularization according to claim 1, characterized in that Step S1 specifically includes the following steps: Step S11: Obtain a public pose estimation training set from the Internet, and divide the training data into source training data and target training data. The source training data includes images and their annotations, and the target training data includes unlabeled images. The labeled data of the source domain is represented as S = (x S ,y S ), where x S is the source image, y S is the corresponding 6D pose label, and the unlabeled data in the target domain is represented as T = x T , where x T represents the target image without annotations; Step S12: Using EfficientPose as the base pose estimator, a channel enhancement module is used to highlight the importance of each feature channel in the part of the EfficientPose network involving feature cascade operations. The channel enhancement module performs global average pooling on the original feature F0, and then regresses through two fully connected layers and activation functions to obtain channel weights. Step S13: Weight the original feature F0 channel by channel to obtain the weighted feature F w , instead of the original feature F0 for network training.

3. The self-supervised pose estimation method based on domain adaptability and texture regularization according to claim 1, characterized in that: Step S3 specifically includes the following steps: Step S31: Perform a projection transformation on the RGB image, create a mask by the dot product transformation of rotation R and translation T to represent the pose, and use a projection head composed of four convolutional layers and two ReLU activation function linear layers and a multi-layer perceptron to generate a pose embedding representation vector; Step S32: Adopt entropy-based alignment method to minimize the embedding distance between the source and target domains; use z a represents the representation embedding of the source sample, z b Indicates that target samples from the same category are represented by embeddings; Step S33: Create a negative sample pair, add Gaussian noise based on the transformed value; use z' a represents z after adding noise a , z' b represents z after adding noise b ; Setting of transformation values, using true values ​​in the source domain and predicted values ​​in the target domain; Using dot product operation to calculate the representation embedding similarity between each pair of samples; Entropy p of positive sample pairs po The calculation formula is as follows: Among them, the parameter τ represents the control centrality, k is the index of the negative sample pair, z' a,k Indicates z' a The kth negative sample pair, z' b,k Indicates z' b The kth negative sample pair, N represents the total number of sample pairs; the entropy p of the negative sample pair ne The calculation formula is as follows: In summary, the entropy-based alignment loss L EA The calculation formula is as follows:

Images