Self-supervised attitude estimation method based on domain adaptivity and texture regularization
By adopting domain adaptability and texture regularization methods in 6D pose estimation, the problem of domain differences between real and synthetic data of deep learning models is solved, the adaptability and generalization ability of the model is improved, and the matching and generalization ability of the model is enhanced, and the Sensitivity of posture changes.
Patent Information
- Application Number
- CN202410976477.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-20
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-20
AI Technical Summary
The lack of sufficiently labeled data in 6D pose estimation of deep learning methods leads to limited model generalization capabilities, especially in the problem of domain differences between real and synthetic data.
The self-supervised pose estimation method based on domain adaptability and texture regularization is adopted, and the difference between the source domain and the target domain is reduced through channel enhancement module, multi-level domain adaptation module, entropy-based alignment method and texture regularization. Improve the adaptability and generalization ability of the model.
Effectively extract and utilize feature channel information, reduce the impact of domain offset, improve the model's performance in the target domain and sensitivity to pose changes, and enhance the model's stability and generalization ability.
Smart Images

Figure CN118941852A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of pattern recognition and computer vision, and in particular to a self-supervised posture estimation method based on domain adaptability and texture regularization. Background Art
[0002] In spatial interaction with objects, it is crucial to accurately understand the translation and rotation of the target in 3D space. Inferring the 6D pose parameters of the target from an image is a core task in 3D computer vision, which is widely used in fields including augmented reality, virtual reality, autonomous driving, and robotic manipulation. In recent years, with the rise of deep learning, many studies are devoted to applying this technology to the 6D pose estimation task.
[0003] However, directly applying deep learning methods to 6D pose estimation faces many challenges, one of the most important of which is the lack of sufficient labeled data from real environments. Since 6D pose annotation is expensive and requires a lot of manpower and time investment, it is difficult to train high-performance deep learning models. To overcome this problem, existing research works have begun to adopt the method of virtually generating synthetic data. The advantage of synthetic data is that it can simplify the process of accurately generating ground truth and is low-cost. However, corresponding to these advantages are the domain differences between real data (target) and synthetic data (source), which poses challenges to the generalization ability of the model. Therefore, how to reduce or eliminate this domain difference while taking advantage of the advantages of synthetic data has become a hot topic and difficulty in current research. Summary of the invention
[0004] In view of this, the object of the present invention is to provide a self-supervised pose estimation method based on domain adaptability and texture regularization.
[0005] To achieve the above object, the present invention adopts the following technical solution: a self-supervised posture estimation method based on domain adaptivity and texture regularization, comprising the following steps:
[0006] Step S1: Based on the EfficientPose pose estimator, a channel enhancement module is used in each cascade operation to assign different weights to each channel and adaptively enhance each feature channel;
[0007] Step S2: Use image-level and instance-level multi-level domain adaptation modules to learn domain-invariant features, and use H-divergence to measure the differences between samples with different distributions;
[0008] Step S3: An entropy-based alignment method is used to minimize the representation embedding distance between the source domain and the target domain to capture visual representative features;
[0009] Step S4: Use texture regularization to improve the model's sensitivity to posture changes and finally complete posture estimation and recognition.
[0010] In a preferred embodiment, step S1 specifically includes the following steps:
[0011] Step S11: Obtain a public posture estimation training set from the Internet, and divide the training data into source training data and target training data, wherein the source training data includes images and their annotations, and the target training data includes unlabeled images; the labeled data of the source domain is represented as S = (x S ,y S ), where x S is the source image, y S is the corresponding 6D pose label, and the unlabeled data in the target domain is represented by T = x T , where x T represents the target image without annotations;
[0012] Step S12: Using EfficientPose as the basic pose estimator, in the part of the EfficientPose network involving feature cascade operations, a channel enhancement module is used to highlight the importance of each feature channel; the channel enhancement module performs global average pooling on the original feature F0, and then obtains the channel weight through two fully connected layers and activation function regression;
[0013] Step S13: Weight the original feature F0 channel by channel to obtain the weighted feature F w , replacing the original feature F0 for network training.
[0014] In a preferred embodiment, step S2 includes the following steps:
[0015] Step S21: Use a domain classifier to perform domain classification, where the domain classifier U consists of four convolutional layers and two linear layers with ReLU activation functions;
[0016] Step S22: Use H-divergence to measure the difference between two groups of samples with different distributions, define the distance between the two domains, and minimize the feature vector of the network domain distance; d U (S, T) represents the distance between two domains under the domain classifier U, and the calculation formula is as follows:
[0017]
[0018] Among them, err S (u(o)), err T (u(o)) are the prediction errors of the classifier on the source domain and target domain samples, respectively. u represents the function under the domain classifier U, o represents the input sample, and u(o) represents the result of the input sample under the domain classifier function;
[0019] Step S23: Use the multi-level domain adaptation domain at the image level and instance level to learn domain invariant features, and train the domain classifier by minimizing the cross entropy loss; the image level representation is the feature map of the feature extractor, and the loss L of the image level representation is img The calculation formula is as follows:
[0020] L img =-∑ i [D i log p i +(1-D i )log(1-p i )]
[0021] Among them, D i represents the image-level representation of the i-th image, p i represents the domain classifier prediction result of the i-th image;
[0022] The instance-level representation is the feature vector of the region of interest, and the loss L of the instance-level representation is obj The calculation formula is as follows:
[0023] L obj =-∑ i,j [D i log p i,j +(1-D i )log(1-p i,j )]
[0024] Among them, p i,j The domain classifier prediction result for the jth region of interest under the i-th image;
[0025] Step S24: Use the gradient reversal layer to minimize the loss functions at the image level and instance level, so that the domain classifier can distinguish different domain features and achieve multi-level domain adaptation.
[0026] In a preferred embodiment, step S3 specifically includes the following steps:
[0027] Step S31: Project the RGB image, create a mask by rotation R and translation T dot product transformation to represent the posture, and use a projection head composed of four convolutional layers and two Relu activation function linear layers and a multi-layer perceptron to generate a posture embedding representation vector;
[0028] Step S32: Adopt entropy-based alignment method to minimize the embedding distance between the source and target domains; use z a represents the representation embedding of the source sample, z b Indicates that target samples from the same category are represented by embeddings;
[0029] Step S33: Create a negative sample pair, add Gaussian noise based on the transformed value; use z′ a represents z after adding noise a , z′ b represents z after adding noise b ; Setting of transformation values, using true values in the source domain and predicted values in the target domain; Using dot product operations to calculate the representation embedding similarity between each pair of samples; Entropy p of positive sample pairs po The calculation formula is as follows:
[0030]
[0031] Among them, the parameter τ represents the control centrality, k is the index of the negative sample pair, and z′ a,k Represents z′ a The kth negative sample pair, z′ b,k Represents z′ b The kth negative sample pair, N represents the total number of sample pairs;
[0032] Entropy p of negative sample pairs ne The calculation formula is as follows:
[0033]
[0034] In summary, the entropy-based alignment loss L EA The calculation formula is as follows:
[0035]
[0036] In a preferred embodiment, step S4 specifically includes the following steps:
[0037] Step S41: Establish a regularization method for synthesizing textures, using the current prediction area block Q pre The pose estimation result Y generates a synthetic block Q syn and the synthesis mask M syn , for the predicted mask M pre Perform an erosion operation, using the fill mask M pad Fill the boundary and fill the foreground boundary of the real image with the synthetic color to compensate for the inconsistent boundaries. The calculation formula of the filling mask is as follows:
[0038] M pad =M syn (1-M pre )
[0039] Step S42: Texture regularization E feat The boundary features of the predicted patch and the synthesized patch are aligned by minimizing the MSE error of the deep features extracted by the pre-trained VGG19 network, and the texture regularization loss Lreg The calculation formula is as follows:
[0040] L reg =E feat (Q pre , M pre Q gt +M pad Q syn )+λ e E feat (M pre Q pre +(1-M pre )Q gt , Q gt )
[0041] Among them, λ e Indicates E feat The weight parameter, Q gt Indicates the true value of the current region block;
[0042] The total loss is calculated as follows:
[0043] Loss = L EA +L pose +λ·L reg
[0044] Among them, L pose is the loss function of the EfficientPose pose estimator, and λ is the regularization weight hyperparameter.
[0045] Compared with the prior art, the present invention has the following beneficial effects:
[0046] 1. Through the channel enhancement module, channel weights are learned to effectively extract and utilize the information in each feature channel, improve the representation ability of features, and make the model better adapt to different postures and scene changes.
[0047] 2. Introduce a multi-level domain adaptation module, consider domain adaptation at the image level and instance level at the same time, learn domain-invariant features, reduce the impact of domain shift, use H-divergence to measure the differences between samples with different distributions, and evaluate the differences between domains, which helps to improve the generalization ability and stability of the model.
[0048] 3. The introduction of entropy-based alignment method can better capture the differences between the source domain and the target domain, improve the performance of the model on the target domain, minimize the representation embedding distance, and help the model capture visual representative features.
[0049] 4. Texture regularization is introduced to increase the model's sensitivity to posture changes and improve the performance of the model in complex scenes under noise. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 The present invention is a flowchart of a method implementation of a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0052] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.
[0054] like Figure 1 As shown, the present invention provides a self-supervised posture estimation method based on domain adaptability and texture regularization, comprising the following steps:
[0055] Step S1: Based on the EfficientPose pose estimator, a channel enhancement module is used in each cascade operation to assign different weights to each channel and adaptively enhance each feature channel. Specifically, the following steps are included:
[0056] Step S11: Obtain a public pose estimation training set from the Internet and divide the training data into two parts: one is the source training data, including images and their annotations, and the other is the target training data, including unlabeled images. The labeled data of the source domain is represented as S = (x S ,y S ), where x S is the source image, y S is the corresponding 6D pose label, and the unlabeled data in the target domain is represented by T = x T , where x T represents the target image without annotations;
[0057] Step S12: Using EfficientPose as the basic pose estimator, in the part of the EfficientPose network involving feature cascade operations, a channel enhancement module is used to highlight the importance of each feature channel. The channel enhancement module performs global average pooling on the original feature F0, and then obtains the channel weight through two fully connected layers and activation function regression.
[0058] Step S13: Weight the original feature F0 channel by channel to obtain the weighted feature F w , replacing the original feature F0 for network training.
[0059] Step S2: Use image-level and instance-level multi-level domain adaptation modules to learn domain-invariant features, and use H-divergence to measure the differences between samples with different distributions. Specifically, it includes the following steps:
[0060] Step S21: Use the domain classifier to perform domain classification. The domain classifier U consists of four convolutional layers and two linear layers with ReLU activation functions.
[0061] Step S22: Use H-divergence to measure the difference between two groups of samples with different distributions, define the distance between the two domains, and minimize the feature vector of the network domain distance. U (S, T) represents the distance between two domains under the domain classifier U, and the calculation formula is as follows:
[0062]
[0063] Among them, err S (u(a)), err T (u(a)) are the prediction errors of the classifier on the source domain and target domain samples, respectively, u represents the function under the domain classifier U, o represents the input sample, and u(o) represents the result of the input sample under the domain classifier function;
[0064] Step S23: Use the multi-level domain adaptation domain at the image level and instance level to learn domain invariant features and train the domain classifier by minimizing the cross entropy loss. The image-level representation is the feature map of the feature extractor, and the loss L of the image-level representation is img The calculation formula is as follows:
[0065] L img =-∑ i [D i log p i +(1-D i )log(1-p i )]
[0066] Among them, D i represents the image-level representation of the i-th image, p i Represents the domain classifier prediction result of the i-th image.
[0067] The instance-level representation is the feature vector of the region of interest, and the loss L of the instance-level representation is obj The calculation formula is as follows:
[0068] L obj =-∑ i,j [D ilog p i,j +(1-D i )log(1-p i,j )]
[0069] Among them, p i,j The domain classifier prediction result for the j-th region of interest under the i-th image.
[0070] Step S24: Use the gradient reversal layer to minimize the loss functions at the image level and instance level, so that the domain classifier can distinguish different domain features and achieve multi-level domain adaptation.
[0071] Step S3: Use an entropy-based alignment method to minimize the representation embedding distance between the source domain and the target domain to capture visual representative features. Specifically, it includes the following steps:
[0072] Step S31: Perform a projection transformation on the RGB image, create a mask by rotation R and translation T dot product transformation to represent the posture, and use a projection head consisting of four convolutional layers and two Relu activation function linear layers and a multi-layer perceptron to generate a posture embedding representation vector.
[0073] Step S32: Adopt an entropy-based alignment method to minimize the embedding distance between the source and target domains. a represents the representation embedding of the source sample, z b Indicates that target samples from the same category are represented by embeddings.
[0074] Step S33: Create a negative sample pair and add Gaussian noise based on the transformed value. a represents z after adding noise a , z′ b represents z after adding noise b . The transformation value is set to use the true value in the source domain and the predicted value in the target domain. The dot product operation is used to calculate the representation embedding similarity between each pair of samples. The entropy p of the positive sample pair po The calculation formula is as follows:
[0075]
[0076] Among them, the parameter τ represents the control centrality, k is the index of the negative sample pair, and z′ a,k Represents z′ a The kth negative sample pair, z′ b,k Represents z′ b The kth negative sample pair, N represents the total number of sample pairs;
[0077] Entropy p of negative sample pairs ne The calculation formula is as follows:
[0078]
[0079] In summary, the entropy-based alignment loss L EA The calculation formula is as follows:
[0080]
[0081] Step S4: Use texture regularization to improve the model's sensitivity to posture changes, and finally complete posture estimation and recognition. Specifically, it includes the following steps:
[0082] Step S41: Establish a regularization method for synthesizing textures, using the current prediction area block Q pre The pose estimation result Y generates a synthetic block Q syn and the synthesis mask M syn , for the predicted mask M pre Perform an erosion operation, using the fill mask M pad Fill the boundary and fill the foreground boundary of the real image with the synthetic color to compensate for the boundary inconsistency. The calculation formula of the filling mask is as follows:
[0083] M pad =M syn (1-M pre )
[0084] Step S42: Texture regularization E feat The boundary features of the predicted patch and the synthesized patch are aligned by minimizing the MSE error of the deep features extracted by the pre-trained VGG19 network, and the texture regularization loss L reg The calculation formula is as follows:
[0085] L reg =E feat (Q pre , M pre Q gt +M pad Q syn )+λ e E feat (M pre Q pre +(1-M pre )Q gt , Q gt )
[0086] Among them, λ e Indicates E feat The weight parameter, Q gt Indicates the true value of the current region block.
[0087] The total loss is calculated as follows:
[0088] Loss = L EA +Lpose +λ·L reg
[0089] Among them, L pose is the loss function of the EfficientPose pose estimator, and λ is the regularization weight hyperparameter.
[0090] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.
Claims
1. A self-supervised pose estimation method based on domain adaptivity and texture regularization, characterized in that: The following steps are involved: Step S1: Based on the EfficientPose pose estimator, a channel enhancement module is used in each cascade operation to assign different weights to each channel and adaptively enhance each feature channel; Step S2: Use image-level and instance-level multi-level domain adaptation modules to learn domain-invariant features, and use H-divergence to measure the differences between samples with different distributions; Step S3: An entropy-based alignment method is used to minimize the representation embedding distance between the source domain and the target domain to capture visual representative features; Step S4: Use texture regularization to improve the model's sensitivity to posture changes and finally complete posture estimation and recognition.
2. The method for self-supervised pose estimation based on domain adaptability and texture regularization according to claim 1, characterized in that: Step S1 specifically includes the following steps: Step S11: Obtain a public posture estimation training set from the Internet, and divide the training data into source training data and target training data, wherein the source training data includes images and their annotations, and the target training data includes unlabeled images; the labeled data of the source domain is represented as S = (x S ,y S ), where x S is the source image, y S is the corresponding 6D pose label, and the unlabeled data in the target domain is represented by T = x T , where x T represents the target image without annotations; Step S12: Using EfficientPose as the basic pose estimator, in the part of the EfficientPose network involving feature cascade operations, a channel enhancement module is used to highlight the importance of each feature channel; the channel enhancement module performs global average pooling on the original feature F0, and then obtains the channel weight through two fully connected layers and activation function regression; Step S13: Weight the original feature F0 channel by channel to obtain the weighted feature F w , replacing the original feature F0 for network training.
3. The method for self-supervised pose estimation based on domain adaptability and texture regularization according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: Use a domain classifier to perform domain classification, where the domain classifier U consists of four convolutional layers and two linear layers with ReLU activation functions; Step S22: Use H-divergence to measure the difference between two groups of samples with different distributions, define the distance between the two domains, and minimize the feature vector of the network domain distance; d U (S, T) represents the distance between two domains under the domain classifier U, and the calculation formula is as follows: Among them, err S (u(o)), err T (u(o)) are the prediction errors of the classifier on the source domain and target domain samples, respectively. u represents the function under the domain classifier U, o represents the input sample, and u(o) represents the result of the input sample under the domain classifier function; Step S23: Use the multi-level domain adaptation domain at the image level and instance level to learn domain invariant features, and train the domain classifier by minimizing the cross entropy loss; the image level representation is the feature map of the feature extractor, and the loss L of the image level representation is img The calculation formula is as follows: L img =-∑ i [D i log p i +(1-D i )log(1-p i )] Among them, D i represents the image-level representation of the i-th image, p i represents the domain classifier prediction result of the i-th image; The instance-level representation is the feature vector of the region of interest, and the loss L of the instance-level representation is obj The calculation formula is as follows: L obj =-∑ i,j [D i log p i,j +(1-D i )log(1-p i,j )] Among them, p i,j The domain classifier prediction result for the jth region of interest under the i-th image; Step S24: Use the gradient reversal layer to minimize the loss functions at the image level and instance level, so that the domain classifier can distinguish different domain features and achieve multi-level domain adaptation.
4. The method for self-supervised pose estimation based on domain adaptability and texture regularization according to claim 1, characterized in that: Step S3 specifically includes the following steps: Step S31: Project the RGB image, create a mask by rotation R and translation T dot product transformation to represent the posture, and use a projection head composed of four convolutional layers and two Relu activation function linear layers and a multi-layer perceptron to generate a posture embedding representation vector; Step S32: Adopt entropy-based alignment method to minimize the embedding distance between the source and target domains; use z a represents the representation embedding of the source sample, z b Indicates that target samples from the same category are represented by embeddings; Step S33: Create a negative sample pair, add Gaussian noise based on the transformed value; use z′ a represents z after adding noise a , z′ b represents z after adding noise b ; Setting of transformation values, using true values in the source domain and predicted values in the target domain; Using dot product operations to calculate the representation embedding similarity between each pair of samples; Entropy p of positive sample pairs po The calculation formula is as follows: Among them, the parameter τ represents the control centrality, k is the index of the negative sample pair, and z′ a,k Represents z′ a The kth negative sample pair, z′ b,k Represents z′ b The kth negative sample pair, N represents the total number of sample pairs; Entropy p of negative sample pairs ne The calculation formula is as follows: In summary, the entropy-based alignment loss L EA The calculation formula is as follows:
5. The method for self-supervised pose estimation based on domain adaptability and texture regularization according to claim 1, characterized in that: Step S4 specifically includes the following steps: Step S41: Establish a regularization method for synthesizing textures, using the current prediction area block Q pre The pose estimation result Y generates a synthetic block Q syn and the synthesis mask M syn , for the predicted mask M pre Perform an erosion operation, using the fill mask M pad Fill the boundary and fill the foreground boundary of the real image with the synthetic color to compensate for the inconsistent boundaries. The calculation formula of the filling mask is as follows: M pad =M syn (1-M pre ) Step S42: Texture regularization E feat The boundary features of the predicted patch and the synthesized patch are aligned by minimizing the MSE error of the deep features extracted by the pre-trained VGG19 network, and the texture regularization loss L reg The calculation formula is as follows: 50 reg =E feat (Q pre ,M pre Q gt +M pad Q syn )+λ e From feat (M pre Q pre +(1-M pre )Q gt ,Q gt ) Among them, λ e Indicates E feat The weight parameter, Q gt Indicates the true value of the current region block; The total loss is calculated as follows: Loss=L EA +L pose +λ·L reg Among them, L pose is the loss function of the EfficientPose pose estimator, and λ is the regularization weight hyperparameter.
Citation Information
Patent Citations
Facial expression recognition method based on regional self-attention convolutional neural network
CN114842534A
Cross-modal unsupervised domain adaptive medical image segmentation method
CN115731178A
Virtual try-on method adopting arm area retention strategy
CN117689437A
Model and method for multi-source domain adaptation by aligning partial features
US20220138495A1
