A hybrid domain adaptation method for category-level object pose estimation task

Abstract

The application designs a hybrid domain adaptation method for a category-level object pose estimation task, obtains global features of an object to be estimated based on a baseline model DPDN, designs an adversarial domain adaptation module, discriminates whether the global features come from synthetic or real data, prompts a feature extraction module in the DPDN to extract domain-invariant features and the domain adaptation module to be in opposition, so as to narrow the feature distribution of the real data and the synthetic data; based on the baseline model DPDN, a difference-based domain adaptation module is designed, the feature distribution difference between the synthetic data and the real data is directly calculated through a statistical index, the difference is taken as an additional loss, the feature extraction module in the DPDN model is adjusted to the feature space, and the purpose of further narrowing the feature distribution of the real data and the synthetic data is achieved; finally, a hybrid domain adaptation module that fuses the difference domain adaptation and the adversarial domain adaptation is realized; the feature distribution between the synthetic data and the real data is narrowed, and the precision of the DPDN model in the real test scene is improved.

Description

Technical Field

[0001] This invention belongs to the technical field of computer vision, specifically relating to a hybrid domain adaptation method for category-level object pose estimation tasks. Background Technology

[0002] Rigid body pose estimation aims to acquire 3D translation and rotation information of a target rigid body in the camera coordinate system, and is a key technology in autonomous driving, robotics, and augmented reality. In autonomous driving, acquiring the pose information of rigid bodies on the road allows for precise perception of road conditions, helping vehicles effectively avoid obstacles and make appropriate decisions, thus improving the reliability and safety of autonomous driving. In robotics, predicting object pose information enables robotic arms to perform appropriate operations, achieving basic tasks such as accurate grasping and placement. In augmented reality, using rigid body pose information allows for the precise overlay of virtual information onto real-world objects, enhancing the real world and helping people improve work and communication efficiency.

[0003] Based on the size of the estimation range, object pose estimation can be divided into instance-level pose estimation and category-level pose estimation. Early instance-level pose estimation methods could only estimate poses for a small subset of known 3D model instances. In 2019, Wang He et al. introduced the category-level Object Normalized Coordinate Space (NOCS), proposing a technical approach that uniformly maps all instances under a category to the NOCS space, thus extending pose estimation algorithms from the instance level to the category level for the first time. Category-level object pose estimation does not require a known 3D model and can achieve pose estimation for several categories of objects.

[0004] Patent CN116245940A proposes a category-level six-DOF object pose estimation method based on structural difference perception. It enhances geometric features through an information interaction enhancement module and then fuses semantic and geometric information through a semantic dynamic fusion module, thereby improving the accuracy of 6D pose estimation. Patent CN113393503A proposes a category-level object 6D pose estimation method based on segmentation-driven shape prior deformation. It utilizes a multimodal feature instance segmentation network to segment data into image patches and point clouds, and recovers the object's 6D pose after shape prior deformation and point cloud matching.

[0005] While addressing the limitations of 3D models, category-level pose estimation also faces the challenge of excessively high training costs. Firstly, acquiring and collecting real-world data with complete 6D pose annotations is difficult. Compared to tasks like object detection, which only require categories and 2D bounding boxes, images with accurate 6D pose annotations require specialized hardware and significant time and manpower for acquisition, and are more prone to inaccurate annotations that could negatively impact training results. Secondly, training category-level pose estimation models requires far more data than instance-level models, making the collection of sufficient training data even more challenging. Synthetic data created using 3D engines, including RGB images and object point clouds, offers advantages over real-world data, such as accurate annotation, ease of collection, and rich scene representation. Therefore, using synthetic data for pose estimation is a good approach to address the issues of data scarcity and annotation difficulties, making it both reasonable and necessary for practical applications. The high training costs mean that current public benchmarks in the field of category-level pose estimation consist of a large synthetic dataset (CAMERA) and a small real-world dataset (REAL).

[0006] Using synthetic data can address the problem of insufficient training data, but it introduces a domain gap issue between real and synthetic data. Due to differences in object texture, lighting, and environment, there is a data distribution difference between data synthesized using 3D engines and real data, known as the Sim2Real domain gap. Since the theoretical foundation of deep learning assumes that training and test samples are independent and identically distributed to ensure good performance of the trained model during testing, this domain gap will obviously lead to a significant drop in performance when the model trained on synthetic data is tested on real data.

[0007] Domain adaptation is an important task in transfer learning, aiming to transfer a model from the source domain to the target domain even without labeled data in the target domain, ensuring good test performance in the target domain. In deep learning, the source domain refers to the data used during training, and the target domain refers to the data used during testing. Combining domain adaptation techniques can solve many problems faced by deep learning tasks, such as high labeling costs and poor generalization performance, thus improving the performance of deep models in practical applications.

[0008] Currently, most class-level pose estimation methods focus on improving accuracy by modifying the network, using a mixture of large amounts of synthetic data and relatively small amounts of real data for training. This aims to reduce training costs while mitigating the impact of domain gaps on network performance. However, in practical applications, even with only a relatively small amount of real data, labeling the poses of this data still requires significant time and manpower. A few works have explored class and pose estimation methods that do not require real annotations, but their accuracy lags significantly behind methods using real annotations. Therefore, further research is needed to explore class-level pose estimation methods that are unaffected by domain gaps and possess good estimation performance without using real annotations. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention proposes a hybrid domain adaptation method for category-level object pose estimation tasks. This method integrates two domain adaptation strategies: difference-based and adversarial-based. A domain adaptation module is added to the network structure of the baseline model DPDN. This module measures the distribution difference between features extracted from synthetic and real data in the training data, and incorporates this difference as a domain loss into the total training loss. With multiple iterations and backpropagation during training, the domain loss gradually decreases along with the total network loss. The feature extraction module extracts fewer and fewer features from synthetic and real data, making the subsequent pose estimation module unaffected by domain gaps and maintaining high pose estimation accuracy in real-world applications.

[0010] A hybrid domain adaptation method for category-level object pose estimation tasks specifically includes the following steps:

[0011] Step 1: Obtain the global features of the object to be estimated based on the baseline model DPDN, and design an adversarial domain adaptation module. By distinguishing whether the global features come from synthetic or real data, the feature extraction module in DPDN is prompted to extract domain-invariant features to compete with the domain adaptation module, thereby narrowing the feature distribution of real data and synthetic data.

[0012] The baseline model DPDN's network structure includes two modules: feature extraction and pose estimation. For the input sample, the feature extraction module first obtains the features, and then the pose estimation module outputs the predicted pose result.

[0013] The design is based on an adversarial domain adaptation module. Based on the idea of ​​adversarial training, a domain discriminator is added after the feature extraction module of the baseline model DPDN. This forces the feature extraction module to extract domain-invariant features from synthetic and real data to compete with the domain discriminator module, thereby narrowing the feature distribution of synthetic and real data.

[0014] Step 1.1: Extract global features; establish a training set, and then extract a batch of training samples with a total of B from the training set, including B1 synthetic data and B2 real data. Each sample contains an RGB image I and the corresponding object point cloud P; input this batch of training samples into the DPDN model. For each training sample, the feature extraction module outputs the global feature vector FG of the object to be estimated.

[0015] Step 1.1.1: Extract dense local features and geometric features of RGB and point cloud; Input the RGB image I of the object and the corresponding point cloud P of the object into the RGB feature extractor and point cloud feature extractor in the feature extraction module of the baseline model DPDN respectively to obtain dense pixel-level RGB features FIo and geometric features FPo. The feature tensor dimension is (n, d), where n is the number of points in the point cloud and d is the feature dimension of each point.

[0016] Step 1.1.2: Extract RGB global features and point cloud global features; input the local RGB feature FIo and geometric feature FPo into the Global Average Pooling Layer, perform global average pooling on the first dimension of the feature tensor to obtain the RGB global feature vector FI and the point cloud global feature vector FP, and transform the feature dimension to d;

[0017] Step 1.1.3: Obtain the global feature FG of the object to be estimated; use the torch.cat method to concatenate the RGB global feature FI and the point cloud global feature FP to obtain the global feature vector FG of the object, with a feature dimension of 2d;

[0018] Step 1.2: Use the domain discriminator to discriminate the global feature vector FG of the object. The discrimination result is a probability. The higher the probability, the more likely the discriminator thinks that the feature is from real data. Conversely, the lower the probability, the more likely it is from synthetic data.

[0019] Step 1.2.1: Generate domain discrimination results; Input a 2d-dimensional global feature vector FG, which passes through a domain discriminator consisting of three linear layers and two LeakyReLU layers, and finally outputs a one-dimensional real value p. The smaller p is, the more likely the discriminator thinks the feature is from synthetic data, and vice versa.

[0020] Step 1.2.2: Calculate the domain discrimination loss L adv Call torch.nn.BCEWithLogitsLoss() to calculate the cross-entropy loss between p and the true value of the sample class, and obtain the domain discrimination loss value Ladv; the calculation formula is as follows:

[0021] L adv =-(ylog(σ(p))+(1-y)log(1-σ(p)))

[0022] Where y represents the class label of the sample, y is 1 if the sample is a synthetic sample, and y is 0 if it is a real sample, and σ represents the Sigmoid function;

[0023] Step 1.3: Generate a backward gradient through a gradient reversal layer to achieve adversarial training; a gradient reversal layer (GRL) is added between the domain discriminator and the feature extractor. During forward propagation, the GRL layer does not perform calculations but only passes the input value of the i-th layer to the (i+1)-th layer; during backward propagation, the GRL layer multiplies the gradient value output by the (i+1)-th layer by -α, inverts it, and then passes it to the i-th layer; the formula is shown below:

[0024]

[0025] Where FP represents the Forward Propagation stage, BP represents the Backward Propagation stage, and X represents the gradient tensor; the α factor represents the influence of the domain discriminant loss on the feature extraction module, with a value range of (0, 1). α is set to 0 for the first E rounds, meaning the gradient generated by the adversarial training with the discriminant module in the first E rounds is 0. After the Eth round of training, it is considered that the feature extraction module has the ability to extract effective features, and α is gradually increased to allow the domain discriminator module and the feature extractor to begin adversarial training. The formula for calculating α is shown below, where epoch represents the current training round:

[0026]

[0027] After the forward propagation in steps 1.1-1.2 completes and generates the domain discrimination loss, the backward propagation of the loss is initiated, and the parameters of the domain discriminator are updated normally in the direction of stronger discrimination capability;

[0028] Step 2: Based on the baseline model DPDN, design a difference-based domain adaptation module. The difference in feature distribution between synthetic data and real data is directly calculated through statistical indicators. This difference is used as an additional loss, so that the feature extraction module in the DPDN model can adjust the feature space to further narrow the feature distribution between real data and synthetic data.

[0029] The difference-based domain adaptation module directly calculates the difference between the features of synthetic data and the feature distribution of real data through statistical measurement methods, and adds the difference value as the loss to the total loss of the baseline model DPDN. As the training process progresses, the loss is continuously reduced to reduce the difference between the features of synthetic data and real data. MMD (Maximum Mean Discrepancy) is selected as the difference measurement method for feature distribution.

[0030] Step 2.1: Extract global features from synthetic and real data; using the same batch of training samples described in Step 1.1, including B1 synthetic data samples and B2 real data samples, each sample containing an RGB image I and the corresponding object point cloud P; the feature extraction steps are the same as in Step 1.1, and the final output is the global feature tensor FG of this batch of samples, with tensor dimensions of (B, 2d), where the global feature tensor dimension of the synthetic data is (B1, 2d) and the global feature tensor dimension of the real data is (B2, 2d);

[0031] Step 2.2: Calculate the MMD loss L between synthetic data and real data. mmd The MMD distance between two distributions P and Q is calculated using the following formula:

[0032]

[0033] Where p i q represents a sample from distribution P, where n represents the number of samples in distribution P; j Let m represent the sample in distribution Q, m represent the number of samples in distribution Q, and H represent the high-dimensional Hilbert kernel space.

[0034] Step 2.2.1: Define the MMDLoss class, which inherits from the torch.nn.Module parent class in the PyTorch library;

[0035] Step 2.2.2: Define the initialization function __init__; the initialization function completes the assignment of values ​​to the parameters and member variables; the function parameters and member variables include the Gaussian kernel expansion factor kernel_mul, the number of Gaussian kernels used kernel_num, and the Gaussian kernel sigma value fix_sigma;

[0036] Step 2.2.3: Define the Gaussian kernel function `guassian_kernel`. The input parameters are source domain data (`source`) and target domain data (`target`), with tensor shapes of (N, D1) and (M, D2) respectively, and the Gaussian kernel expansion factor `kernel_mul`, the number of Gaussian kernels used (`kernel_num`), and the Gaussian kernel sigma value `fix_sigma`. First, calculate the total number of samples `n_samples` after merging the data from the two domains, merging the data from the two domains into a single dataset. Next, calculate the squared Euclidean distance `L2_distance`, where `L2_distance[i][j]` represents the squared distance between sample `i` and sample `j`. Finally, if `fix_sigma` is None, calculate the normalized sum of the squared Euclidean distances between samples as the adaptive bandwidth; otherwise, use `fix_sigma` to calculate the Gaussian kernel bandwidth. Based on different bandwidth values, calculate the kernel values ​​of multiple Gaussian kernels and return their sum.

[0037] Step 2.2.4: Define the `forward` function, which is the process for calculating the MMD distance between two tensors. The input parameters are the source data (`source`) and the target data (`target`), with tensor shapes of (N, D1) and (M, D2) respectively, and the Gaussian kernel expansion factor `kernel_mul`, the number of Gaussian kernels used (`kernel_num`), and the Gaussian kernel sigma value (`fix_sigma`). First, the `guassian_kernel` function is called to calculate the Gaussian kernel matrix between the source data (`source`) and the target data (`target`). The mean Xx within the source data and the mean Yy within the target data are calculated in the Gaussian kernel matrix. The mean Xy between the source and target data in the Gaussian kernel matrix and the mean Yx of the submatrix at the intersection of the source and target data in the Gaussian kernel matrix are also calculated. Finally, the MMD distance between the two distributions is calculated based on the average of these four means. The MMD distance is then returned.

[0038] Step 2.2.5: Calculate the MMD distance; First, create an object of type MMDLoss, kernel_mul = 2.0, kernel_num = 5, fix_sigma = None; Input the global feature tensor dimension of the synthetic data (B1, 2d) and the global feature tensor dimension of the real data (B2, 2d) into the forward function to calculate the MMD distance value between the two distributions, which is the MMD loss Lmmd;

[0039] Step 3: Based on the baseline model DPDN, and combining Step 1 and Step 2, implement a hybrid domain adaptation module that integrates differential domain adaptation and adversarial domain adaptation; together, bring the feature distribution between synthetic data and real data closer together, and improve the accuracy of the DPDN model in real test scenarios;

[0040] Step 3.1: Model Fusion; Based on the implementation of domain adaptation modules with different strategies in Step 1 and Step 2, the common global feature extraction operation in the two steps is extracted, and the two modules are fused to form a hybrid domain adaptation module; Specifically, global feature extraction is performed only once on the sample, and then the global features are input into the difference-based and adversarial-based domain adaptation modules respectively to calculate MMD distance and domain discrimination loss;

[0041] Step 3.2: Loss Fusion; The two domain losses are weighted and summed using weighting factors to obtain the overall loss L of the hybrid domain adaptation module. domain The calculation is shown in the following formula; and the domain adaptation loss is added to the calculation of the total training loss of the baseline model DPDN through the weighting factor again, together with the pose estimation loss, to constitute the overall training loss of the baseline model DPDN.

[0042] L domain =λ a L adv +λ b L mmd

[0043] L=λ1L pose +λ2L domain

[0044] Where λ a ,λ b λ1 and λ2 are balance factors.

[0045] Beneficial technical effects of the present invention:

[0046] Category-level object pose estimation tasks face the challenge of high training costs due to real pose annotations in practical applications, and no existing methods have yet offered a solution. This invention designs a hybrid domain adaptation method for category-level object pose tasks, which can significantly reduce the training cost of pose estimation networks. After applying the proposed domain adaptation method to the pose estimation network, it only needs to be trained using inexpensive and readily available synthetic data and unlabeled real data. During training, the feature extraction module of the pose estimation network will focus on extracting task-related useful features and ignore the differences between synthetic and real data, thereby significantly improving the accuracy of the network in real data testing compared to training with only synthetic data. Attached Figure Description

[0047] Figure 1Explanation diagram of the domain gap problem in this invention embodiment;

[0048] Figure 2 A schematic diagram comparing the synthetic dataset and the real dataset for the category-level pose estimation task in this invention.

[0049] Figure 3 This invention provides a structural block diagram of a hybrid domain adaptation method for category-level object pose estimation tasks. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments;

[0051] This invention addresses the Sim2Real domain gap problem by combining domain adaptation techniques from the field of transfer learning. It proposes a hybrid domain adaptation method for category-level object pose estimation tasks, achieving high-precision category-level pose estimation without real-label training, further reducing training costs and improving the usability of category-level object pose estimation methods. A diagram illustrating the domain gap problem is attached. Figure 1 As shown;

[0052] The first key point of this invention lies in deconstructing the pose estimation network model, dividing the network into two main modules: feature extraction and pose estimation. Global features are extracted from dense local features, providing a concise input for the subsequent domain adaptation module. The second key point is the implementation of two domain adaptation methods. The key to the adversarial domain adaptation module is the gradient inversion layer, which enables adversarial training between the domain discriminator and the feature extraction module, forcing the feature extraction module to focus more on domain-invariant features. The key to the difference-based domain adaptation module is the implementation of MMD distance calculation based on multi-Gaussian kernels, accurately measuring the distribution difference between synthetic and real data features. The third key point is the fusion of the two domain adaptation methods. After implementing the two methods separately, the common global feature extraction part is extracted to further simplify the network structure. Simultaneously, the losses of the two domain adaptation methods are combined using weight factors as the overall loss of the domain adaptation module, making the overall structure concise and clear.

[0053] A hybrid domain adaptation method for category-level object pose estimation tasks, as shown in the appendix. Figure 3 As shown, the specific steps include:

[0054] Step 1: Obtain the global features of the object to be estimated based on the baseline model DPDN, and design an adversarial domain adaptation module. By distinguishing whether the global features come from synthetic or real data, the feature extraction module in DPDN is prompted to extract domain-invariant features to compete with the domain adaptation module, thereby narrowing the feature distribution of real data and synthetic data.

[0055] The baseline model DPDN's network structure includes two modules: feature extraction and pose estimation. For the input sample, the feature extraction module first obtains the features, and then the pose estimation module outputs the predicted pose result.

[0056] The design is based on an adversarial domain adaptation module. Based on the idea of ​​adversarial training, a domain discriminator is added after the feature extraction module of the baseline model DPDN. This forces the feature extraction module to extract domain-invariant features from synthetic and real data to compete with the domain discriminator module, thereby narrowing the feature distribution of synthetic and real data.

[0057] Step 1.1: Extract global features; Obtain data from the public datasets CAMERA and REAL, establish a training set, and then extract a batch of training samples with a total of B from the training set, including B1 synthetic data from CAMERA and B2 real data from REAL. Each sample contains an RGB image I and the corresponding object point cloud P; Input this batch of training samples into the DPDN model. For each training sample, the feature extraction module outputs the global feature vector FG of the object to be estimated.

[0058] Step 1.1.1: Extract dense local features and geometric features of RGB and point cloud; Input the RGB image I of the object and the corresponding point cloud P of the object into the RGB feature extractor and point cloud feature extractor in the feature extraction module of the baseline model DPDN respectively to obtain dense pixel-level RGB features FIo and geometric features FPo, with feature tensor dimensions of (1024, 128).

[0059] Step 1.1.2: Extract RGB global features and point cloud global features; input the local RGB feature FIo and geometric feature FPo into the Global Average Pooling Layer, perform global average pooling on the first dimension of the feature tensor to obtain the RGB global feature vector FI and the point cloud global feature vector FP, and transform the feature dimension to 128;

[0060] Step 1.1.3: Obtain the global feature FG of the object to be estimated; use the torch.cat method to concatenate the RGB global feature FI and the point cloud global feature FP to obtain the global feature vector FG of the object, with a feature dimension of 256;

[0061] Step 1.2: Use the domain discriminator to discriminate the global feature vector FG of the object. The discrimination result is a probability. The higher the probability, the more likely the discriminator thinks that the feature is from real data. Conversely, the lower the probability, the more likely it is from synthetic data.

[0062] Step 1.2.1: Generate domain discrimination results; Input a 256-dimensional global feature vector FG, which is passed through a domain discriminator consisting of three linear layers and two LeakyReLU layers, and finally outputs a one-dimensional real value p. The smaller p is, the more likely the discriminator thinks the feature is from synthetic data, and vice versa.

[0063] Step 1.2.2: Calculate the domain discrimination loss L adv Call torch.nn.BCEWithLogitsLoss() to calculate the cross-entropy loss between p and the true value of the sample class, and obtain the domain discrimination loss value Ladv; the calculation formula is as follows:

[0064] L adv =-(ylog(σ(p))+(1-y)log(1-σ(p)))

[0065] Where y represents the class label of the sample, y is 1 if the sample is a synthetic sample, and y is 0 if it is a real sample, and σ represents the Sigmoid function;

[0066] Step 1.3: Generate a backward gradient through a gradient reversal layer to achieve adversarial training; a gradient reversal layer (GRL) is added between the domain discriminator and the feature extractor. During forward propagation, the GRL layer does not perform calculations but only passes the input value of the i-th layer to the (i+1)-th layer; during backward propagation, the GRL layer multiplies the gradient value output by the (i+1)-th layer by -α, inverts it, and then passes it to the i-th layer; the formula is shown below:

[0067]

[0068] Where FP represents the Forward Propagation stage, BP represents the Backward Propagation stage, and X represents the gradient tensor; the α factor represents the influence of the domain discriminant loss on the feature extraction module, with a value range of (0, 1). Since the feature extractor's ability is weak in the early stages of network training and cannot extract features well, it is meaningless to discriminate features at this time. Therefore, α is set to 0 for the first 10 rounds, meaning the gradient generated by the adversarial training with the discriminator module is 0 for the first 10 rounds. After the 10th round of training, it is considered that the feature extraction module has the ability to extract effective features, and α is gradually increased to allow the domain discriminator module and the feature extractor to begin adversarial training. The formula for calculating α is shown below, where epoch represents the current training round:

[0069]

[0070] After the forward propagation in steps 1.1-1.2 generates the domain discrimination loss, backpropagation of the loss is initiated. The parameters of the domain discriminator are updated normally in the direction of stronger discrimination ability. Without a gradient backpropagation layer, to ensure that the domain discrimination loss gradually decreases during training, the parameters of the feature extractor will be updated in the direction of extracting increasingly distinguishable features, enabling the domain discriminator to more easily determine whether a feature comes from synthetic or real data, thereby reducing the domain discrimination loss caused by discrimination errors. However, with the addition of a gradient backpropagation layer, as backpropagation continues during training, the parameters of the feature extractor will be updated in the opposite direction, tending to extract increasingly difficult-to-distinguish domain-invariant features. In this case, to avoid discrimination errors, the domain discriminator will further enhance its domain discrimination ability to compete with the feature extractor that generates increasingly difficult-to-distinguish features. Based on this adversarial training, the distribution of features extracted by the feature extraction module from synthetic and real data becomes increasingly similar, thereby eliminating the influence of domain gaps on the subsequent pose estimation module.

[0071] Step 2: Based on the baseline model DPDN, design a difference-based domain adaptation module. The difference in feature distribution between synthetic data and real data is directly calculated through statistical indicators. This difference is used as an additional loss, so that the feature extraction module in the DPDN model can adjust the feature space to further narrow the feature distribution between real data and synthetic data.

[0072] The difference-based domain adaptation module directly calculates the difference between the features of synthetic data and the feature distribution of real data through statistical measurement methods, and adds the difference value as the loss to the total loss of the baseline model DPDN. As the training process progresses, the loss is continuously reduced to reduce the difference between the features of synthetic data and real data. MMD (Maximum Mean Discrepancy) is selected as the difference measurement method for feature distribution.

[0073] Step 2.1: Extract global features from synthetic and real data; using the same batch of training samples described in Step 1.1, including B1 synthetic data samples and B2 real data samples, each sample containing an RGB image I and the corresponding object point cloud P; the feature extraction steps are the same as in Step 1.1, and finally output the global feature tensor FG of this batch of samples, with tensor dimensions of (B, 256), where the global feature tensor dimensions of the synthetic data are (B1, 256) and the global feature tensor dimensions of the real data are (B2, 256);

[0074] Step 2.2: Calculate the MMD loss L between synthetic data and real data. mmd The difference between two distributions is measured by mapping random variables of two distributions to a high-dimensional space and calculating the supremum of the difference between their expected values. The MMD distance between two distributions P and Q is calculated using the following formula:

[0075]

[0076] Where p i q represents a sample from distribution P, where n represents the number of samples in distribution P; j Let m represent the sample in distribution Q, m represent the number of samples in distribution Q, and H represent the high-dimensional Hilbert kernel space.

[0077] Step 2.2.1: Define the MMDLoss class, which inherits from the torch.nn.Module parent class in the PyTorch library;

[0078] Step 2.2.2: Define the initialization function __init__; the initialization function completes the assignment of values ​​to the parameters and member variables; the function parameters and member variables include the Gaussian kernel expansion factor kernel_mul, the number of Gaussian kernels used kernel_num, and the Gaussian kernel sigma value fix_sigma;

[0079] Step 2.2.3: Define the Gaussian kernel function `guassian_kernel`. The input parameters are source domain data (`source`) and target domain data (`target`), with tensor shapes of (N, D1) and (M, D2) respectively, and the Gaussian kernel expansion factor `kernel_mul`, the number of Gaussian kernels used (`kernel_num`), and the Gaussian kernel sigma value `fix_sigma`. First, calculate the total number of samples `n_samples` after merging the data from the two domains, merging the data from the two domains into a single dataset. Next, calculate the squared Euclidean distance `L2_distance`, where `L2_distance[i][j]` represents the squared distance between sample `i` and sample `j`. Finally, if `fix_sigma` is None, calculate the normalized sum of the squared Euclidean distances between samples as the adaptive bandwidth; otherwise, use `fix_sigma` to calculate the Gaussian kernel bandwidth. Based on different bandwidth values, calculate the kernel values ​​of multiple Gaussian kernels and return their sum.

[0080] Step 2.2.4: Define the `forward` function, which is the process for calculating the MMD distance between two tensors. The input parameters are the source data (`source`) and the target data (`target`), with tensor shapes of (N, D1) and (M, D2) respectively, and the Gaussian kernel expansion factor `kernel_mul`, the number of Gaussian kernels used (`kernel_num`), and the Gaussian kernel sigma value (`fix_sigma`). First, the `guassian_kernel` function is called to calculate the Gaussian kernel matrix between the source data (`source`) and the target data (`target`). The mean Xx within the source data and the mean Yy within the target data are calculated in the Gaussian kernel matrix. The mean Xy between the source and target data in the Gaussian kernel matrix and the mean Yx of the submatrix at the intersection of the source and target data in the Gaussian kernel matrix are also calculated. Finally, the MMD distance between the two distributions is calculated based on the average of these four means. The MMD distance is then returned.

[0081] Step 2.2.5: Calculate the MMD distance; First, create an object of type MMDLoss, kernel_mul = 2.0, kernel_num = 5, fix_sigma = None; Input the global feature tensor dimension of the synthetic data (B1, 256) and the global feature tensor dimension of the real data (B2, 256) into the forward function to calculate the MMD distance value between the two distributions, that is, to obtain the MMD loss Lmmd;

[0082] Step 3: Based on the baseline model DPDN, and combining Step 1 and Step 2, implement a hybrid domain adaptation module that integrates differential domain adaptation and adversarial domain adaptation; together, bring the feature distribution between synthetic data and real data closer together, and improve the accuracy of the DPDN model in real test scenarios;

[0083] Step 3.1: Model Fusion; Based on the implementation of domain adaptation modules with different strategies in Step 1 and Step 2, the common global feature extraction operation in the two steps is extracted, and the two modules are fused to form a hybrid domain adaptation module; Specifically, global feature extraction is performed only once on the sample, and then the global features are input into the difference-based and adversarial-based domain adaptation modules respectively to calculate MMD distance and domain discrimination loss;

[0084] Step 3.2: Loss Fusion; The two domain losses are weighted and summed using weighting factors to obtain the overall loss L of the hybrid domain adaptation module. domain The calculation is shown in the following formula; and the domain adaptation loss is added to the calculation of the total training loss of the baseline model DPDN through the weighting factor again, together with the pose estimation loss, to constitute the overall training loss of the baseline model DPDN.

[0085] Ldomain =λ a L adv +λ b L mmd

[0086] L=λ1L pose +λ2L domain

[0087] Where λ a ,λ b λ1 and λ2 are balance factors.

[0088] Step 4: Training DPDN after adding the hybrid domain adaptation module;

[0089] In terms of data, this invention uses synthetic data from the publicly available dataset CAMER and unlabeled real data from REAL for training in the field of category-level pose estimation. For training, the open-source framework PyTorch is used for neural network training; the input RGB image size is 192×192, and the number of point clouds is 1024; a ResNet-18-based PSP network is used as the RGB feature extractor, and PointNet++ is used as the point cloud feature extractor, with each pixel-wise local feature having a dimension of 128; ADAM is used as the optimizer, with 30 training epochs and a total of 120,000 iterations; the total data size of each batch of samples is B=48, and the ratio of synthetic data B1 to real data B2 is 1:1; the hyperparameters λ1, λ2, λa, and λb are set to 1.0, 5.0, 0.05, and 0.01, respectively. A comparison diagram of the synthetic and real datasets for the category-level pose estimation task of this invention is attached. Figure 2 As shown;

[0090] This invention uses the publicly available dataset REAL in the field of category-level pose estimation for evaluation and comparison. In implementation, the parameter settings involved in the invention include: the number of points N in the point cloud is set to 1024, the feature dimension d is set to 128, E in the adversarial domain adaptation module is set to 10, and the balance factors λ1, λ2, λa, and λb are set to 1.0, 5.0, 0.05, and 0.01, respectively. The proposed domain adaptation method is implemented based on the baseline model DPDN. As shown in Table 1, after adding the domain adaptation strategy, the accuracy of this invention on the real dataset test set is significantly improved, with improvements exceeding 10% on multiple metrics compared to the baseline model. Furthermore, as shown in Table 2, this invention achieves higher accuracy than similar unsupervised methods, reaching accuracy comparable to methods using real annotations without requiring real annotations.

[0091] Table 1. Performance comparison of the present invention and the baseline model on the REAL dataset;

[0092]

[0093] Table 2 shows the performance comparison of the present invention and other methods on the REAL dataset;

[0094]

[0095] In addition, the domain adaptation module added by this invention only needs to be added during training. After training, the model feature extractor can extract good domain-invariant features without being affected by the domain gaps. Therefore, no additional inference of the domain adaptation module is required during the inference stage. Only the conventional calculation process of feature extraction and pose estimation is required, which will not generate additional memory and time overhead.

[0096] In summary, this invention proposes a hybrid domain adaptation method for category-level object pose estimation tasks. It only requires synthetic data and unlabeled real data for network training, which greatly reduces the network training cost. Moreover, the model can still maintain high accuracy in real-world application scenarios, which is conducive to the practical application of category-level pose estimation technology.

Claims

1. A hybrid domain adaptation method for category-level object pose estimation tasks, characterized in that, Specifically, the following steps are included: Step 1: Obtain the global features of the object to be estimated based on the baseline model DPDN, and design an adversarial domain adaptation module. By distinguishing whether the global features come from synthetic or real data, the feature extraction module in DPDN is prompted to extract domain-invariant features to compete with the domain adaptation module, thereby narrowing the feature distribution of real data and synthetic data. Step 1 is as follows: Step 1.1: Extract global features; Establish a training set, and then extract a batch of training samples with a total number of B from the training set, including B1 synthetic data and B2 real data. Each sample contains an RGB image I and the corresponding object point cloud P. Input this batch of training samples into the DPDN model. For each training sample, the feature extraction module outputs the global feature vector FG of the object to be estimated. Step 1.2: Use the domain discriminator to discriminate the global feature vector FG of the object. The discrimination result is a probability. The higher the probability, the more likely the discriminator thinks that the feature is from real data. Conversely, the lower the probability, the more likely it is from synthetic data. Step 1.3: Generate a backward gradient through a gradient reversal layer to achieve adversarial training; a gradient reversal layer (GRL) is added between the domain discriminator and the feature extractor. The GRL layer does not perform calculations during forward propagation; it only passes the input value of the i-th layer to the (i+1)-th layer. During backward propagation, the GRL layer multiplies the gradient value output by the (i+1)-th layer. Invert the value and then pass it to the i-th layer; the formula is as follows: Where FP represents the Forward Propagation phase and BP represents the Backward Propagation phase. Represents the gradient tensor; The factor represents the degree of influence of the domain discriminant loss on the feature extraction module, with a value range of (0, 1). Wheel Set to 0, i.e., the previous The gradient generated by the adversarial and discriminative modules in the first round is 0; waiting for the second... E After one round of training, it is considered that the feature extraction module has the ability to extract effective features, and then... The size gradually increases, causing the domain discriminator module and the feature extractor to begin adversarial training. The calculation formula is shown below, where This represents the current training round: After the forward propagation in steps 1.1-1.2 completes and generates the domain discrimination loss, the backward propagation of the loss is initiated, and the parameters of the domain discriminator are updated normally in the direction of stronger discrimination capability; Step 2: Based on the baseline model DPDN, design a difference-based domain adaptation module. The difference in feature distribution between synthetic data and real data is directly calculated through statistical indicators. This difference is used as an additional loss, so that the feature extraction module in the DPDN model can adjust the feature space to further narrow the feature distribution between real data and synthetic data. In step 2, the difference-based domain adaptation module directly calculates the difference between the features of synthetic data and the feature distribution of real data using statistical measurement methods. The difference value is added as a loss to the total loss of the baseline model DPDN. As the training process progresses, the loss is continuously reduced to decrease the difference between the features of synthetic data and real data. MMD (Maximum Mean Discrepancy) is selected as the difference measurement method for feature distribution. Step 3: Based on the baseline model DPDN, and combining Step 1 and Step 2, implement a hybrid domain adaptation module that integrates differential domain adaptation and adversarial domain adaptation; together, bring the feature distribution between synthetic data and real data closer together, and improve the accuracy of the DPDN model in real test scenarios.

2. The hybrid domain adaptation method for category-level object pose estimation tasks according to claim 1, characterized in that, The baseline model DPDN network structure described in step 1 includes two modules: feature extraction and pose estimation. For the input sample, the feature extraction module first obtains the features, and then the pose estimation module outputs the predicted pose result.

3. The hybrid domain adaptation method for category-level object pose estimation tasks according to claim 1, characterized in that, Step 1 describes the design of an adversarial domain adaptation module. Based on the idea of ​​adversarial training, a domain discriminator is added after the feature extraction module of the baseline model DPDN. This forces the feature extraction module to extract domain-invariant features from synthetic and real data to compete with the domain discriminator module, thereby narrowing the feature distribution of synthetic and real data.

4. The hybrid domain adaptation method for category-level object pose estimation tasks according to claim 1, characterized in that, Step 1.1 specifically involves: Step 1.1.1: Extract dense local features and geometric features from RGB and point clouds; input the RGB image I of the object and the corresponding point cloud P into the RGB feature extractor and point cloud feature extractor in the feature extraction module of the baseline model DPDN, respectively, to obtain dense pixel-level RGB features FIo and geometric features FPo, with the feature tensor dimension being ( , ),in The number of points in the point cloud. The feature dimension for each point; Step 1.1.2: Extract RGB global features and point cloud global features; input the local RGB feature FIo and geometric feature FPo into the Global Average Pooling Layer, perform global average pooling on the first dimension of the feature tensor to obtain the RGB global feature vector FI and the point cloud global feature vector FP, and transform the feature dimension to... ; Step 1.1.3: Obtain the global feature FG of the object to be estimated; use the torch.cat method to concatenate the RGB global feature FI and the point cloud global feature FP to obtain the global feature vector FG of the object, with a feature dimension of [missing value]. .

5. The hybrid domain adaptation method for category-level object pose estimation tasks according to claim 1, characterized in that, Step 1.2 specifically involves: Step 1.2.1: Generate domain discrimination results; Input A global feature vector FG is passed through a domain discriminator consisting of three linear layers and two LeakyReLU layers, and finally outputs a one-dimensional real value p. The smaller the p, the more likely the discriminator thinks the feature is from synthetic data, and vice versa. Step 1.2.2: Compute Domain Discriminant Loss Call torch.nn.BCEWithLogitsLoss() to calculate the cross-entropy loss between p and the true value of the sample class, and obtain the domain discrimination loss value Ladv; The calculation formula is as follows: Where y represents the class label of the sample; if the sample is a synthetic sample, y is 1, and if it is a real sample, y is 0. This represents the Sigmoid function.

6. The hybrid domain adaptation method for category-level object pose estimation tasks according to claim 1, characterized in that, Step 2 is as follows: Step 2.1: Extract global features from synthetic and real data; use the same batch of training samples described in Step 1.1, including B1 synthetic data samples and B2 real data samples, each sample containing an RGB image I and the corresponding object point cloud P; The feature extraction steps are the same as in step 1.

1. The final output is the global feature tensor FG of this batch of samples, with a tensor dimension of (B, 2d). The global feature tensor dimension of the synthetic data is (B1, 2d), and the global feature tensor dimension of the real data is (B2, 2d). Step 2.2: Calculate the MMD loss between synthetic data and real data. ; The formula for calculating the MMD distance between two distributions P and Q is as follows: in This represents a sample from distribution P. This represents the number of samples in distribution P; Represents a sample in distribution Q. The number of samples in distribution Q is represented by H, and H represents the high-dimensional Hilbert kernel space. Step 2.2.1: Define the MMDLoss class, which inherits from the torch.nn.Module parent class in the PyTorch library; Step 2.2.2: Define the initialization function __init__; the initialization function completes the assignment of values ​​to the parameters and member variables; the function parameters and member variables include the Gaussian kernel expansion factor kernel_mul, the number of Gaussian kernels used kernel_num, and the Gaussian kernel sigma value fix_sigma; Step 2.2.3: Define the Gaussian kernel function `guassian_kernel`. The input parameters are source domain data (`source`) and target domain data (`target`), with tensor shapes of (N, D1) and (M, D2) respectively, and the Gaussian kernel expansion factor `kernel_mul`, the number of Gaussian kernels used (`kernel_num`), and the Gaussian kernel sigma value `fix_sigma`. First, calculate the total number of samples `n_samples` after merging the data from the two domains, merging the data from the two domains into a single dataset. Next, calculate the squared Euclidean distance `L2_distance`, where `L2_distance[i][j]` represents the squared distance between sample `i` and sample `j`. Finally, if `fix_sigma` is None, calculate the normalized sum of the squared Euclidean distances between samples as the adaptive bandwidth; otherwise, use `fix_sigma` to calculate the Gaussian kernel bandwidth. Based on different bandwidth values, calculate the kernel values ​​of multiple Gaussian kernels and return their sum. Step 2.2.4: Define the `forward` function, which is the process for calculating the MMD distance between two tensors. The input parameters are the source data (`source`) and the target data (`target`), with tensor shapes of (N, D1) and (M, D2) respectively, and the Gaussian kernel expansion factor `kernel_mul`, the number of Gaussian kernels used (`kernel_num`), and the Gaussian kernel sigma value (`fix_sigma`). First, the `guassian_kernel` function is called to calculate the Gaussian kernel matrix between the source data (`source`) and the target data (`target`). The mean Xx within the source data and the mean Yy within the target data are calculated in the Gaussian kernel matrix. The mean Xy between the source and target data in the Gaussian kernel matrix and the mean Yx of the submatrix at the intersection of the source and target data in the Gaussian kernel matrix are also calculated. Finally, the MMD distance between the two distributions is calculated based on the average of these four means. The MMD distance is then returned. Step 2.2.5: Calculate the MMD distance; First, create an object of type MMDLoss with kernel_mul=2.0, kernel_num=5, and fix_sigma=None; Input the global feature tensor dimension of the synthetic data (B1, 2d) and the global feature tensor dimension of the real data (B2, 2d) into the forward function to calculate the MMD distance value between the two distributions, which is the MMD loss Lmmd.

7. The hybrid domain adaptation method for category-level object pose estimation tasks according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Model Fusion; Based on the implementation of domain adaptation modules with different strategies in Step 1 and Step 2, the common global feature extraction operation in the two steps is extracted, and the two modules are fused to form a hybrid domain adaptation module; Specifically, global feature extraction is performed only once on the sample, and then the global features are input into the difference-based and adversarial-based domain adaptation modules respectively to calculate MMD distance and domain discrimination loss; Step 3.2: Loss Fusion; The two domain losses are weighted and summed using weighting factors to obtain the overall loss of the hybrid domain adaptation module. The calculation is shown in the following formula; and the domain adaptation loss is added to the calculation of the total training loss of the baseline model DPDN through the weighting factor again, together with the pose estimation loss, to constitute the overall training loss of the baseline model DPDN. in It is a balancing factor.

Images