A passive domain synthesis-reality object adaptive recognition method based on optimal transmission

Abstract

The application discloses a kind of passive domain synthesis-reality object adaptive identification method based on optimal transmission, which comprises the following steps: first, download VISDA-C dataset, and divide and augment it;Then, based on the network structure of the residual network, use three-stage features to perform optimal transmission calculation at different levels, and construct a new residual framework;The divided data set is sent to the neural network for training until the deep learning network converges, and the weight file is obtained;Finally, the trained neural network and weight file are used to detect test images, and the classification result is output.The application solves the problems that source domain data cannot be obtained during the training phase and the quality of pseudo-labels is too low in the current passive domain adaptive framework, and improves the accuracy of the passive domain adaptive model.

Description

Technical Field

[0001] This invention relates to a passive domain synthesis-adaptive recognition method for real-world objects based on optimal transmission, belonging to the field of computer vision technology. Background Technology

[0002] Deep learning has made tremendous progress in recent years, but its over-reliance on large amounts of data means that the transfer performance of models is limited by the probability distribution of the training data, making it difficult to apply to image recognition tasks under different weather conditions, lighting, and scenes. To reduce the cost of collecting and labeling real-world object images, synthetic images are often used as an aid. Synthetic images have the advantages of being abundant, low-cost, and requiring no labeling. Pre-training the model on synthetic images and then applying it to real-world object recognition using effective domain adaptation methods can improve model reusability and reduce the cost of training the model from scratch.

[0003] However, there is a significant difference between synthetic object images and real object images, namely, the data distributions of the source and target domains are different, resulting in domain differences, which is one challenge. In practical applications, due to reasons such as data privacy and data ownership, it is impossible to obtain a large number of synthetic object images, and only source domain models trained from synthetic object images can be obtained, which is another challenge. In order to simultaneously address the stringent challenges of different image data distributions and passive domain data, the main idea of ​​passive domain adaptation is proposed: the model obtained by training on source domain data serves as the source domain model, and the target domain data has a similar but different distribution from the source domain data. The target domain data is then used to train on the source model to obtain a target model that adapts to the target domain data.

[0004] Currently, passive domain adaptation methods can be mainly divided into domain image generation-based adaptation methods and self-supervised adaptation methods. Domain image generation-based adaptation methods aim to compensate for inaccessible source domains by synthesizing proxy source domain data using a generative model, followed by conventional unsupervised domain adaptation. Standard adversarial learning is then used to learn cross-domain invariant feature patterns for further adaptation. However, such methods introduce additional frameworks and learnable parameters, thus consuming more computational resources.

[0005] Currently, self-supervised adaptation methods require steps such as mining hidden structures in the target domain, generating pseudo-labels, and fine-tuning the model. Xia et al. (Xia H, Zhao H, Ding Z. Adaptive adversarial network for source-free domain adaptation[C] / / Proceedings of the IEEE / CVF internationalconference on computer vision.2021:9010-9019) first adaptively divided target instances into sets of source similarity and source dissimilarity, and then designed a class-aware contrastive module for cross-set distribution alignment. The idea is to enhance the compactness of target instances from the same category and reduce cross-domain differences, thereby promoting effective knowledge transfer from the source model to the target data. Qiu et al. (Source-free domain adaptation via avatar prototype generation and adaptation[J]. arXiv preprint arXiv:2106.15326,2021) used a student-teacher model to generate pseudo-labels for the target data, and through contrastive learning, aligned the features derived from unlabeled target samples with source prototypes with the same category labels, achieving cross-domain prototype adaptation. Both of the above classic methods suffer from low-quality generated pseudo-labels. Due to the significant domain shift between synthetic and real images, a large number of low-quality pseudo-labels are easily generated, which can lead to a substantial performance drop. While the latter method utilizes a student-teacher model to improve the robustness of pseudo-labels to some extent, it introduces a new learnable framework and tightly couples the model parameters of the teacher and student, which also leads to performance bottlenecks. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the existing technologies by proposing a passive domain synthesis-real-world object adaptive recognition method based on optimal transmission. This method effectively solves the problems of source domain data not being available during domain adaptation framework training and poor accuracy caused by poor quality of real-world object pseudo-labels in the current process of recognizing objects from synthetic images to real-world images.

[0007] The technical solution of this invention is:

[0008] A passive domain synthesis-adaptive recognition method for real-world objects based on optimal transmission, comprising the following steps:

[0009] Step 1: Dataset Acquisition: Download the VISDA-C dataset and divide it into training set, test set, and data augmentation set, with a ratio of 6:1 between the training set and the test set;

[0010] Step 2: Constructing a Neural Network: Based on the residual network structure, optimal transfer calculations at different levels are performed using features from three stages. Specifically, branches are selected for weight vector extraction and image feature extraction. Then, bottleneck layers are used to extract features more effectively. Finally, prediction is performed through fully connected (FC) layers to construct a new residual network.

[0011] Step 3: Neural Network Training: Feed the augmented VISDA-C dataset into the neural network constructed in Step 2 for training until the network converges, and obtain the trained neural network and weight file.

[0012] Step 4: Detect the image based on the trained neural network and weight file.

[0013] As a preferred embodiment, in step 2-1: In the residual network, obtain the weight data and image features of each layer of the network, use the weight data as the mapped vector in the optimal transmission, use the image features as the mapping vector in the optimal transmission, and perform optimal transmission calculation on the two.

[0014] Step 2-2: Discretely classify the optimal transmission calculation results to obtain the one-hot code of the corresponding category for each sample, which serves as the optimal transmission class for each sample;

[0015] Steps 2-3: Decompose the original cross-entropy-based domain adaptation loss into three levels of multi-classification loss: shallow, medium, and high. Use the optimal transmission class as the target value and the network output as the calculated value. Calculate the distribution of loss based on transmission reliability.

[0016] Here, transmission reliability refers to the transmission cost in optimal transmission; the calculated loss distribution refers to the loss calculated for different samples based on transmission reliability, the source of the calculation is the sample feature distribution, and the target of the calculation is the maximum possible class information obtained based on transmission reliability.

[0017] Preferably, in step 2-1, the optimal transfer result between features at different levels and model weights is calculated as follows:

[0018]

[0019] Where f represents the features of the source domain network weights and the features of the target domain image output, respectively; C is used to calculate the similarity between the two, expressed as:

[0020]

[0021]

[0022] Where γ represents the transmission cost from the source domain to the target domain, <·,·>F represents the Fibonacci inner product; 1 is a dimension vector, μ represents the empirical distribution, and C A As stated in the above formula, it represents the degree of similarity, where A represents a matrix, n represents the dimension, and the subscripts s and t represent the source domain and the target domain, respectively.

[0023] Preferably, the loss distribution calculation in step 2-3 is as follows: the features of different levels p extracted are predicted using the FC layer, and the pseudo-label is obtained by converting the optimal transmission category value obtained from the optimal transmission into one-hot encoding. The specific optimized loss is expressed as follows:

[0024]

[0025] In the formula, G represents the trained network, L represents the loss, the subscript cls represents the classification loss, the subscripts P and P represent the losses of different layers, p = 1, 2, 3 represent shallow, mid, and high layers respectively, B represents the batch size, and x i Let γ represent the i-th sample. * From the formula above.

[0026] Preferably, step 3 includes the following steps:

[0027] Step 3-1: Based on the size of the target in the dataset, use data augmentation methods such as random pruning and random flipping, as well as the Mixup augmentation method;

[0028] Step 3-2: Optimize using stochastic gradient descent, with the learning rate decaying three times from the initial value, so that the neural network can achieve better distillation results;

[0029] Step 3-3: Try different training hyperparameters on the neural network and train it. When the loss function converges or the maximum number of iterations is reached, stop training to obtain the domain-adapted network file and weight file.

[0030] Preferably, in step 3-1 of this invention, the original image is randomly cropped with a cropping padding size of 4. Each image is then augmented using Mixup random image fusion with a fusion probability of 0.3 or 0.7.

[0031] Preferably, step 3-2 of the present invention involves learning rate decay at different stages of the training process, i.e., at iteration counts of epoch = 40, 60, and 80.

[0032] Preferably, step 4 of the present invention includes the following steps:

[0033] Step 4-1: Feed the test image into the improved residual network backbone to obtain the optimal transmission calculation results at three levels;

[0034] Step 4-2: Calculate a weighted average of the results from the three levels to obtain pseudo-labels;

[0035] Step 4-3: Calculate the accuracy based on the pseudo-labels as the final result.

[0036] Beneficial effects:

[0037] 1. Based on the residual network as the backbone network, this invention uses multi-layer features to calculate the optimal transmission results at different levels. The transmission results at different stages of the network are weighted and fused, which allows the network to learn the global and local semantic information of the features, improves the reliability of pseudo-labels, and enhances the robustness of the model.

[0038] 2. This invention solves the key problem of the inability to obtain source domain data during training by introducing optimal transmission. The improved cross-entropy loss, through loss decomposition, can more effectively mine the structural information contained in the target domain data, thereby improving the accuracy of target image classification. Attached Figure Description

[0039] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0040] Figure 2 This is a flowchart of step 2 of the present invention;

[0041] Figure 3 This is a flowchart of step 3 of the present invention;

[0042] Figure 4 This is a flowchart of step 4 of an embodiment of the present invention;

[0043] Figure 5 The image shows the test results of an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] like Figure 1 As shown, this invention provides a passive domain synthesis-real-object adaptive recognition method based on optimal transmission, comprising the following steps:

[0046] Step 1: Obtain the dataset and divide the VISDA-C dataset into training and test sets.

[0047] Step 2: Construct a neural network, using a residual network as the backbone network, and adding bottleneck layers and FC layers as predictions for the student network. When constructing the network, the network weights of the three layers are used as the mapped vectors for optimal transmission, and the corresponding output network features are used as the mapping vectors. Optimal transmission calculation is then performed on both.

[0048] Step 3: Train the neural network. Feed the divided VISDA-C dataset into the neural network for training until the network converges.

[0049] Step 4: Classification test. Use the trained neural network and weight file to detect the category in the test image to verify the adaptation effect.

[0050] In this embodiment, the present invention specifically adopts the following technical solution:

[0051] Step 1) Download the data from the VISDA-C dataset website and divide the data;

[0052] Step 2) First, select three branches to extract weight vectors and image features, then use the bottleneck layer to extract features more effectively, and finally use the FC layer for prediction.

[0053] like Figure 2 As shown, step 2 of the present invention includes the following steps:

[0054] Step 201) Extract the features of the third layer, the fourth layer, and the FC layer to allow the network to learn important features;

[0055] Step 202) Calculate the optimal transfer result between features at different levels and model weights:

[0056]

[0057] Where f represents the features of the source domain network weights and the features of the target domain image output, respectively. C is used to calculate the similarity between the two, specifically through cosine distance.

[0058]

[0059]

[0060] Where γ represents the transmission cost from the source domain to the target domain, and <·,·>F represents the Fibonacci inner product. 1 is a dimension vector. μ represents the empirical distribution.

[0061] Step 203) The category with the minimum transmission cost calculated from the target domain samples can be regarded as the reliable category. The optimal transmission result form at different levels is changed to one-hot coding, which becomes the target vector of the final loss.

[0062] Step 204) Finally, the FC layer is used to predict the features extracted from different levels p, and the prediction results are constrained using pseudo-labels to train the network (the pseudo-labels are obtained by converting the optimal transmission class value obtained from the optimal transmission into one-hot encoding), as shown in the following formula:

[0063]

[0064] In the formula, G represents the trained network, L represents the loss, the subscript cls represents the classification loss, the subscripts P and P represent the losses of different layers, p = 1, 2, 3 represent shallow, mid, and high layers respectively, B represents the batch size, and x i Let γ represent the i-th sample. * From step 202 above.

[0065] like Figure 3 As shown, step 3 of the present invention includes the following steps:

[0066] Step 301: Before training the network, recalculate the mean and variance of the dataset and normalize the data;

[0067] Step 302: Use random weights as initial weights, set the learning rate to 0.0001, the number of iterations to 100, and the batch size to 32, etc.; and at rounds 40, 60, and 80, decrease the learning rate from the initial value by 0.01 to enable the neural network to achieve better detection results.

[0068] Step 303: Perform data augmentation on the input image using random cropping and random flipping, and then perform Mixup augmentation. Train the image and stop training when the loss function converges or the maximum number of iterations is reached to obtain the self-distilled weight file.

[0069] like Figure 4 As shown, step 4 of the present invention includes the following steps:

[0070] Step 401: Feed the test image into the improved residual network backbone to obtain the convolutional features of the three stages;

[0071] Step 402: Perform optimal transmission calculations on the features of the three stages respectively;

[0072] Step 403: Obtain the prediction results of the three-stage set by weighted averaging as pseudo-labels to further constrain network training.

[0073] Figure 5 To test the detection results using the method of this invention, training and testing were performed on two GeForce RTX 2080Ti graphics cards. During training, the weight decay in the stochastic gradient descent algorithm was set to 0.0001. In each training round, the loss function value was output to the terminal for easy observation of the overall convergence. At the end of each round, the network was validated using a test set, and the validation results were output. Each domain adaptation task will indicate the source and target domains. For example, task AD represents the prediction result of the network trained on source domain A in target domain D. This invention achieves a classification accuracy of 95.18% on VISDA-C.

[0074] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A passive domain synthesis-adaptive recognition method for real-world objects based on optimal transmission, characterized in that, Includes the following steps: Step 1: Obtain the dataset: Download the VISDA-C dataset, divide it into training and testing sets, and perform data augmentation; Step 2: Construct the Neural Network: Using the residual network structure as the backbone network, the target convolutional neural network is divided into three layers—shallow, mid-layer, and high-layer—based on its depth and original structure. Each layer outputs the feature embedding vector of the image. The optimal transfer calculation at different levels is performed using the features from the three stages: First, the three branches—shallow, mid-layer, and high-layer—are selected to extract the weight vector and image features. Then, the bottleneck layer is used to extract features. Finally, the FC layer is used for prediction, outputting a B*N embedding vector, where B is the batch size and N is the number of target categories. Specifically, it includes: Step 2-1: In the residual network, obtain the weight data and image features of each layer. Use the weight data as the mapped vector in optimal transmission and the image features as the mapping vector in optimal transmission, and perform optimal transmission calculation on both. The optimal transfer result between features at different levels and model weights is calculated as follows: ， Where f represents the features of the source domain network weights and the features of the target domain image output, respectively; The similarity between the two can be calculated using C, and expressed as: ， ， in, Represents the transmission cost from the source domain to the target domain, <·,·>F represents the Fibonacci inner product; 1 is a dimension vector. Representing the empirical distribution, C A Indicates the degree of similarity, where A represents a matrix, n represents the dimension, and the subscripts s and t represent the source and target domains, respectively. Step 2-2: Perform discrete classification on the optimal transmission calculation results to obtain the one-hot code, i.e., pseudo-label, of each sample's corresponding category, which serves as the optimal transmission class for each sample; Steps 2-3: Decompose the original cross-entropy-based domain adaptation loss into three levels of multi-class classification loss: shallow, mid-level, and high-level. Use the optimal transmission class as the objective value and the network output as the calculated value. Calculate the loss distribution based on transmission reliability. That is, use the FC layer to predict the features extracted from different levels p. The specific optimized loss is expressed as: ， In the formula, G represents the trained network, L represents the loss, the subscript cls represents the classification loss, the subscripts P and P represent the losses of different layers, p=1,2,3 represent shallow, mid, and high layers respectively, B represents the batch size, and x i This represents the i-th sample; Step 3: Neural Network Training: Feed the augmented VISDA-C dataset into the neural network constructed in Step 2 for training until the network converges, and obtain the trained neural network and weight file. Step 4: Perform detection on the test set based on the neural network and weight file trained in Step 3.

2. The passive domain synthesis-real-object adaptive recognition method based on optimal transmission according to claim 1, characterized in that, Step 3 includes the following steps: Step 3-1: Based on the size of the target in the dataset, use data augmentation methods such as random pruning and random flipping, as well as the Mixup augmentation method to augment the dataset into partitions; Step 3-2: Optimize using stochastic gradient descent, with the learning rate decaying three times from the initial value; Step 3-3: Train the neural network using different training hyperparameters. Stop training when the loss function converges or the maximum number of iterations is reached, and obtain the domain-adapted neural network file and weight file.

3. The passive domain synthesis-adaptive recognition method for real-world objects based on optimal transmission according to claim 2, characterized in that, In step 3-1, the original image is randomly cropped with a crop padding size of 4. Each image is then augmented using Mixup random image fusion with fusion probabilities of 0.3 and 0.

7.

4. The passive domain synthesis-adaptive recognition method for real-world objects based on optimal transmission according to claim 2, characterized in that, In step 3-2, the learning rate decays at different stages of the training process.

5. The passive domain synthesis-real-object adaptive recognition method based on optimal transmission according to claim 1, characterized in that, Step 4 includes the following steps: Step 4-1: Feed the test image into the improved residual network backbone to obtain the optimal transmission calculation results at three levels; Step 4-2: Calculate a weighted average of the results from the three levels to obtain pseudo-labels; Step 4-3: Further constrain network training based on the pseudo-labels obtained in Step 4-2, and use the accuracy calculated based on the pseudo-labels as the final result.

Images