Cross-domain progressive adaptation method based on diffusion model

Abstract

The invention discloses a cross-domain progressive adaptation method based on a diffusion model, and belongs to the technical field of deep learning. According to the method, progressive domain adaptation of a target domain is realized step by step through an energy-based pseudo label labeling strategy and a mixed feature guided diffusion model. The method comprises the following steps of: firstly, generating a pseudo label of target domain data by combining domain characteristics and uncertainty of instances, and ensuring effective labeling of target domain category information; thirdly, pre-training the data of the source domain through a diffusion model, adding noise to the data of the target domain and the intermediate domain, and gradually converting the features of the target domain into the style of the source domain through a reverse denoising mode; in the denoising process, a mixed feature guiding strategy is adopted, structural information and category information of an image are balanced, and an encoder is guided to keep feature consistency of a target domain and style characteristics of a source domain in cross-domain feature conversion. And finally, the diffusion model is removed, and the target domain data is classified and identified only by using the encoder and the classifier. According to the method, the calculation cost of inter-domain migration is remarkably reduced, the adaptive capacity of the model is enhanced, and an efficient and robust solution is provided for the domain to adapt to related tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and specifically designs a cross-domain progressive adaptation method based on a diffusion model. Background Art

[0002] With the rapid development of artificial intelligence and deep learning, although it is easy to produce massive amounts of unlabeled data in many fields, it is very expensive to obtain data labels, especially in specific industries such as medical imaging diagnosis and industrial defect detection. The high cost and time-consuming labeling make traditional machine learning methods difficult to deploy in practice and hit a bottleneck. At the same time, the "domain shift" problem is ubiquitous and is closely related to daily life. For example, in security monitoring image recognition, the image data of the source domain and the target domain differ significantly due to different camera positions, angles, and lighting; in the application of intelligent voice assistants, regional accents, language habits, and background noise interference make voice data show obvious domain differences; in the field of social media sentiment analysis, the distribution of text semantic features varies due to changes in user groups, topic popularity, and expression styles, and the service provider industry is particularly affected. The data distribution between different customers varies greatly, resulting in poor adaptability and poor performance of pre-trained models in new customers or new environments. To cope with this dilemma, domain adaptation technology came into being. It uses labeled data in the source domain and unlabeled data in the target domain to help the model break through the limitations of data distribution differences. In recent years, it has been widely used in many tasks such as security monitoring image classification, speech recognition of intelligent voice assistants, and sentiment analysis of social media texts. By adjusting the pre-training model and combining the characteristics of the target domain data, the distribution differences between the source domain and the target domain are gradually eliminated, which not only avoids the high labeling cost, but also improves the generalization ability and adaptability of the model in the new environment, injecting strong impetus into the development of various fields.

[0003] The progressive domain adaptation technology based on the diffusion model shows unique advantages in solving practical problems. Taking the development of autonomous driving as an example, in the process of developing autonomous driving technology, it is necessary to adapt the vehicle's intelligent driving algorithm to various road conditions and environments. However, it is not realistic and costly to collect a large amount of data on all possible road conditions. The progressive domain adaptation technology based on the diffusion model can use a small amount of sample data in real road conditions to gradually expand and generate a variety of simulated driving scenarios, covering various complex road conditions such as congested streets in cities, rugged bends in remote mountainous areas, and slippery roads in extreme weather. In this way, the autonomous driving algorithm can be fully trained in massive and diverse virtual scenes, greatly improving the ability to cope with complex real-world scenes, effectively solving the practical problems faced by data collection and algorithm training in the process of autonomous driving research and development, and providing strong support for the rapid development of intelligent transportation. Summary of the invention

[0004] In order to solve the problems of large differences in training data distribution, poor model generalization ability and low classification accuracy currently existing in cross-domain tasks, the present invention provides a cross-domain progressive adaptation method based on a diffusion model. This method gradually realizes progressive domain adaptation of the target domain through an energy-based pseudo-label annotation strategy and a diffusion model guided by hybrid features. First, pseudo labels of target domain data are generated in combination with domain features and instance uncertainty to ensure effective annotation of target domain category information. Next, the source domain data is pre-trained through a diffusion model, noise is added to the target domain and intermediate domain data, and the features of the target domain are gradually converted into the style of the source domain by reverse denoising. In the denoising process, a hybrid feature guidance strategy is adopted to balance the structural information and category information of the image, and guide the encoder to maintain the feature consistency of the target domain and the style characteristics of the source domain in cross-domain feature conversion. Finally, the diffusion model is removed, and only the encoder and classifier are used to classify and identify the target domain data. The technical problem proposed by the present invention is solved as follows:

[0005] Specific content:

[0006] A cross-domain progressive adaptation method based on a diffusion model comprises the following steps:

[0007] Step 1: Obtain the required source domain, intermediate domain, and target domain datasets, perform preliminary processing on the images in the datasets, and divide them into training sets and test sets;

[0008] Step 2, preprocessing the obtained data in different domains and normalizing them;

[0009] Step 3: pre-training on the source domain data obtained in step 2 to obtain a pre-trained encoder, classifier, and diffusion model;

[0010] Step 4: Train on the intermediate domain and the target domain, calculate the sample energy using the pre-trained encoder and classifier according to the energy function, the energy function provides the lowest energy value for the correct answer, filter out the pseudo labels with high energy, calculate the uncertainty based on the sample energy, and use the uncertainty as the weight coefficient to enable the nearest centroid classifier to optimize the pseudo labels;

[0011] Step 5: Train on the intermediate domain and target domain based on the obtained pseudo-labels, use the encoder to map the input to the latent space, add noise to the image in the latent space, and use the noise prediction network to denoise. The pseudo-labels are used to construct the optimization process of the hybrid feature-guided encoder. By guiding the optimization to update the source domain feature space, the features of the intermediate domain and the target domain are gradually mapped to the source domain space, thereby effectively achieving cross-domain feature alignment;

[0012] Step 6: After the training is completed, the diffusion model is removed and the trained encoder and classifier are used for classification prediction;

[0013] Step 7: Train the domain adaptation classification network built in steps 3, 4, 5, and 6 on the intermediate domain and target domain datasets.

[0014] Step 1 of the present invention comprises:

[0015] Step 1-1, obtain relevant data sets of the source domain, intermediate domain, and target domain, perform preliminary processing on the data sets, delete irrelevant images, and crop the images to a uniform size;

[0016] Step 1-2: After step 1-1, the data set obtained is divided into a training set and a test set to facilitate subsequent training and testing of the network.

[0017] Step 2 of the present invention comprises:

[0018] Step 2-1, use imageio and opencv to complete the reading of the image, read the corresponding image file according to the input file name, file path and image size, and return an array containing the image;

[0019] Step 2-2, use shuffle and file name to batch read the data obtained in step 2-1, shuffle its index according to the number of data files, and then complete the data reading according to the file corresponding to the index;

[0020] Step 2-3, call the method in transforms to process the data and normalize the data read in step 2-2.

[0021] Step 3 of the present invention comprises:

[0022] Step 3-1, the data in step 2 is first used to pre-train the encoder. In the present invention, the image data is first extracted through the Resnet-18 network, and then converted into two-dimensional features through a pooling layer to reduce the amount of calculation, and then extracted through a fully connected layer and a layer normalization to obtain a pre-trained encoder.

[0023] Step 3-2, input the image features extracted in 3-1 into the classifier, obtain the classification prediction result, optimize the classifier, and obtain a pre-trained classifier.

[0024] Step 3-3, after completing the pre-training of the encoder, use it to convert the source domain data into feature representations, and use these features to train the diffusion model to obtain a pre-trained diffusion model.

[0025] Step 4 of the present invention comprises:

[0026] Step 4-1, perform transfer training on the intermediate domain and the target domain, input the pre-trained encoder into the feature z, and then input z into the classifier to obtain the initial pseudo label y.

[0027] Step 4-2, based on the initial pseudo-label y combined with the energy function E(z, y), provide the lowest energy for the correct answer and provide higher energy for all other incorrect answers to obtain the sample energy value, filter the pseudo-labels by the energy threshold, and remove high-energy and unreliable pseudo-label samples, where E(z, y) is:

[0028]

[0029] Among them, y * is the pseudo label after filtering.

[0030] Step 4-3, based on the pseudo-label samples selected in step 3-2, the pseudo-labels are optimized using the nearest centroid classifier. During the optimization process, the cosine distance from the sample to the class centroid is calculated, and the class centroid is updated to improve the accuracy of the pseudo-label. At the same time, uncertainty calculation is used as the sample weight of the nearest centroid classifier to increase the attention paid to high-confidence samples, thereby improving the distribution quality of the pseudo-labels and obtaining the optimized pseudo-labels. The process is as follows:

[0031]

[0032]

[0033] where w k is the uncertainty weight of the kth class, g t It's an encoder.

[0034] Step 5 of the present invention comprises:

[0035] Step 5-1, mapping the intermediate domain and target domain data into feature vector z through an encoder;

[0036] Step 5-2, sample random noise ∈, ∈~N(0, I) from the standard normal distribution, sample time step t from the uniform distribution from 1 to T, t~Uniform{1,...,T}, and perform Gaussian diffusion noise on z until the result is infinitely close to the Gaussian distribution:

[0037]

[0038]

[0039] Among them, z0 represents the characteristic vector of step 5-1, t represents the time step of the diffusion process, and z trepresents the result of adding noise after t steps, s ranges from 0 to t and represents the current time step, a s ~(0,1) is a hyperparameter that controls the noise injection rate, Represents a s The cumulative result of

[0040] Step 5-3, during the denoising process, according to the formula:

[0041]

[0042] Among them, ∈ θ Represents the noise prediction network, and replaces z in step 5-4 t Transform to get the predicted estimate of z0

[0043] Step 5-4, Mixup feature fusion z′ with z0, and then and z′ are respectively passed through the classifier to obtain label information and y′, combining the two into a pseudo label Perform gradient guidance to make the optimization direction closer to the source domain:

[0044]

[0045] y′=φ C (z′),

[0046]

[0047] Among them, λ t is the fusion coefficient related to the time step, φ C is the classifier, w is the gradient guidance coefficient, z t-1 The result of the denoising step is obtained after multiple iterations, which guides the optimization direction of the encoder.

[0048] In step 5-5, the training process starts from the intermediate domain close to the source domain and gradually transitions to the target domain data. By gradually introducing the target domain data, the model gradually adapts to the characteristics of the target domain during training.

[0049] Step 6 of the present invention comprises:

[0050] In step 6-1, only the encoder and diffusion model are fine-tuned during the training process, and the classifier parameters are frozen.

[0051] Step 6-2: After training, remove the diffusion model during inference and use only the encoder and classifier for classification prediction.

[0052] The present invention proposes a cross-domain progressive adaptation method based on a diffusion model, which has the following advantages and beneficial effects compared with the prior art:

[0053] The present invention combines an energy-based pseudo-label optimization strategy with a gradient-guided diffusion model, which can generate pseudo-labels for the target domain more accurately and gradually smooth the distribution differences between the source domain and the target domain. Compared with existing methods, the present invention generates accurate pseudo-labels by combining domain features with instance uncertainty, reduces label noise, and improves cross-domain adaptation accuracy and robustness. It is particularly suitable for scenarios where data is scarce or the cost of annotation is high. The gradient-guided diffusion model gradually transforms the target domain features into the source domain style by introducing noise and reverse denoising, while optimizing feature alignment and maintaining the consistency of target domain features, thereby enhancing the model's adaptability and generalization performance in different data domains. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The overall flow chart of the method of the present invention is

[0055] Figure 2 This is the overall network architecture diagram

[0056] Figure 3 This is a schematic diagram of the diffusion model.

[0057] Figure 4 This is a schematic diagram of pseudo label construction

[0058] Figure 5 This is a visualization diagram of the classification results of the training process. DETAILED DESCRIPTION

[0059] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0060] The present invention is implemented including:

[0061] The overall training process of the method proposed by the present invention is as follows: Figure 1 As shown in Figure 2, the overall network architecture is as follows: Figure 2 As shown, the specific construction steps of the domain progressive adaptation method in the embodiment of the present invention are as follows:

[0062] Step 1: Obtain the required source domain, intermediate domain, and target domain datasets, perform preliminary processing on the images in the datasets, and divide them into training sets and test sets;

[0063] Step 2, preprocessing the obtained data in different domains and normalizing them;

[0064] Step 3: pre-training on the source domain data obtained in step 2 to obtain a pre-trained encoder, classifier, and diffusion model;

[0065] Step 4: Train on the intermediate domain and the target domain, calculate the sample energy using the pre-trained encoder and classifier according to the energy function, the energy function provides the lowest energy value for the correct answer, filter out the pseudo labels with high energy, calculate the uncertainty based on the sample energy, and use the uncertainty as the weight coefficient to enable the nearest centroid classifier to optimize the pseudo labels;

[0066] Step 5: Train on the intermediate domain and target domain based on the obtained pseudo-labels, use the encoder to map the input to the latent space, add noise to the image in the latent space, and use the noise prediction network to denoise. The pseudo-labels are used to construct the optimization process of the hybrid feature-guided encoder. By guiding the optimization to update the source domain feature space, the features of the intermediate domain and the target domain are gradually mapped to the source domain space, thereby effectively achieving cross-domain feature alignment;

[0067] Step 6: After the training is completed, the diffusion model is removed and the trained encoder and classifier are used for classification prediction;

[0068] Step 7: Train the domain adaptation classification network built in steps 3, 4, 5, and 6 on the intermediate domain and target domain datasets.

[0069] Step 1 of the present invention comprises:

[0070] Step 1-1, obtain relevant data sets of the source domain, intermediate domain, and target domain, perform preliminary processing on the data sets, delete irrelevant images, and crop the images to a uniform size;

[0071] Step 1-2: After step 1-1, the data set obtained is divided into a training set and a test set. The data in the training set accounts for 80% of the total data set, and the remaining data is divided into a test set to facilitate the subsequent training and testing of the model.

[0072] Step 2 of the present invention comprises:

[0073] Step 2-1, use imageio and opencv to complete the reading of the image, read the corresponding image file according to the input file name, file path and image size, and return an array containing the image;

[0074] Step 2-2, use shuffle and file name to batch read the data obtained in step 2-1, shuffle its index according to the number of data files, and then complete the data reading according to the file corresponding to the index;

[0075] Step 2-3, call the method in transforms to process the data and normalize the data read in step 2-2.

[0076] Step 3 of the present invention comprises:

[0077] Step 3-1, the data in step 2 is first used to pre-train the encoder. In the present invention, the image data is first extracted through the Resnet-18 network, and then converted into two-dimensional features through a pooling layer to reduce the amount of calculation, and then extracted through a fully connected layer and a layer normalization to obtain a pre-trained encoder.

[0078] Step 3-2, input the image features extracted in 3-1 into the classifier, obtain the classification prediction result, optimize the classifier, and obtain a pre-trained classifier.

[0079] Step 3-3, after completing the pre-training of the encoder, use it to convert the source domain data into feature representations, and use these features to train the diffusion model to obtain a pre-trained diffusion model.

[0080] Step 4 of the present invention comprises:

[0081] Step 4-1, perform transfer training on the intermediate domain and the target domain, input the pre-trained encoder into the feature z, and then input z into the classifier to obtain the initial pseudo label y.

[0082] Step 4-2, based on the initial pseudo-label y combined with the energy function E(z, y), provide the lowest energy for the correct answer and provide higher energy for all other incorrect answers to obtain the sample energy value, filter the pseudo-labels by the energy threshold, and remove high-energy and unreliable pseudo-label samples, where E(z, y) is:

[0083]

[0084] Among them, y * is the pseudo label after filtering.

[0085] Step 4-3, based on the pseudo-label samples selected in step 3-2, the pseudo-labels are optimized using the nearest centroid classifier. During the optimization process, the cosine distance from the sample to the class centroid is calculated, and the class centroid is updated to improve the accuracy of the pseudo-label. At the same time, uncertainty calculation is used as the sample weight of the nearest centroid classifier to increase the attention paid to high-confidence samples, thereby improving the distribution quality of the pseudo-labels and obtaining the optimized pseudo-labels. The process is as follows:

[0086]

[0087]

[0088] where w k is the uncertainty weight of the kth class, g t It's an encoder.

[0089] Step 5 of the present invention comprises:

[0090] Step 5-1, mapping the intermediate domain and target domain data into feature vector z through an encoder;

[0091] Step 5-2, sample random noise ∈, ∈~N(0, I) from the standard normal distribution, sample time step t from the uniform distribution from 1 to T, t~Uniform{1,...,T}, and perform Gaussian diffusion noise on z until the result is infinitely close to the Gaussian distribution:

[0092]

[0093]

[0094] Among them, z0 represents the characteristic vector of step 5-1, t represents the time step of the diffusion process, and z t represents the result of adding noise after t steps, s ranges from 0 to t and represents the current time step, a s ~(0,1) is a hyperparameter that controls the noise injection rate, Represents a s The cumulative result of

[0095] Step 5-3, during the denoising process, according to the formula:

[0096]

[0097] Among them, ∈ θ Represents the noise prediction network, and replaces z in step 5-4 t Transform to get the predicted estimate of z0

[0098] Step 5-4, Mixup feature fusion z′ with z0, and then and z′ are respectively passed through the classifier to obtain label information and y′, combining the two into a pseudo label Perform gradient guidance to make the optimization direction closer to the source domain:

[0099]

[0100]

[0101]

[0102] Among them, λ t is the fusion coefficient related to the time step, φ C is the classifier, w is the gradient guidance coefficient, z t-1 The result of the denoising step is obtained after multiple iterations, which guides the optimization direction of the encoder.

[0103] In step 5-5, the training process starts from the intermediate domain close to the source domain and gradually transitions to the target domain data. By gradually introducing the target domain data, the model gradually adapts to the characteristics of the target domain during training.

[0104] Step 6 of the present invention comprises:

[0105] In step 6-1, only the encoder and diffusion model are fine-tuned during the training process, and the classifier parameters are frozen.

[0106] Step 6-2: After training, remove the diffusion model during inference and use only the encoder and classifier for classification prediction.

[0107] The present invention combines an energy-based pseudo-label optimization strategy with a gradient-guided diffusion model, which can gradually reduce the distribution difference between the source domain and the target domain, and improve the accuracy and robustness of cross-domain feature alignment. Compared with existing methods, the present invention generates accurate pseudo-labels by combining domain features with instance uncertainty, reduces label noise, improves cross-domain adaptation accuracy, optimizes cross-domain feature alignment, maintains target domain feature consistency, and enhances the model's adaptability and generalization performance in different data domains.

[0108] The present invention provides a cross-domain progressive adaptation method based on a diffusion model. The above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A cross-domain progressive adaptation method based on a diffusion model, characterized in that: The following steps are involved: Step 1: Obtain the required source domain, intermediate domain, and target domain datasets, perform preliminary processing on the images in the datasets, and divide them into training sets and test sets; Step 2, preprocessing the obtained data in different domains and normalizing them; Step 3: pre-training on the source domain data obtained in step 2 to obtain a pre-trained encoder, classifier, and diffusion model; Step 4: Train on the intermediate domain and the target domain, calculate the sample energy using the pre-trained encoder and classifier according to the energy function, the energy function provides the lowest energy value for the correct answer, filter out the pseudo labels with high energy, calculate the uncertainty based on the sample energy, and use the uncertainty as the weight coefficient to enable the nearest centroid classifier to optimize the pseudo labels; Step 5: Train on the intermediate domain and target domain based on the obtained pseudo-labels, use the encoder to map the input to the latent space, add noise to the image in the latent space, and use the noise prediction network to denoise. The pseudo-labels are used to construct the optimization process of the hybrid feature-guided encoder. By guiding the optimization to update the source domain feature space, the features of the intermediate domain and the target domain are gradually mapped to the source domain space, thereby effectively achieving cross-domain feature alignment; Step 6: After the training is completed, the diffusion model is removed and the trained encoder and classifier are used for classification prediction; Step 7: Train the domain adaptation classification network built in steps 3, 4, 5, and 6 on the intermediate domain and target domain datasets.

2. The method according to claim 1, characterized in that Step 2 includes: Step 2-1, use imageio and opencv to complete the reading of the image, read the corresponding image file according to the input file name, file path and image size, and return an array containing the image; Step 2-2, use shuffle and file name to batch read the data obtained in step 2-1, shuffle its index according to the number of data files, and then complete the data reading according to the file corresponding to the index; Step 2-3, call the method in transforms to process the data and normalize the data read in step 2-2.

3. The method according to claim 1, characterized in that Step 3 includes: Step 3-1, the data in step 2 is first used to pre-train the encoder. In the present invention, the image data is first extracted through the Resnet-18 network, and then converted into two-dimensional features through a pooling layer to reduce the amount of calculation, and then extracted through a fully connected layer and a layer normalization to obtain a pre-trained encoder. Step 3-2, input the image features extracted in 3-1 into the classifier, obtain the classification prediction result, optimize the classifier, and obtain a pre-trained classifier. Step 3-3, after completing the pre-training of the encoder, use it to convert the source domain data into feature representations, and use these features to train the diffusion model to obtain a pre-trained diffusion model.

4. The method according to claim 1, characterized in that Step 4 includes: Step 4-1, perform transfer training on the intermediate domain and the target domain, input the pre-trained encoder into the feature z, and then input z into the classifier to obtain the initial pseudo label y. Step 4-2, based on the initial pseudo-label y combined with the energy function E(z, y), provide the lowest energy for the correct answer and provide higher energy for all other incorrect answers to obtain the sample energy value, filter the pseudo-labels by the energy threshold, and remove high-energy and unreliable pseudo-label samples, where E(z, y) is: Among them, y * is the pseudo label after filtering. Step 4-3, based on the pseudo-label samples selected in step 3-2, the pseudo-labels are optimized using the nearest centroid classifier. During the optimization process, the cosine distance from the sample to the class centroid is calculated, and the class centroid is updated to improve the accuracy of the pseudo-label. At the same time, uncertainty calculation is used as the sample weight of the nearest centroid classifier to increase the attention paid to high-confidence samples, thereby improving the distribution quality of the pseudo-labels and obtaining the optimized pseudo-labels. The process is as follows: where w k is the uncertainty weight of the kth class, g t It's an encoder.

5. The method according to claim 1, characterized in that Step 5 includes: Step 5-1, mapping the intermediate domain and target domain data into feature vector z through an encoder; Step 5-2, sample random noise ∈, ∈~N(0, I) from the standard normal distribution, sample time step t from the uniform distribution from 1 to T, t~Uniform{1,...,T}, and perform Gaussian diffusion noise on z until the result is infinitely close to the Gaussian distribution: Among them, z0 represents the characteristic vector of step 5-1, t represents the time step of the diffusion process, and z t represents the result of adding noise after t steps, s ranges from 0 to t and represents the current time step, a s ~(0,1) is a hyperparameter that controls the noise injection rate, Represents a s The cumulative result of ; Step 5-3, during the denoising process, according to the formula: Among them, ∈ θ Represents the noise prediction network, and replaces z in step 5-4 t Transform to get the predicted estimate of z0 Step 5-4, Mixup feature fusion z′ with z0, and then and z′ are respectively passed through the classifier to obtain label information and y′, combining the two into a pseudo label Perform gradient guidance to make the optimization direction closer to the source domain: Among them, λ t is the fusion coefficient related to the time step, φ C is the classifier, w is the gradient guidance coefficient, z t-1 The result of the denoising step is obtained after multiple iterations, which guides the optimization direction of the encoder. In step 5-5, the training process starts from the intermediate domain close to the source domain and then trains towards the target domain.

6. The method according to claim 1, characterized in that Step 6 includes: In step 6-1, only the encoder and diffusion model are fine-tuned during the training process, and the classifier parameters are frozen. Step 6-2: After training, remove the diffusion model during inference and use only the encoder and classifier for classification prediction.

Images