A cross-domain data enhancement method for domain generalization task

Abstract

The application discloses a cross-domain data enhancement method for a domain generalization task, comprising sampling a plurality of samples from original data sets of K domains, and inputting the samples into a network model in sequence for training. The cross-domain data enhancement method for the domain generalization task obtains new feature maps by performing cross-domain data enhancement operations on feature maps of another domain through non-repeated random selection of historical statistics of one of the domains, the feature maps of the remaining domains are also subjected to the cross-domain data enhancement operations in the same way, and finally, the new feature maps of all the domains are obtained, then forward propagation is continued, the training of a batch of sample data is completed, and through an exponential moving average algorithm, the statistics of new batches of samples can be continuously collected, and finally, the obtained statistics approximately represent complete style information of the original data sets of each domain, so that the purpose of extracting the overall style information of the domain is achieved, and the extracted style information is real and comprehensive.

Description

Technical Field

[0001] This invention belongs to the field of domain generalization in computer vision, and specifically relates to a cross-domain data augmentation method for domain generalization tasks. Background Technology

[0002] Deep learning has demonstrated outstanding performance in both academia and industry. However, classic deep learning methods are typically only applicable to independent and identically distributed (IID) scenarios, where the training and test sets share the same distribution. In practical applications, training and actual test data often exhibit distributional discrepancies due to differences in generation mechanisms and acquisition methods, such as differences in image style or shooting environment. In such out-of-distribution (OOD) scenarios, models trained on source domain data (training set) will experience a significant performance drop when tested on target domain data (test set). Domain generalization methods aim to generalize the model to an unknown target test domain with a different data distribution than the source domain, simultaneously accessing data from multiple source domains to train a more robust model. Collecting training data from multiple domains is often difficult; therefore, many domain generalization methods employ special data augmentation techniques to synthesize entirely new samples across domains, enabling the model to learn more generalized feature representations.

[0003] Classic data augmentation methods are roughly as follows: rotation, inversion transformation, scaling, translation, contrast transformation, etc. However, the above data augmentation methods cannot achieve the effect of cross-domain synthetic samples. Cross-domain data augmentation technology often requires more special design schemes. At present, cross-domain data augmentation methods for domain generalization tasks can be roughly divided into two categories: image-level data augmentation and feature-level data augmentation. (1) Image-level data augmentation: Data augmentation methods based on Fourier transform exchange amplitude spectra between multiple source domain images, and finally synthesize new images by inverse Fourier transform; methods based on style transfer models use pre-trained models to exchange styles between multiple source domains to generate new images. (2) Feature-level data augmentation: Cross-domain data augmentation is performed on the feature representations of the intermediate layers of the network. It mainly involves exchanging the amplitude spectra after Fourier transform between feature maps of different source domains, or exchanging the statistics (mean and variance) of feature maps to normalize the original features, and finally synthesizing more diverse feature representations.

[0004] The main idea behind the two main cross-domain data augmentation methods is to combine image styles from other domains with the original image content to generate entirely new samples. The aim is to enable the model to capture invariant image content while ignoring changing image styles, ultimately hoping that the model will focus more on image content when testing on a target domain with unknown styles, thus achieving better generalization. However, these cross-domain data augmentation methods are often performed one-to-one, exchanging information between individual data points from two different domains while preserving content information to synthesize a new sample. This often leads to unreasonable style transformation because the style information extracted from a single data point cannot represent the overall distribution of that domain, and there are also certain differences between different data points within the same domain, resulting in incomplete style information in the synthesized sample. Summary of the Invention

[0005] The purpose of this invention is to address the problems raised in the background art by proposing a cross-domain data augmentation method for domain-generalized tasks.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] The present invention proposes a cross-domain data augmentation method for domain generalization tasks, which includes sampling a number of samples from the original datasets of K domains, and inputting each sample into a network model for training in sequence. When passing through a specified network layer, the feature map output by the network layer is randomly selected to perform data augmentation operation with a preset probability.

[0008] After selecting to perform data augmentation, the mean and variance of the feature maps are calculated according to their respective domains to form K domain statistics. For the first batch of samples, the statistics are defined as historical statistics. For subsequent batches of samples, the statistics obtained from the current batch of samples are updated to the historical statistics of the previous batch of samples based on the exponential moving average algorithm.

[0009] Without repetition, randomly select historical statistics from one domain to perform cross-domain data augmentation on the feature map of another domain to obtain a new feature map. Continue this process until new feature maps are obtained for all domains, thus completing the data augmentation operation.

[0010] After completing the data augmentation operation, continue forward propagation to complete the training of this batch of network models.

[0011] Preferably, several samples are sampled from the original datasets of K domains, and each sample is sequentially input into the network model for training, including:

[0012] From the original datasets of K domains Several samples are collected, one of which is represented as Where DK This represents the original dataset of the Kth domain. This indicates that the original dataset for each domain has N elements. k Data, B K This represents a sample sampled from the original dataset of the Kth domain. Represents each neighborhood N in a sample B There are N data points, and N B <N k .

[0013] Preferably, the feature map output by the network layer is randomly selected with a preset probability to perform data augmentation operations, including:

[0014] If you choose not to perform data augmentation, then continue forward propagation;

[0015] If data augmentation is selected, the mean and variance of the feature maps are calculated for each domain, resulting in K domain statistics, calculated using the following formulas:

[0016] μ k , σ k =mean(f k ), var(f k )

[0017]

[0018]

[0019] The statistics for the K domains are represented as (μ = {μ1, μ2, ..., μ...} K}, σ={σ1,σ2…,σ K}), μ k Let σ represent the mean of the k-th domain. k Let f represent the variance of the k-th neighborhood. k Let {f1, f2, ..., fk} represent the feature map of the k-th domain, and let {f1, f2, ..., fkk} be the feature map output from this network layer when a sample is input. K}, f k，h，w Let represent the pixel value of the feature map of the k-th neighborhood with height H and width W respectively, and coordinates (h, w).

[0020] Preferably, for the first batch of samples, the statistic is defined as a historical statistic. For subsequent batches of samples, the historical statistic of the previous batch of samples is updated with the statistic obtained from the current batch of samples based on the exponential moving average algorithm, including:

[0021] For the first batch of samples, the statistic is defined as the historical statistic (μ). old , σ old ), and save it locally;

[0022] For subsequent batches of samples, the statistic (μ) obtained from the current batch of samples is calculated using the exponential moving average algorithm. new , σ new Update the historical statistics (μ) of the previous batch of samples. old , σ old The updated formula is as follows:

[0023] μ old =mμ old +(1-m)μ new

[0024] σ old =mσ old +(1-m)σ new

[0025] Where m represents the weight parameter, and m = 0.9.

[0026] Preferably, without repetition, historical statistics from one domain are randomly selected to perform cross-domain data augmentation on the feature map of another domain to obtain new feature maps, until new feature maps for all domains are obtained, thus completing the data augmentation operation, including:

[0027] Cross-domain data augmentation is a style normalization operation, and the calculation formula is as follows:

[0028]

[0029] Wherein, formula (1) represents the historical statistics of the i-th domain being randomly selected for the feature map f of the j-th domain. j Perform style normalization to obtain new feature maps μ i and σ i Let μ represent the mean and variance of the i-th domain, respectively. j and σ j Let represent the mean and variance of the j-th domain, respectively;

[0030] Finally, new feature maps for all domains are obtained.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] This cross-domain data augmentation method for domain generalization tasks obtains new feature maps by randomly selecting historical statistics from one domain without repetition and performing cross-domain data augmentation on the feature maps of another domain. The feature maps of other domains are also augmented in the same way, resulting in new feature maps for all domains. Then, forward propagation is continued to complete the training of a batch of sample data. Furthermore, by using the exponential moving average algorithm, statistics of new batches of samples can be continuously collected. The final statistics approximately represent the complete style information of the original dataset for each domain, achieving the goal of extracting overall style information of the domain in a realistic and comprehensive manner. Attached Figure Description

[0033] Figure 1 This is a flowchart of the cross-domain data augmentation method for domain-generalization tasks according to the present invention.

[0034] Figure 2 This is a block diagram of the cross-domain data enhancement method of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] It should be noted that when a component is referred to as being "connected" to another component, it can be directly connected to the other component or there may be an intervening component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to limit the application.

[0037] like Figure 1-2 As shown, a cross-domain data augmentation method for domain-generalization tasks includes:

[0038] S1. Sample several samples from the original datasets of K domains and input each sample into the network model for training.

[0039] Specifically, from the original datasets of K domains Several samples are collected, one of which is represented as Where D K This represents the original dataset of the Kth domain. This indicates that the original dataset for each domain has N elements. k Data, B KThis represents a sample sampled from the original dataset of the Kth domain. Represents each neighborhood N in a sample B There are N data points, and N B <N k .

[0040] S2. When passing through a specified network layer, randomly select whether to perform data augmentation operations on the feature map output by that network layer with a preset probability, including:

[0041] It should be noted that the preset probability value is not restricted and ranges from (0, 1), such as 0.5. By performing data augmentation operations on the specified network layer, the extracted domain information becomes more complete and comprehensive. The network model can be any convolutional neural network or Transformer network, and the specified network layer can be any convolutional layer, normalization layer, or activation function layer in a convolutional neural network, or a self-attention layer in a Transformer network.

[0042] If you choose not to perform data augmentation, then continue forward propagation;

[0043] If data augmentation is selected, the mean and variance of the feature maps are calculated for each domain, resulting in K domain statistics (i.e., mean and variance), calculated as follows:

[0044] μ k , σ k =mean(f k ), var(f k )

[0045]

[0046]

[0047] The statistics for the K domains are represented as (μ = {μ1, μ2, ..., μ...} K}, σ={σ1,σ2…,σ K}), μ k Let σ represent the mean of the k-th domain. k Let f represent the variance of the k-th neighborhood. k Let {f1, f2, ..., fk} represent the feature map of the k-th domain, and let {f1, f2, ..., fkk} be the feature map output from this network layer when a sample is input. K}, f k，h，w Let represent the pixel value of the feature map of the k-th neighborhood with height H and width W respectively, and coordinates (h, w).

[0048] S3. For the first batch of samples, the statistic is defined as the historical statistic.

[0049] Specifically, for the first batch of samples, the statistic is defined as the historical statistic (μ). old , σ old The statistics are then saved locally, and subsequent batches of samples continuously update the historical statistics using the obtained statistics.

[0050] S4. Randomly select historical statistics from one domain without repetition to perform cross-domain data augmentation on the feature map of another domain to obtain a new feature map. Repeat this process until new feature maps for all domains are obtained, thus completing the data augmentation operation.

[0051] To illustrate the selection of data domains: If there are four domains, a, b, c, and d, when performing data augmentation on domain a, a domain is randomly selected from b, c, and d to augment the data in domain a (e.g., domain c is selected); when performing data augmentation on domain b, a domain is randomly selected from the remaining a and d domains to augment the data in domain b (e.g., domain a is selected); when performing data augmentation on domain c, a domain is randomly selected from the remaining b and d domains to augment the data in domain c (e.g., domain d is selected); and when performing data augmentation on domain d, the remaining b domains are used to augment the data in domain d.

[0052] Specifically, such as Figure 2 As shown, cross-domain data augmentation is a style normalization operation, and the calculation formula is as follows:

[0053]

[0054] Wherein, formula (1) represents the historical statistics of the i-th domain being randomly selected for the feature map f of the j-th domain. j Perform style normalization to obtain new feature maps μ i and σ i Let μ represent the mean and variance of the i-th domain, respectively. j and σ j Let represent the mean and variance of the j-th domain, respectively;

[0055] The feature maps for other domains are also subjected to style normalization in the same way, ultimately resulting in new feature maps for all domains. (There are a total of K new feature maps), and then the forward propagation continues to complete the training of the first batch of sample data (that is, complete the training of this batch of network models).

[0056] S5. After completing the training of the first batch of sample data, when the next batch is input into the specified network layer (i.e., whether to perform data augmentation operation on the feature map output by the network layer is randomly selected with a preset probability), repeat S2 to obtain the statistics (μ) of this batch of samples. new , σ newThen, based on the exponential moving average algorithm, the statistic (μ) obtained from this batch of samples is... new , σ new Update the historical statistics (μ) of the previous batch of samples. old , σ old The updated formula is as follows:

[0057] μ old =mμ old +(1-m)μ new

[0058] σ old =mσ old +(1-m)σ new

[0059] Where m represents the weight parameter, and m = 0.9;

[0060] Then repeat S4 until the training of this batch of samples is complete.

[0061] Subsequent batches of samples were also augmented in the same way to complete the training. As more batches of samples participated in the training, the final collected mean and variance will approximate the distribution of the original dataset, thus achieving the goal of extracting overall style information of the domain.

[0062] This cross-domain data augmentation method for domain generalization tasks obtains new feature maps by randomly selecting historical statistics from one domain without repetition and performing cross-domain data augmentation on the feature maps of another domain. The feature maps of other domains are also augmented in the same way, resulting in new feature maps for all domains. Then, forward propagation is continued to complete the training of a batch of sample data. Furthermore, by using the exponential moving average algorithm, statistics of new batches of samples can be continuously collected. The final statistics approximately represent the complete style information of the original dataset for each domain, achieving the goal of extracting overall style information of the domain in a realistic and comprehensive manner.

[0063] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] The embodiments described above are merely specific and detailed examples of the embodiments described in this application, and should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this patent application should be determined by the appended claims.

Claims

1. A cross-domain data augmentation method for domain-generalization tasks, characterized in that: The cross-domain data augmentation method for domain-generalization tasks includes: Several image samples are sampled from the original datasets of K domains, and each image sample is sequentially input into the network model for training. When passing through a specified network layer, the feature map output by the network layer is randomly selected to perform data augmentation operation with a preset probability. After selecting to perform data augmentation, the mean and variance of the feature maps are calculated according to their respective domains to form K domain statistics. For the first batch of image samples, the statistics are defined as historical statistics. For subsequent batches of image samples, the statistics obtained from the current batch of image samples are updated with the historical statistics of the previous batch of image samples based on the exponential moving average algorithm. Without repetition, randomly select historical statistics from one domain to perform cross-domain data augmentation on the feature map of another domain to obtain a new feature map. Continue this process until new feature maps are obtained for all domains, thus completing the data augmentation operation. After completing the data augmentation operation, continue forward propagation to complete the training of this batch of network models.

2. The cross-domain data augmentation method for domain-generalization tasks as described in claim 1, characterized in that: The process of sampling several image samples from the original datasets of K domains and sequentially inputting each image sample into the network model for training includes: From the original datasets of K domains Several image samples are collected, one of which is represented as... Where D K This represents the original dataset of the Kth domain. This indicates that the original dataset for each domain has N elements. k Data, B K This represents an image sample sampled from the original dataset of the Kth domain. Representing each neighborhood N in an image sample B There are N data points, and N B <N k .

3. The cross-domain data augmentation method for domain-generalization tasks as described in claim 2, characterized in that: The step of randomly selecting whether to perform data augmentation on the feature map output by the network layer with a preset probability includes: If you choose not to perform data augmentation, then continue forward propagation; If data augmentation is selected, the mean and variance of the feature maps are calculated for each domain, resulting in K domain statistics, calculated using the following formulas: μ k ,σ k =mean(f k ),var(f k ) The statistics for the K domains are represented as (μ={μ1,μ2…,μ) K }, σ={σ1,σ2…,σ K }), μ k Let σ represent the mean of the k-th domain. k Let f represent the variance of the k-th neighborhood. k Let {f1, f2, ..., fk} represent the feature map of the k-th region, and let {f1, f2, ..., fkk} be the feature map output from this network layer when an image sample is input to it. K }, f k,h,w Let represent the pixel value of the feature map of the k-th neighborhood with height H and width W respectively, and coordinates (h, w).

4. The cross-domain data augmentation method for domain-generalization tasks as described in claim 3, characterized in that: For the first batch of image samples, the statistics are defined as historical statistics. For subsequent batches of image samples, the statistics obtained from the current batch of image samples are updated with the historical statistics of the previous batch of image samples based on the exponential moving average algorithm, including: For the first batch of image samples, the statistic is defined as the historical statistic (μ). old ,σ old ), and save it locally; For subsequent batches of image samples, the statistical measure (μ) obtained from the current batch of image samples is calculated based on the exponential moving average algorithm. new ,σ new Update the historical statistics (μ) of the previous batch of image samples. old ,σ old The updated formula is as follows: m old =mm old +(1-m)m new s old =mσ old +(1-m)σ new Where m represents the weight parameter, and m = 0.

9.

5. The cross-domain data augmentation method for domain-generalization tasks as described in claim 3, characterized in that: The process of randomly selecting historical statistics from one domain without repetition to perform cross-domain data augmentation on the feature map of another domain to obtain a new feature map continues until new feature maps for all domains are obtained, thus completing the data augmentation operation, including: Cross-domain data augmentation is a style normalization operation, and the calculation formula is as follows: Wherein, formula (1) represents the historical statistics of the i-th domain being randomly selected for the feature map f of the j-th domain. j Perform style normalization to obtain new feature maps μ i and σ i Let μ represent the mean and variance of the i-th domain, respectively. j and σ j Let represent the mean and variance of the j-th domain, respectively; Finally, new feature maps for all domains are obtained.

Images