A domain adaptation image classification model training method and device

Abstract

The application discloses a domain self-adaption image classification model training method and device, the method comprises the following steps: preprocessing data in a source domain and a target domain, respectively obtaining a source domain data set composed of data samples of the source domain and a target domain data set composed of target domain data samples; using an attention mechanism to combine the source domain data set and the target domain data set to generate an intermediate domain data set; training an image classification network by using the source domain data set and the intermediate domain data set to generate a first image classification model; and training the first image classification model by using the intermediate domain data set and the target domain data set to obtain a domain self-adaption image classification model. The application combines an intermediate domain by using an attention mechanism, then performs multi-level feature alignment, finally divides the domain self-adaption process into two steps, and effectively solves the problem of performance decline of the image classification network caused by the large difference between the data distribution of the source domain and the target domain.

Description

Technical Field

[0001] This invention belongs to the field of image classification technology, and specifically relates to a domain-adaptive image classification model training method and apparatus. Background Technology

[0002] Image classification, a fundamental task in computer vision, has been widely applied in fields such as autonomous driving, video surveillance, and facial recognition. In recent years, deep learning-based image classification has been extensively explored and researched. These methods utilize a large number of publicly labeled samples to train convolutional neural networks with different structures to form deep image classification networks, achieving satisfactory results in terms of classification accuracy and speed on shared datasets. This approach effectively reduces the incompleteness caused by manually designed features and reconciles the contradiction between feature generality and task specificity.

[0003] As is well known, the effectiveness of deep networks for image classification relies on two fundamental assumptions: first, training and testing samples come from a common dataset or different datasets with similar distributions; second, the training phase has a massive amount of labeled samples. However, these assumptions are difficult to meet in most practical situations. Since domain differences mainly stem from the differences in background between the source and target domains, when the image styles and data distributions of the target and source domains differ significantly, a classification model trained on the source domain will struggle to generalize to the target domain. Secondly, since supervised samples in image classification tasks require labeling object categories, this becomes extremely labor-intensive or even impossible when there are many object types. Furthermore, the lack of a unified standard for manual labeling inevitably introduces human bias. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a domain adaptive image classification model training method and apparatus to solve the above-mentioned technical problems.

[0005] To achieve the above and other related objectives, this invention provides a domain-adaptive image classification model training method, comprising:

[0006] Preprocess the data in the source domain and the target domain to obtain the source domain dataset composed of data samples from the source domain and the target domain dataset composed of data samples from the target domain, respectively.

[0007] The source domain dataset and the target domain dataset are combined using an attention mechanism to generate an intermediate domain dataset;

[0008] The image classification network is trained using the source domain dataset and the intermediate domain dataset to generate a first image classification model;

[0009] The first image classification model is trained using the intermediate domain dataset and the target domain dataset to obtain a domain-adaptive image classification model.

[0010] In an optional embodiment of the present invention, the preprocessing of data in the source domain and the target domain to obtain a source domain dataset composed of data samples from the source domain and a target domain dataset composed of data samples from the target domain, respectively, includes:

[0011] The source domain data is normalized to obtain the source domain dataset.

[0012] The target domain data is obtained by performing a size normalization operation on the target domain data.

[0013] In an optional embodiment of the present invention, the step of using an attention mechanism to combine the source domain dataset and the target domain dataset to generate an intermediate domain dataset includes:

[0014] The residual attention network is used to train all images in the preprocessed source domain dataset to identify the most discriminative parts in each image and obtain the residual attention network model.

[0015] The fully connected layers of the residual attention network model are replaced with a class activation graph method to obtain an attention mechanism-generated image network;

[0016] The intermediate domain dataset is generated using the attention mechanism image generation network based on the source domain dataset and the target domain dataset. An image from the source domain and an image from the target domain are input to the attention mechanism image generation network to generate an intermediate domain image.

[0017] In an optional embodiment of the present invention, the step of generating the intermediate domain dataset using the attention mechanism image generation network based on the source domain dataset and the target domain dataset includes:

[0018] The system receives an input image from the source domain and an image from the target domain, and uses the attention mechanism to generate an image network to process the source domain image to generate a heatmap.

[0019] The heatmap is cropped out from the part that the attention mechanism generates the image network with the lowest recognition.

[0020] A local image from the target domain image at the same location as the cropped portion of the heatmap is pasted as a patch into the corresponding location in the heatmap to generate an intermediate domain image in the intermediate domain dataset.

[0021] In an optional embodiment of the present invention, the image classification network is trained using the source domain dataset and the intermediate domain dataset to generate a first image classification model, specifically including:

[0022] The source domain dataset and the intermediate domain dataset are sent to the feature extractor in the feature alignment module for feature extraction to obtain a feature map.

[0023] The image classification network is trained using the feature maps to generate the first image classification model.

[0024] In an optional embodiment of the present invention, the source domain dataset and the intermediate domain dataset are sent to the feature extractor in the feature alignment module for feature extraction to obtain a feature map, specifically including:

[0025] The source domain dataset and the intermediate domain dataset are fed into a low-order feature extractor to obtain a low-order feature map.

[0026] The low-order feature maps are fed into the mid-order feature extractor to obtain mid-order feature maps;

[0027] The intermediate-order feature map is fed into a higher-order feature extractor to obtain the final feature map used to train the image classification network.

[0028] In an optional embodiment of the present invention, the source domain dataset and the intermediate domain dataset are sent to the feature extractor in the feature alignment module for feature extraction to obtain a feature map, and the method further includes:

[0029] The low-order feature map in the feature map is fed into the pixel-level neighborhood discriminator in the feature alignment module for training, in order to classify the domain to which each pixel in the low-order feature map belongs.

[0030] The intermediate-order feature map in the feature map is fed into the global-level neighborhood discriminator in the feature alignment module for training to obtain the global features of the intermediate-order feature map.

[0031] In an optional embodiment of the present invention, training the first image classification model using the intermediate domain dataset and the target domain dataset to obtain a domain-adaptive image classification model includes:

[0032] The intermediate domain dataset and the target domain dataset are sent to the feature extractor in the feature alignment module for feature extraction to obtain a feature map.

[0033] The image classification network is trained using the feature maps to generate the domain-adaptive image classification model.

[0034] In an optional embodiment of the present invention, when a residual attention network is used to train all images in the preprocessed source domain dataset, the feature map H(x) output by the residual attention network can be represented as:

[0035] H(x) = (1 + M(x)) * T(x),

[0036] Where T(x) refers to the backbone branch of the residual attention network, which outputs a feature map for the network's input x, and M(x) refers to the mask branch of the residual attention network, which outputs a feature map for the network's input x.

[0037] To achieve the above and other related objectives, the present invention also discloses a domain-adaptive image classification model training device, characterized in that it comprises:

[0038] The data processing module is used to preprocess the data in the source domain and the target domain to obtain the source domain dataset composed of data samples from the source domain and the target domain dataset composed of data samples from the target domain, respectively.

[0039] The intermediate domain generation module is used to combine the obtained source domain dataset and the target domain dataset using an attention mechanism to generate an intermediate domain dataset.

[0040] The first step, the domain adaptation module, is used to train the image classification network using the source domain dataset and the intermediate domain dataset to generate a first image classification model.

[0041] The second step, the domain adaptation module, is used to train the first image classification model using the intermediate domain dataset and the target domain dataset to obtain a domain-adaptive image classification model.

[0042] Beneficial effects:

[0043] This invention first synthesizes an intermediate domain dataset from the source domain dataset and the target domain dataset to learn the category knowledge of the source domain data and the background knowledge of the target domain data. Then, a two-step domain adaptation process is performed. The first step trains the intermediate domain from the source domain dataset, and the second step trains the intermediate domain from the target domain. In the domain adaptation process, feature maps of different levels are used to train the two discriminators, which effectively solves the problem of image classification network performance degradation caused by the large difference in the distribution of source and target domain data. Attached Figure Description

[0044] Figure 1 This is a flowchart of a domain-adaptive image classification model training method according to the present invention.

[0045] Figure 2 for Figure 1The detailed flowchart of step S2.

[0046] Figure 3 for Figure 1 The detailed flowchart of a sub-step of step S3.

[0047] Figure 4 A block diagram of a domain-adaptive image classification model training device. Detailed Implementation

[0048] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.

[0049] It should be noted that the illustrations provided in this embodiment are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0050] The effectiveness of deep networks for image classification is attributed to two fundamental assumptions: first, training and testing samples come from a common dataset or different datasets with similar distributions; second, the training phase has a massive amount of labeled samples. However, these assumptions are difficult to meet in most practical situations. Since domain differences mainly stem from the differences in background between the source and target domains, when the image styles and data distributions of the target and source domains differ significantly, a classification model trained on the source domain will struggle to generalize to the target domain. Secondly, since supervised samples in image classification tasks require labeling object categories, this becomes extremely labor-intensive or even impossible when there are many object types. Furthermore, the lack of a unified standard for manual labeling inevitably introduces human bias.

[0051] In view of the problems in the prior art described above, this invention discloses a domain-adaptive image classification model training method. Now, in conjunction with... Figure 1 The present embodiment discloses a domain-adaptive image classification model training method, which includes the following steps:

[0052] S1, preprocess the data in the source domain and the target domain to obtain the source domain dataset composed of data samples from the source domain and the target domain dataset composed of data samples from the target domain, respectively.

[0053] S2, use an attention mechanism to combine the source domain dataset and the target domain dataset to generate an intermediate domain dataset;

[0054] S3, The image classification network is trained using the source domain dataset and the intermediate domain dataset to generate a first image classification model;

[0055] S4, the first image classification model is trained using the intermediate domain dataset and the target domain dataset to obtain a domain-adaptive image classification model.

[0056] The following is in conjunction with the appendix Figure 1 Appendix Figure 2 and appendix Figure 3 Let's discuss the implementation process of each step in detail:

[0057] First, step S1 is executed to preprocess the data in the source domain and the target domain, respectively obtaining the source domain dataset composed of data samples from the source domain and the target domain dataset composed of data samples from the target domain.

[0058] In this step, the source domain dataset and the target domain dataset are normalized to obtain the preprocessed source domain dataset and the preprocessed target domain dataset.

[0059] Next, execution step S2 is performed, using an attention mechanism to combine the source domain dataset and the target domain dataset to generate an intermediate domain dataset. See also... Figure 2 As shown, this step specifically includes:

[0060] S21, Use a residual attention network to train all images in the preprocessed source domain dataset to identify the most discriminative part of each image and obtain the residual attention network model.

[0061] It should be noted that the most discriminative part of each image refers to the part of the image that we need the most. For example, if we are training on an image of a bicycle, and the images we collect contain not only the bicycle image but also the surrounding background, then the bicycle image is the most discriminative part, while the surrounding background image is the part that the residual attention network considers to be the least discriminative.

[0062] It should be noted that each attention module in the residual attention network consists of a mask branch and a backbone branch. The backbone branch's function is to transform the network's input x into its output feature T(x), while the mask branch's output M(x) is equivalent to the soft weights of T(x). Therefore, the final feature map H(x) output by the residual attention network can be expressed as:

[0063] H(x) = (1 + M(x)) * T(x) (1)

[0064] Formula (1) can avoid the feature attenuation that is prone to occur in simple attention module stacking, and the feature H(x) output by the residual attention network can maintain the good quality of the original feature.

[0065] S22, replace the fully connected layers of the residual attention network model with a class activation graph method to obtain an attention mechanism image generation network;

[0066] It's important to note that Class Activation Map (CAM) is a crucial method for visualizing convolutional neural networks. It obtains a heatmap by weighted summation of feature maps. Global average pooling is performed on the input n-dimensional feature map, and the result is the weight of that feature map. Average pooling involves summing the values ​​of all pixels in the entire image and dividing by the total number of pixels to obtain a single weight. The formula for global average pooling is as follows:

[0067] (2)

[0068] h represents the height of the feature map, w represents the width of the feature map, and A represents the feature map itself. The weights of all generated feature maps are then calculated. Then, the weighted feature map is the heatmap generated by the attention mechanism.

[0069] S23, based on the source domain dataset and the target domain dataset, the intermediate domain dataset is generated using the attention mechanism-based image generation network, specifically:

[0070] The system receives an input image from the source domain and an image from the target domain, and uses the attention mechanism to generate an image network to process the source domain image to generate a heatmap.

[0071] The heatmap is cropped out from the part that the attention mechanism generates the image network with the lowest recognition.

[0072] A local image from the target domain image at the same location as the cropped portion of the heatmap is pasted as a patch into the corresponding location in the heatmap to generate an intermediate domain image in the intermediate domain dataset.

[0073] As an example, we take two images as input: the first image is from the source domain, and the second image is from the target domain. Using the class activation map, we can obtain a heatmap generated by the image network after the first image passes through an attention mechanism. We divide this heatmap into n*n grids and select the N parts with the lowest scores (those considered by the residual attention network to have the lowest recognition ability) as the parts of interest in the first image for cropping. We assume the two input images are of the same size. After determining the cropping position in the first image, we crop the corresponding position from the second image and paste it as a patch into the first image. For example, in a 224*224 image, if it is divided into 7*7 grids, then each image patch will be 32*32 pixels. Here, the value of N can be... up to n 2 (i.e., the entire image).

[0074] After the intermediate domain dataset is synthesized, step S3 is executed, which uses the source domain dataset and the intermediate domain dataset to train the image classification network to generate a first image classification model. This step specifically includes:

[0075] S31, the source domain dataset and the intermediate domain dataset are sent to the feature extractor in the feature alignment module for feature extraction to obtain a feature map;

[0076] S32, The image classification network is trained using the feature map to generate the first image classification model.

[0077] It should be noted that the feature alignment module includes at least a low-level feature extractor, a mid-level feature extractor, a high-level feature extractor, a pixel-level neighborhood discriminator, and a global-level neighborhood discriminator. The pixel-level neighborhood discriminator consists of a convolutional layer with a stride of 2 and a kernel size of 3, an average pooling layer, and a fully connected layer. The global-level neighborhood discriminator consists of an image-level neighborhood discriminator and a gradient flipping layer. Please refer to [link / reference]. Figure 3 As shown, in a specific embodiment, step S31 specifically includes:

[0078] S311, The source domain dataset and the intermediate domain dataset are fed into a low-order feature extractor to obtain a low-order feature map.

[0079] S312, The low-order feature maps are fed into the mid-order feature extractor to obtain mid-order feature maps;

[0080] S313, The intermediate-order feature map is fed into the high-order feature extractor to obtain the final feature map used to train the image classification network.

[0081] It should be noted that after obtaining the low-order feature map, the low-order feature map is sent to the pixel-level neighborhood discriminator in the feature extraction module for training, in order to classify the neighborhood to which each point on the low-order feature map belongs.

[0082] After obtaining the low-level feature map, the mid-level feature map is fed into the global-level domain discriminator in the feature extraction module for training, so as to perform overall domain classification on the mid-level feature map as a whole.

[0083] It should be noted that after the low-order input features Is and It are fed into the pixel-level domain discriminator Dp through the gradient flip layer, we use the squared difference loss function to train the pixel-level domain discriminator Dp. Here, Dp is a fully convolutional network with a kernel size of 1, and the loss function is as follows:

[0084] (3)

[0085] (4)

[0086] Where Is and It represent the feature images generated by the convolutional layer from the source and target domain images, respectively, and the width of the feature image is W and the height is H.

[0087] The global-level neighborhood discriminator consists of a gradient flipping layer and an image-level neighborhood discriminator (Dg). Dg comprises convolutional layers with a stride of 2 and a kernel size of 3, an average pooling layer, and a fully connected layer. The feature map size is halved after each convolution, and the fully connected layer finally outputs the confidence score of whether the feature map belongs to the source or target domain. The loss function is shown below:

[0088] (5)

[0089] (6)

[0090] Finally, the above loss functions are added together according to formula (5) to obtain the overall domain classification loss:

[0091] (7)

[0092] It's important to note that when training the pixel-level and global neighborhood discriminators using the source domain S and the intermediate domain S1, the intermediate domain S' is essentially the target domain. Conversely, when training the pixel-level and global neighborhood discriminators using the intermediate domain S' and the target domain T, the intermediate domain S' serves as the source domain, and the target domain T is the target domain. In summary, regardless of whether the pixel-level and global neighborhood discriminators are trained using the source domain S and the intermediate domain S1, or the intermediate domain S' and the target domain T, the classification loss function follows Equation 3-7.

[0093] Finally, step S4 is executed, using the intermediate domain dataset and the target domain dataset to train the first image classification model to obtain a domain-adaptive image classification model. Specifically:

[0094] The intermediate domain dataset and the target domain dataset are sent to the feature extractor in the feature alignment module for feature extraction to obtain a feature map.

[0095] The image classification network is trained using the feature maps to generate the domain-adaptive image classification model.

[0096] In summary, this invention discloses a domain-adaptive image classification model training method comprising: preprocessing data in the source and target domains to obtain a source domain dataset composed of data samples from the source domain and a target domain dataset composed of data samples from the target domain; using an attention mechanism to combine the source and target domain datasets to generate an intermediate domain dataset; training an image classification network using the source and intermediate domain datasets to generate a first image classification model; and training the first image classification model using the intermediate and target domain datasets to obtain a domain-adaptive image classification model. This invention effectively solves the problem of image classification network performance degradation caused by significant differences in data distribution between the source and target domains by using an attention mechanism to synthesize an intermediate domain, performing multi-level feature alignment, and finally dividing the domain adaptation process into two steps.

[0097] like Figure 4As shown, this invention also proposes a domain-adaptive image classification model training device 100. This device applies the domain-adaptive image classification model training method described in the above embodiments. Specifically, the domain-adaptive image classification model training device 100 includes a data processing module 1, an intermediate domain generation module 2, a first-step domain adaptation module 3, and a second-step domain adaptation module 4. The data processing module 1 preprocesses the data in the source and target domains to obtain a source domain dataset composed of source domain data samples and a target domain dataset composed of target domain data samples, respectively. The intermediate domain generation module 2 uses an attention mechanism to combine the source domain dataset and the target domain dataset to generate an intermediate domain dataset. The first-step domain adaptation module 3 trains the image classification network using the source domain dataset and the intermediate domain dataset to generate a first image classification model. The second-step domain adaptation module 4 trains the first image classification model using the intermediate domain dataset and the target domain dataset to obtain a domain-adaptive image classification model.

[0098] The above description is merely a preferred embodiment of this application and an explanation of the technical principles used. Those skilled in the art should understand that the scope involved in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the inventive concept. For example, technical solutions formed by replacing the above-mentioned features with technical features with similar functions disclosed in this application (but not limited to) each other.

[0099] Apart from the technical features described in the specification, the other technical features are known to those skilled in the art. To highlight the innovative features of this invention, the other technical features will not be described in detail here.

Claims

1. A method for training a domain adaptive image classification model, the method comprising: The method comprises the following steps: preprocessing data in a source domain and a target domain to obtain a source domain data set composed of data samples of the source domain and a target domain data set composed of data samples of the target domain; generating an intermediate domain data set by using an attention mechanism in combination with the source domain data set and the target domain data set, specifically including: training all pictures in the source domain data set after preprocessing by using a residual attention network to identify the most discriminative part in each picture and obtain a residual attention network model; replacing the full connection layer of the residual attention network model with a class activation map method to obtain an attention mechanism generated picture network; generating the intermediate domain data set by using the attention mechanism generated picture network according to the source domain data set and the target domain data set; training an image classification network by using the source domain data set and the intermediate domain data set to generate a first image classification model, specifically including: performing feature extraction on the source domain data set and the intermediate domain data set in a feature extractor in a feature alignment module to obtain a feature map; training the image classification network by using the feature map to generate the first image classification model; training the first image classification model by using the intermediate domain data set and the target domain data set to obtain a domain adaptive image classification model; wherein the generating the intermediate domain data set by using the attention mechanism generated picture network according to the source domain data set and the target domain data set comprises: receiving an input of one image from the source domain and one image from the target domain, and processing the image from the source domain by using the attention mechanism generated picture network to generate a heat map; cropping the part considered to be the lowest by the attention mechanism generated picture network in the heat map; pasting the local image in the same position as the cropped part in the heat map in the image from the target domain as a patch to the corresponding position in the heat map to generate one intermediate domain image in the intermediate domain data set.

2. The domain-adaptive image classification model training method of claim 1, wherein, The preprocessing data in the source domain and the target domain to obtain the source domain data set composed of data samples of the source domain and the target domain data set composed of data samples of the target domain comprises: performing size normalization operation on the source domain data to obtain the source domain data set; performing size normalization operation on the target domain data to obtain the target domain data set. 3.The domain adaptive image classification model training method of claim 1, characterized in that, The feature extraction on the source domain data set and the intermediate domain data set in the feature extractor in the feature alignment module to obtain the feature map specifically comprises: feeding the source domain data set and the intermediate domain data set into a low-order feature extractor to obtain a low-order feature map; feeding the low-order feature map into a middle-order feature extractor to obtain a middle-order feature map; feeding the middle-order feature map into a high-order feature extractor to obtain the feature map used for training the image classification network.

4. The domain-adaptive image classification model training method according to claim 3, characterized in that, The feature extraction on the source domain data set and the intermediate domain data set in the feature extractor in the feature alignment module to obtain the feature map further comprises: The low-order feature map in the feature map is sent to a pixel-level domain discriminator in the feature alignment module for training, so as to classify the domain to which each pixel point on the low-order feature map belongs; The middle-order feature map in the feature map is sent to a global-level domain discriminator in the feature alignment module for training, so as to obtain the global feature of the middle-order feature map. 5.The domain adaptive image classification model training method of claim 1, wherein, The first image classification model is trained by using the intermediate domain data set and the target domain data set, so as to obtain a domain self-adaptive image classification model, including: The intermediate domain data set and the target domain data set are sent to a feature extractor in a feature alignment module for feature extraction, so as to obtain a feature map; The image classification network is trained by using the feature map, so as to generate the domain self-adaptive image classification model. 6.The domain adaptive image classification model training method of claim 1, wherein, When all pictures in the source domain data set after preprocessing are trained by using a residual attention network, a feature map H(x) output by the residual attention network can be represented as: H(x) = (1 + M(x)) * T(x), where T(x) refers to a main branch of the residual attention network, and outputs a feature map for the input x of the network, and M(x) refers to a mask branch of the residual attention network, and outputs a feature map for the input x of the network.

7. A domain-adaptive image classification model training device, characterized in that, It includes: A data processing module is configured to preprocess data in a source domain and a target domain, and obtain a source domain data set composed of data samples of the source domain and a target domain data set composed of data samples of the target domain; An intermediate domain generation module is configured to jointly use an attention mechanism to obtain the source domain data set and the target domain data set, and generate an intermediate domain data set, including: using a residual attention network to train all pictures in the source domain data set after preprocessing, so as to identify the most discriminative part in each picture, and obtain a residual attention network model; replacing a full connection layer of the residual attention network model with a class activation map method to obtain an attention mechanism generated picture network; and generating the intermediate domain data set by using the attention mechanism generated picture network according to the source domain data set and the target domain data set; A first step domain self-adaption module is configured to train an image classification network by using the source domain data set and the intermediate domain data set, so as to generate a first image classification model, including: sending the source domain data set and the intermediate domain data set to a feature extractor in a feature alignment module for feature extraction, so as to obtain a feature map; and training the image classification network by using the feature map, so as to generate the first image classification model; A second step domain self-adaption module is configured to train the first image classification model by using the intermediate domain data set and a target domain data set, so as to obtain a domain self-adaptive image classification model. The method comprises the following steps: receiving an input of one image from the source domain and one image from the target domain, and processing the image from the source domain by using the attention mechanism to generate a heat map; cutting the part of the heat map considered to be the lowest by the attention mechanism; and pasting the local image in the same position of the heat map as the cut part in the image from the target domain as a patch to the corresponding position of the heat map to generate one intermediate domain image in the intermediate domain data set.

Images