Image classification method based on domain generalization learning

Abstract

The application discloses an image classification method based on domain generalization learning, comprising the following steps: recording target domain data and source domain data and corresponding label information of the target domain data and the source domain data in a target domain data set D T And a source domain data set D S A domain discriminator is constructed by using a convolutional neural network Mobile V2 to generate a domain score; a basic ResNet network in a domain generalization network model is improved, and a hybrid domain attention is constructed; local maximum sensitivity is used instead of local random sensitivity; a loss function is constructed, and the network model is trained and optimized; the trained domain generalization network model is used to calculate the image classification accuracy on the target domain data set D T , and the image classification ability and the generalization ability of the current network model are judged. The application alleviates the negative transfer problem caused by false correlation in the deep domain generalization method, enhances the generalization ability of the network model, improves the image classification accuracy, and further realizes more flexible and accurate downstream applications.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image classification and domain generalization learning, and particularly relates to an image classification method based on domain generalization learning. BACKGROUND

[0002] In recent years, the superior performance of deep learning in the field of computer vision has attracted widespread attention in image classification. The image classification method mainly uses sufficient labeled data to accurately train a high-availability classifier model. However, for actual image classification application scenarios, it is very difficult to obtain labeled data. Transfer learning alleviates the bottleneck in the image classification task. The core idea of transfer learning is to reasonably utilize the similarity between old and new knowledge (such as data, tasks or models) to build a transfer bridge from old knowledge to new knowledge, so as to learn new knowledge faster and better, and complete the current task. In the image classification task, the transfer learning idea is used to find labeled data (source domain data) similar to the target data (target domain data) to build a model, and the similarity between the labeled data and the target data is used to complete the current image classification task faster and better.

[0003] At present, the main research direction of transfer learning is domain adaptation and domain generalization, and there are a large number of researches. Domain adaptation emphasizes the use of rich supervised knowledge in the source domain to enhance the training of the target domain to solve the learning problem in the target domain. Domain adaptation requires a large amount of labeled source domain data and a large amount of target domain data for model training. However, in actual scenarios, the target domain data may be very small or we may have no knowledge of the target domain data, and at this time, domain adaptation will be greatly limited. Unlike domain adaptation, domain generalization considers a more practical research scenario, which assumes that the samples of the target domain are not available during the training process, and the purpose is to learn a domain-invariant model in multiple source domains, so that it can be directly generalized to unknown target domains, thereby realizing effective knowledge reuse. The focus of domain generalization is to emphasize that the trained model has certain learning ability for any unknown target domain. Therefore, domain generalization is more suitable for real-world application scenarios.

[0004] Current methods for domain generalization learning in image classification can be divided into two categories: non-deep domain generalization methods and deep domain generalization methods. Non-deep domain generalization methods focus on extracting shallow features from low-dimensional images. The semantic information expressed by the extracted image features is limited, making it difficult to obtain excellent network models. Deep domain generalization methods can extract rich and representative features from complex high-dimensional images, training more ideal network models. However, most deep domain generalization methods are based on a few commonly used large-scale pre-trained convolutional neural network models. Convolutional neural networks pre-trained on ImageNet show a preference for color and texture in image classification. For images that are not primarily represented by shape, such as hand-drawn line drawings and cartoons, the performance of neural network image classification will decrease. Moreover, most deep domain generalization methods ignore whether all features extracted by the network model contribute to improving prediction accuracy. Focusing on all visual features can easily lead to the network model over-paying attention to irrelevant features, resulting in a decline in the network model's generalization ability. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose an image classification method based on domain generalization learning. This method can improve the neural network's preference for shape, make the features extracted by the network model more robust, and at the same time, during the feature extraction process, it can increase the network model's attention to task-related features, suppress task-irrelevant features, alleviate the negative transfer problem caused by spurious correlations, and further reduce the model's sensitivity to small perturbations in the input, thereby enhancing the model's generalization ability and improving the accuracy of image classification.

[0006] To achieve the above objectives, the technical solution provided by this invention is: an image classification method based on domain generalization learning, comprising the following steps:

[0007] 1) Preprocess all image data in the image dataset into tensor format. Select one domain from the preprocessed dataset as the target domain data and all remaining domains as the source domain data. Record the target domain data and source domain data along with their corresponding ground truth label information in the target domain dataset D. T and source domain dataset D S In the text, T represents the target domain, S represents the source domain, and the real label information includes the domain category information, semantic category information, and jigsaw puzzle arrangement category information of the current data;

[0008] 2) Construct the domain discriminator in the domain generalization network model using the lightweight convolutional neural network Mobile V2, based on the source domain dataset D obtained in step 1). S Train the Mobile V2 convolutional neural network and use it based on the source domain dataset D. S Domain category information for all data generates a domain score W.dc wherein, W dc is the two-dimensional vector data extracted by the domain discriminator;

[0009] 3) Improving the basic ResNet network in the domain generalization network model, the basic ResNet network is improved in three parts, the first part is to add a domain discriminator before the first layer convolutional layer conv1 of the basic ResNet network; the second part is to construct the output layer of the ResNet network, which is the semantic category output, the domain category output and the puzzle arrangement category output respectively; the third part is to add a mixed domain attention mechanism before and after the residual module, specifically the improved CBAM attention mechanism; the improvement of the CBAM attention mechanism is to input the domain score into the channel attention and spatial attention two modules respectively, wherein the channel attention and spatial attention are two modules of CBAM; after different dimension conversion of the domain score, the domain score is fused with the features extracted from the two modules of CBAM respectively;

[0010] 4) Improving the generalization error model LGEM, using local maximum sensitivity instead of local random sensitivity; the sensitivity is a measure that measures the output difference between the training sample z j and the invisible sample z u in its Q-neighborhood range; wherein Q is a hyperparameter, and the Q-neighborhood is a hypercube; z j is the jth training sample in the source domain dataset D S , z u is the sample generated after adding perturbation to z j , and z u is in the neighborhood range of z j ;

[0011] 5) Based on the domain discriminator and the improved ResNet network, a new type of domain generalization network model is constructed and trained, based on the output layer of the improved ResNet network, a loss function is constructed, and the local maximum sensitivity obtained in step 4) is introduced into the loss function, and the domain generalization network model is trained and optimized;

[0012] 6) Using the trained domain generalization network model to calculate the image classification accuracy on the target domain dataset D T , which is used to judge the image classification ability and generalization ability of the current domain generalization network model.

[0013] Further, in step 1), the preprocessing is using transforms on the target domain dataset D T and the source domain dataset D SThe image data is pre-processed, including image data brightness processing, image data saturation processing and image data pixel value normalization processing, and the image data is converted into a tensor format that can be directly input into the neural network; at the same time, the source domain data set D S According to the ratio of hyperparameter b, b% of image data is selected, and each image data is divided into k patches, and the patches are moved to reassemble image data with the same size as the original image; shuffling and reassembling will produce k! arrangements, in order to reduce the calculation, P kinds of puzzle arrangement are selected; at the same time of data preprocessing, the image data and the corresponding true label information are recorded in the target domain data set D T and the source domain data set D S , wherein the field class information of the current data has N classes, the semantic class information of the current data has Y classes, and the puzzle arrangement class information of the current data has P classes.

[0014] Further, in step 2), the image data in the source domain data set D S is used as input, and a feature extraction method is used to extract features from the image data, and the extracted feature map is input into the field discriminator to map to the field class information, and the field score is obtained; the field score W dc is calculated as follows:

[0015]

[0016] In the formula, , z is a training sample from the source domain data set D S , G ex (·) is a feature extractor, F dc (·) is a field discriminator, and the dimension of the field score is 1xN.

[0017] Further, the step 3) comprises the following steps:

[0018] 3.1) The first layer of the basic ResNet network is added to the field discriminator before the convolution layer conv1, which is used to obtain the field score W dc ; then the output layer of the ResNet network is constructed, and on the basis of the semantic class output of the basic ResNet network, the field class output and the puzzle arrangement class output are added; finally, the mixed domain attention mechanism is added before and after the residual module, which is the improved CBAM attention mechanism;

[0019] 3.2) Construct the field channel attention module in the mixed domain attention mechanism; use the source domain data set D SThe image data is used as input. First, the first convolutional layer conv1 in the ResNet network extracts features from the image data to obtain an intermediate feature map X. Then, the intermediate feature map X and the neighborhood score W are used to further extract features from the image data. dc Simultaneously, the input is fed into the domain channel attention module; in the domain channel attention module, the input intermediate feature map X is compressed in spatial dimension through max pooling and average pooling respectively to obtain the channel feature vector. and Wherein, the dimensions of the intermediate feature map X are H×W×C, where H represents the height of the intermediate feature map, W represents the width of the intermediate feature map, C represents the number of channels of the intermediate feature map, and the channel feature vectors are... and The dimensions of W are all 1×1×C; since the dimensions of the neighborhood score and the channel feature vector are different, a dimension transformation operation is required. dc and and Keep the same feature dimensions of 1×1×C, then W dc , and Perform element-wise summation and merging to generate the domain channel attention feature map W. ca The specific calculations are as follows:

[0020]

[0021] In the formula, G represents a real number. ews (·) is an element-wise summation and merging operation, G sh1 (·) represents a dimension transformation operation. and It is the channel feature vector output after max pooling and average pooling, and the neighborhood channel attention feature map W. ca The dimension is 1×1×C;

[0022] Then, the domain channel attention feature map W ca Multiplying with the intermediate feature map X completes the intermediate feature map weighting operation, thereby achieving adaptive feature optimization. The specific calculation is as follows:

[0023]

[0024] In the formula, G represents a real number. wt (·) is a weighting operation, and X' is an intermediate feature map with domain channel attention weighting, with dimensions H×W×C;

[0025] 3.3) Construct the domain spatial attention module in the hybrid domain attention mechanism; combine the intermediate feature map X' weighted by domain channel attention and the domain score W.dc The input field space attention module; in the field space attention module, the input field channel attention weighted intermediate feature map X' is compressed in channel dimension through maximum value pooling and average value pooling respectively, to obtain a spatial feature vector and wherein the dimension of the field channel attention weighted intermediate feature map X' is HxWxC, the dimension of the spatial feature vector and is HxWx1; since the field score and the spatial feature vector have different dimensions, a dimension transformation operation is needed to keep W dc and and the same feature dimension HxWx1, and then W dc , and connect based on the channel direction, and then a convolution operation is performed to reduce the feature map of HxWx3 to a field space attention feature map W sa of 1 channel, and the specific calculation is as follows:

[0026]

[0027] In the formula, represents a real number, G cov (·) is a convolution operation, G sh2 (·) is a dimension transformation operation, and are the spatial feature vectors output after maximum value pooling and average value pooling, and the dimension of the field space attention feature map W sa is HxWx1;

[0028] Then, the field space attention feature map W sa is multiplied by the intermediate feature map X to complete the intermediate feature map weighting operation to realize adaptive feature optimization, and the specific calculation is as follows:

[0029]

[0030] In the formula, represents a real number, G wt (·) is a weighting operation, X" is a field space attention weighted intermediate feature map, and the dimension is HxWxC.

[0031] Further, in step 4), the Q is a given hyperparameter, wherein the value range of the perturbation Δz is represented as Δz i is the i-th input feature of the perturbation Δz, and n is the number of input features; the j-th training sample z in the source domain data set D S ​j A corresponding set of invisible sample data S Q (z j ) is represented as S Q (z j ) = {z u |z u = z j + Δz; |Δz i | < Q}; the training sample z j and the invisible sample z u The output difference within the Q-neighborhood of the invisible sample is the local random sensitivity E SM , which is calculated as follows:

[0032]

[0033] where M represents the number of perturbations obtained within the Q-neighborhood, m represents the mth perturbation, represents the output distribution of the training sample;

[0034] Since the local random sensitivity is an average value of the output difference between multiple invisible samples and the training sample, the average value ignores the sensitivity extreme value, and therefore the local maximum sensitivity is used instead of the local random sensitivity; an adversarial method is used to find an adversarial perturbation r adv that maximizes the output difference between the invisible sample and the training sample; for the adversarial side, the adversarial perturbation maximizes the difference between the predicted result of the invisible sample after adding the perturbation and the actual result, while for the model, the adversarial perturbation minimizes the difference between the predicted result of the invisible sample after adding the perturbation and the actual result; therefore, the adversarial perturbation r adv is calculated as follows:

[0035] r adv : = argmax D[p(y j |z j ), p(y j |z j + r, θ)], ||r|| ≤ ε

[0036] L LMS = D[p(y j |z j ), p(y j |z j + r adv , θ)]

[0037] where the adversarial perturbation r advThe solution is to maximize the output difference between the unseen samples after perturbation and the training samples, where r represents the perturbation, θ represents the parameters of the domain generalization network model, ε represents the canonical constraint of the adversarial direction, and ε controls the boundary of the perturbation selection. j |z j ),p(y j |z j [+r,θ)] represents the Kullback-Leibler error between two distributions, where p(y j |z j ) represents the j-th training sample x with semantic category information y. j The true distribution, p(y) j |z j +r,θ) represents the j-th training sample x with semantic category information y. j The output distribution generated by the domain generalization network model with model parameters θ after adding a perturbation r; y j Represents the source domain dataset D S The j-th training sample z j semantic category information; L LMS p(y) represents the local maximum sensitivity. j |z j +r adv (θ) represents the j-th training sample x with semantic category information y. j Adding anti-perturbation r adv The output distribution is then generated by the domain generalization network model with model parameters θ;

[0038] Calculating the adversarial perturbation is an optimization problem, and it is difficult to find a precise value to represent it. Therefore, the following linear estimation method is used to obtain the adversarial perturbation r in the positive gradient direction. adv Approximate solution:

[0039]

[0040]

[0041] In the formula, g represents the positive gradient. This indicates that the gradient is calculated for the output difference between the unseen sample after perturbation and the training sample. The gradient refers to the derivative of the function, which represents the slope of the tangent line to the function at a given point. A positive gradient indicates the direction of the fastest ascent of the function at a given point.

[0042] Furthermore, in step 5), the total loss function includes three loss functions constructed based on the improved ResNet network output layer and the local maximum sensitivity obtained in step 4). The total loss function L total The calculation is as follows:

[0043] L total =L clf +αL jig +βL dc +γL LMS

[0044] In the formula, alpha, beta, gamma are three hyperparameters, used to balance all loss functions; L clf represents the semantic category output loss function of the data set, L dc represents the field category output loss function, L jig represents the jigsaw arrangement category output loss function, L LMS represents the local maximum sensitivity, L clf , L dc and L jig are all based on cross-entropy loss calculation, and L LMS is based on Kullback-Leibler calculation; wherein the loss function refers to calculating the difference between the predicted label information of the field generalization network model for the input image data and the real label information; the real label information is saved in the data set when preprocessed in step 1); after obtaining the loss function, the model parameters are solved by the gradient descent algorithm, and finally the trained and optimized field generalization network model is obtained, which can complete the image classification task.

[0045] Further, in step 6), the trained field generalization network model is directly generalized to the target domain data set D T , and the image classification task is completed on the target domain data set D T and the image classification accuracy is calculated, without the need for training and optimization of the field generalization network model, and the image classification ability and generalization ability of the current field generalization network model are judged by the image classification accuracy.

[0046] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0047] 1. The method of the present application enhances the shape preference of the field generalization network model, making the features extracted by the network model more robust, and further improving the image classification ability of the network model.

[0048] 2. The method of the present application proposes a hybrid domain attention mechanism, which improves the attention of the field generalization network model to task-related features and suppresses task-irrelevant features during feature extraction, thereby alleviating the negative transfer problem caused by false correlation.

[0049] 3. The method of the present application proposes a local maximum sensitivity, which makes the network model robust to small perturbations of the input, reduces the sensitivity of the network model to small perturbations of the input, and further improves the image classification ability and generalization ability of the network model.

[0050] 4. The method of the present invention has a wide range of applications in image classification tasks. It is simple to operate, highly adaptable, and has broad application prospects.

[0051] In summary, the method of this invention can learn a domain-generalized network model from multiple source domain data, which can be directly generalized to unknown target domains, thereby achieving effective knowledge reuse and alleviating bottlenecks in practical image classification applications. This invention alleviates the negative transfer problem caused by spurious correlations, a common issue in deep domain generalization methods for image classification, while reducing the network model's sensitivity to small input perturbations, enhancing its generalization ability, and improving image classification accuracy. Furthermore, it enables more flexible and accurate downstream applications. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the logic flow of the present invention.

[0053] Figure 2 This is a network structure diagram of the generalization model in the field of this invention.

[0054] Figure 3 This is a structural diagram of the hybrid domain attention mechanism of the present invention. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0056] like Figure 1 As shown, this embodiment discloses an image classification method based on domain generalization learning, which includes the following steps:

[0057] 1) Use the domain generalization public dataset PACS, taking one domain as the target domain data and all remaining domains as the source domain data. In this case, the target domain data contains one domain, and the source domain data contains three domains. The public dataset is not limited to PACS. Record the target domain data and source domain data along with their corresponding ground truth labels in the target domain dataset D. T and source domain dataset D S In the dataset, the true label information includes the domain category information, semantic category information, and jigsaw puzzle arrangement category information of the current data. For the PACS dataset, the target domain dataset has N=1 domain category information, and the source domain dataset has N=3 domain category information. The target domain dataset and the source domain dataset have the same semantic category information, with a total of Y=7 semantic category information. The target domain dataset has P=0 jigsaw puzzle arrangement category information, and the source domain dataset has P jigsaw puzzle arrangement category information. The transforms preprocessing method is used to preprocess the target domain dataset D. T and source domain dataset D SThe image data is pre-processed respectively, including image data brightness processing, image data saturation processing, image data pixel value normalization processing, and the image data is converted into a 225x225 tensor format; at the same time, the source domain data set D S According to the ratio of hyperparameter b, b% of image data is selected, and each image data is divided into 9 patches, and the patches are moved to reassemble image data with the same size as the original image; shuffling and reassembling will produce 9! arrangement modes, in order to reduce the calculation, 30 kinds of puzzle arrangement modes are selected.

[0058] 2) As shown in Figure 2 , a domain generalization model network is constructed, and a lightweight convolutional neural network Mobile V2 is used to construct a domain discriminator; using the image data in the source domain data set D S as input, using a feature extraction method to extract features from the image data, inputting the extracted feature map into the domain discriminator, mapping to the domain category information, and obtaining the domain score; the domain score W dc is calculated as follows:

[0059]

[0060] In the formula, , z is a training sample from the source domain data set D S , G ex (·) is a feature extractor, F dc (·) is a domain discriminator, and the dimension of the domain score is 1xN.

[0061] 3) As shown in Figure 2 , a domain generalization model network is constructed, and the basic ResNet network in the domain generalization network model is improved; the domain generalization model network uses ResNet network as the backbone network, which is composed of basic ResNet network and hybrid domain attention mechanism (specifically improved CBAM attention mechanism); wherein the backbone network is not limited to ResNet network, and in the specific implementation process, AlexNet network is also verified; the improved ResNet network includes the following steps:

[0062] 3.1) The first layer of the basic ResNet network is added to the domain discriminator before the convolution layer conv1, which is used to obtain the domain score W dc ; then the output layer of the ResNet network is constructed, and on the basis of the semantic category output of the basic ResNet network, the domain category output and the puzzle arrangement category output are increased; finally, the hybrid domain attention mechanism, specifically the improved CBAM attention mechanism, is added before and after the residual module;

[0063] 3.2) Constructing the domain channel attention module in the mixed domain attention mechanism; using the image data in the source domain dataset D S as input, first performing feature extraction on the image data by the first convolutional layer conv1 in the ResNet network to obtain the intermediate feature map X, and then inputting the intermediate feature map X and the domain score W dc into the domain channel attention module; as shown in Figure 3 , in the domain channel attention module, the input intermediate feature map X is compressed in spatial dimension by maximum pooling and average pooling respectively to obtain channel feature vectors and , wherein the dimension of the intermediate feature map X is HxWxC, wherein H represents the height of the intermediate feature map, W represents the width of the intermediate feature map, and C represents the number of channels of the intermediate feature map, and the dimensions of the channel feature vectors and are both 1x1xC; since the domain score and the channel feature vectors are different in dimension, a dimension transformation operation is needed to keep W dc , and the same feature dimension of 1x1xC, and then perform element-wise sum merging on W dc , and to generate the domain channel attention feature map W ca , the specific calculation is as follows:

[0064]

[0065] wherein represents a real number, G ews (·) is an element-wise sum merging operation, G sh1 (·) is a dimension transformation operation, and are the channel feature vectors output after maximum pooling and average pooling, and the dimension of the domain channel attention feature map W ca is 1x1xC;

[0066] Then, multiply the domain channel attention feature map W ca with the intermediate feature map X to complete the intermediate feature map weighting operation to achieve adaptive feature optimization, the specific calculation is as follows:

[0067]

[0068] wherein represents a real number, G wt (·) is a weighting operation, and X' is the intermediate feature map weighted by the domain channel attention, with a dimension of HxWxC;

[0069] 3.3) Constructing the domain space attention module in the mixed domain attention mechanism; as shown in the following formula: Figure 3 The intermediate feature map X' weighted by the domain channel attention and the domain score W dc are input into the domain space attention module; in the domain space attention module, the input intermediate feature map X' weighted by the domain channel attention is compressed in the channel dimension by maximum pooling and average pooling respectively to obtain the spatial feature vectors and wherein the dimension of the intermediate feature map X' weighted by the domain channel attention is H×W×C, and the dimensions of the spatial feature vectors and are both H×W×1; since the domain score and the spatial feature vectors are different in dimension, a dimension transformation operation is needed to keep W dc and and the same feature dimension H×W×1, and then W dc , and are connected based on the channel direction, and then a convolution operation is performed to reduce the feature map of H×W×3 to a domain space attention feature map W sa of 1 channel, and the specific calculation is as shown in the following formula:

[0070]

[0071] wherein represents a real number, G cov (·) is a convolution operation, G sh2 (·) is a dimension transformation operation, and are the spatial feature vectors output after maximum pooling and average pooling, and the dimension of the domain space attention feature map W sa is H×W×1;

[0072] Then, the domain space attention feature map W sa is multiplied by the intermediate feature map X to complete the weighting operation of the intermediate feature map, so as to realize adaptive feature optimization, and the specific calculation is as shown in the following formula:

[0073]

[0074] wherein represents a real number, G wt (·) is a weighting operation, and X" is the intermediate feature map weighted by the domain space attention, with the dimension of H×W×C.

[0075] 4) improved generalization error model LGEM, using local maximum sensitivity instead of local random sensitivity; the sensitivity is a measure of the output difference between the training sample z j and the corresponding unseen sample z u in its Q-neighborhood; Q is a given hyper-parameter, when Q = 1.0; where the range of the perturbation Δz is represented as Δz i is the i-th input feature of the perturbation Δz, n is the number of input features; the j-th training sample z S in the source domain dataset D j is represented as S Q (z j ) in the Q-neighborhood; S Q (z j ) = {z u |z u = z j + Δz; |Δz i | < Q}; the output difference between the training sample z j and the unseen sample z u in the Q-neighborhood is the local random sensitivity E SM , which is calculated as follows:

[0076]

[0077] where M represents the number of perturbations obtained in the Q-neighborhood, m represents the m-th perturbation, represents the output distribution of the training sample;

[0078] Since the local random sensitivity is an average value of the output difference between multiple unseen samples and the training sample, the average value ignores the sensitivity extreme value, therefore, the local maximum sensitivity is used instead of the local random sensitivity; using the adversarial method, an adversarial perturbation r adv is found which maximizes the output difference between the unseen sample and the training sample; the adversarial perturbation refers to the maximum difference between the predicted result of the unseen sample after adding the perturbation and the true result for the adversarial side, while for the model, it refers to the minimum difference between the predicted result of the unseen sample after adding the perturbation and the true result; therefore, the adversarial perturbation r adv is calculated as follows:

[0079] r adv : = argmax D [p (y j | z j ), p (y j | z j + r, θ)], ||r|| ≤ ε

[0080] L LMS=D[p(y j |z j ),p(y j |z j +r adv ,θ)]

[0081] In the formula, the counter-disturbance r adv The solution is to maximize the output difference between the unseen samples after perturbation and the training samples, where r represents the perturbation, θ represents the parameters of the domain generalization network model, and ε represents the canonical constraint of the adversarial direction, controlling the limit of the perturbation selection. Here, ε = 1.0, and D[p(y j |z j ),p(y j |z j [+r,θ)] represents the Kullback-Leibler error between two distributions, where p(y j |z j ) represents the j-th training sample x with semantic category information y. j The true distribution, p(y) j |z j +r,θ) represents the j-th training sample x with semantic category information y. j The output distribution generated by the domain generalization network model with model parameters θ after adding a perturbation r; y j Represents the source domain dataset D S The j-th training sample z j semantic category information; L LMS p(y) represents the local maximum sensitivity. j |z j +r adv (θ) represents the j-th training sample x with semantic category information y. j Adding anti-perturbation r adv The output distribution is then generated by the domain generalization network model with model parameters θ;

[0082] Calculating the adversarial perturbation is an optimization problem, and it is difficult to find a precise value to represent it. Therefore, the following linear estimation method is used to obtain the adversarial perturbation r in the positive gradient direction. adv Approximate solution:

[0083]

[0084]

[0085] In the formula, g represents the positive gradient. Gradient is the differential of a function, representing the rate of change of the function with respect to its input, and is denoted by the symbol 'f'(x) or 'f'(x) or 'f'(x).

[0086] 5) The total loss function of the domain generalization network model comprises three loss functions constructed based on the improved ResNet network output layer and the local maximum sensitivity obtained in step 4), and the total loss function L total The calculation is as follows:

[0087] L total = L clf + aL jig + bL dc + gL LMS

[0088] In the formula, a, b, g are three hyperparameters used to balance all loss functions; L clf represents the data set semantic category output loss function, L dc represents the domain category output loss function, L jig represents the puzzle arrangement category output loss function, L LMS represents the local maximum sensitivity, L clf , L dc and L jig are all based on cross-entropy loss calculation, and L LMS is based on Kullback-Leibler calculation; wherein the loss function refers to the difference between the predicted label information of the domain generalization network model for the input image data and the real label information; after obtaining the loss function, the model parameters are solved by the gradient descent algorithm, and finally the trained and optimized domain generalization network model is obtained, which can complete the image classification task.

[0089] 6) The trained domain generalization network model is directly generalized to the target domain data set D T , and the image classification task is completed on the target domain data set D T and the image classification accuracy is calculated, without the need for training and optimization of the domain generalization network model, and the image classification ability and generalization ability of the current domain generalization network model are evaluated by the image classification accuracy.

[0090] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application shall be equivalent replacement methods, and all are included in the protection scope of the present application.

Claims

1. An image classification method based on domain generalization learning, characterized by, Comprise the following steps: 1) all image data in the image data set is preprocessed into tensor format, one domain in the preprocessed data set is selected as target domain data, and all the remaining domains are selected as source domain data, and the target domain data and the source domain data are recorded with the corresponding true label information of both in the target domain data set D T and the source domain data set D S , wherein T represents the target domain, S represents the source domain, and the true label information includes the field category information, the semantic category information and the puzzle arrangement category information of the current data; 2) using a lightweight convolutional neural network Mobile V2 to build a domain discriminator in a domain generalization network model, according to the source domain data set D S Train the convolutional neural network Mobile V2, and generate the domain score W S according to the domain category information of all data in the source domain data set D dc , wherein W dc is a two-dimensional vector data extracted by the domain discriminator; 3) The basic ResNet network in the domain generalization network model is improved, and the basic ResNet network is improved in three parts, the first part is to add a domain discriminator before the first layer convolutional layer conv1 of the basic ResNet network; The second part is to construct the output layer of the ResNet network, which is the semantic category output, the domain category output and the puzzle arrangement category output respectively; The third part is to add a mixed domain attention mechanism before and after the residual module, which is an improved CBAM attention mechanism; The improvement of the CBAM attention mechanism is to input the domain score into the channel attention and spatial attention modules respectively, wherein the channel attention and spatial attention are two modules of CBAM; After different dimension conversion of the domain score, the domain score is fused with the features extracted from the two modules of CBAM respectively; 4) Improved Generalization Error Model (LGEM) using local maximum sensitivity instead of local random sensitivity; the sensitivity is a measure of the output difference between a training sample z j and its Q-neighborhood of invisible samples z u ; where Q is a hyper-parameter, Q-neighborhood is a hypercube; z j is the jth training sample in the source domain dataset D S ; z u is the perturbed sample generated from z j and z u is within the neighborhood of z j . 5) A new domain generalization network model is constructed and trained based on the domain discriminator and the improved ResNet network, a loss function is constructed based on the output layer of the improved ResNet network, and the local maximum sensitivity obtained in step 4) is introduced into the loss function, and the domain generalization network model is trained and optimized; 6) using the trained domain generalization network model on the target domain dataset D T The image classification accuracy is calculated above, which is used to judge the image classification ability and generalization ability of the current domain generalization network model. 2.The image classification method based on domain generalization learning according to claim 1, characterized in that, In step 1), the preprocessing is to use transforms to pre-process the image data in the target domain dataset D T and the source domain dataset D S , respectively, including image data brightness processing, image data saturation processing, and image data pixel value normalization processing, and convert the image data into a tensor format that can be directly input into a neural network; at the same time, the source domain dataset D S will also select b% of the image data according to the ratio of the hyperparameter b, and divide each image data into k patches, move the patches to reassemble image data of the same size as the original image; shuffling and reassembling will produce k! arrangement methods, in order to reduce the calculation, select P kinds of jigsaw arrangement methods; while data preprocessing, the image data and the corresponding true label information are also recorded in the target domain dataset D T and the source domain dataset D S , wherein the field category information of the current data has N categories, the semantic category information of the current data has Y categories, and the jigsaw arrangement category information of the current data has P categories. 3.The domain generalization learning based image classification method of claim 2, wherein, In step 2), the image data in the source domain data set D S is used as input, feature extraction is performed on the image data using a feature extraction method, the extracted feature map is input into a domain discriminator, is mapped to domain category information, and a domain score W dc is obtained, which is calculated as follows: wherein denotes a real number, z is a training sample from the source domain dataset D S ex (·) is a feature extractor, F dc (·) is a domain discriminator, the dimension of the domain score is 1 x N.​ 4.The image classification method based on domain generalization learning according to claim 3, characterized in that, The step 3) comprises the following steps: 3.1) Add a domain discriminator before the first convolutional layer conv1 of the basic ResNet network to obtain the domain score W dc Then, the output layer of the ResNet network is constructed, which increases the domain category output and the puzzle arrangement category output on the basis of the semantic category output of the basic ResNet network. Finally, the hybrid domain attention mechanism, specifically the improved CBAM attention mechanism, is added before and after the residual module. 3.2) Constructing the domain channel attention module in the mixed domain attention mechanism; using the image data in the source domain dataset D S as input, first extracting features from the image data by the first layer convolutional layer conv1 in the ResNet network to obtain the intermediate feature map X, then inputting the intermediate feature map X and the domain score W dc into the domain channel attention module; in the domain channel attention module, the input intermediate feature map X is compressed in spatial dimension by maximum pooling and average pooling respectively to obtain channel feature vectors and wherein the dimension of the intermediate feature map X is HxWxC, H represents the height of the intermediate feature map, W represents the width of the intermediate feature map, and C represents the channel number of the intermediate feature map; the dimensions of the channel feature vectors and are both 1x1xC; since the domain score and the channel feature vector have different dimensions, a dimension transformation operation is needed to keep W dc and and the same feature dimension of 1x1xC, then performing element-wise summation and merging of W dc , and to produce the domain channel attention feature map W ca , the specific calculation is as follows: In the formula, represents a real number, G ews (·) is an element-wise sum merging operation, G sh1 (·) is a dimension transformation operation, and is the channel feature vector output after maximum value pooling and average value pooling, and the domain channel attention feature map W ca The dimension of is 1×1×C; Then, the domain channel attention feature map W ca is multiplied with the intermediate feature map X to complete the intermediate feature map weighting operation to realize adaptive feature optimization, and the specific calculation is as shown below: In the formula, denotes a real number, G wt (·) is a weighted operation, X' is the intermediate feature map weighted by the domain channel attention, and the dimension is HxWxC; 3.3) Constructing the domain space attention module in the mixed domain attention mechanism; the intermediate feature map X' weighted by the domain channel attention and the domain score W dc At the same time, input the domain space attention module; in the domain space attention module, the input intermediate feature map X' weighted by the domain channel attention is compressed in the channel dimension respectively through the maximum value pooling and the average value pooling to obtain the spatial feature vector and Wherein, the dimension of the intermediate feature map X' weighted by the domain channel attention is HxWxC, the dimension of the spatial feature vector and Both are HxWx1; since the domain score and the spatial feature vector are different in dimension, a dimension transformation operation is needed to keep W dc and and The same feature dimension HxWx1, and then W dc , and Connect based on the channel direction, and then perform a convolution operation to reduce the feature map of HxWx3 to a domain space attention feature map W sa of 1 channel. The specific calculation is as follows: In the formula, represents a real number, G cov (·) is a convolution operation, G sh2 (·) is a dimension transformation operation, and is the spatial feature vector output after maximum value pooling and average value pooling, and the field space attention feature map W sa The dimension of is HxWx1; Then, the field space attention feature map W sa is multiplied with the intermediate feature map X to complete the intermediate feature map weighting operation to realize adaptive feature optimization, and the specific calculation is as shown below: wherein denotes a real number, G wt (·) is a weighted operation, X" is the intermediate feature map of the domain space attention weighting, with dimension HxWxC. 5.The image classification method based on domain generalization learning according to claim 4, characterized in that, In step 4), the Q is a given hyper-parameter, where the range of values of the perturbation Δz is denoted as Δz i is the i-th input feature of the perturbation Δz, n is the number of input features; the source domain dataset D S is the j-th training sample z j in the Q-neighborhood range, the corresponding set of unseen sample data S Q (z j ) is denoted as S Q (z j ) = {z u |z u = z j + Δz; |Δz i | < Q}; the training sample z j and the unseen sample z u The output difference in the Q-neighborhood range is the local random sensitivity E SM , which is calculated as follows: In the formula, M represents the number of disturbances obtained in the Q neighborhood, m represents the mth disturbance, represents the output distribution of the training sample; Since the local random sensitivity is to measure the average of the output difference between multiple invisible samples and training samples, the average will ignore the sensitivity extreme case, so the local maximum sensitivity is used instead of the local random sensitivity; using the adversarial method, find an adversarial perturbation r that can maximize the output difference between invisible samples and training samples adv ; the adversarial perturbation refers to the maximum difference between the predicted result of the invisible sample after adding the perturbation and the true result for one party, and the minimum difference between the predicted result of the invisible sample after adding the perturbation and the true result for the model; therefore, the adversarial perturbation r adv is calculated as follows: r adv :=argmaxD[p(y j |z j ),p(y j |z j +r,θ)],||r||≤ε L LMS = D[p(y j |z j ), p(y j |z j + r adv , θ)] where solving for the adversarial perturbation r adv is by maximizing the output difference between the perturbed unseen sample and the training sample, where r represents the perturbation, θ represents the parameters of the domain generalization network model, ε represents the norm constraint of the adversarial direction, controls the limit of the perturbation selection, D[p(y j |z j ), p(y j |z j +r, θ)] represents the Kullback-Leibler error between the two distributions, where p(y j |z j ) represents the true distribution of the jth training sample z j with semantic category information y, and p(y j |z j +r, θ) represents the output distribution of the jth training sample z j after adding the perturbation r by the domain generalization network model with model parameters θ; y j represents the semantic category information of the jth training sample z S in the source domain data set D j ; L LMS represents the local maximum sensitivity, and p(y j |z j +r adv , θ) represents the output distribution of the jth training sample z j after adding the adversarial perturbation r adv by the domain generalization network model with model parameters θ; For the calculation of the adversarial perturbation is an optimization problem, it is difficult to find such an accurate value of the adversarial perturbation, so use the following linear estimation method in the positive gradient direction to get the approximate solution of the adversarial perturbation r adv : where g represents a positive gradient, represents calculating the gradient between the output difference between the invisible sample after adding the disturbance and the training sample, wherein the gradient refers to the differential of the function, representing the slope of the tangent line of the function at a given point, and the positive gradient represents the direction in which the function rises fastest at the given point.

6. The image classification method based on domain generalization learning according to claim 5, characterized in that, In step 5), the total loss function includes three loss functions constructed based on the improved ResNet network output layer and the local maximum sensitivity obtained in step 4), and the total loss function L total is calculated as follows: L total = L clf + αL jig + βL dc + γL LMS where α, β, γ are three hyper-parameters to balance all loss functions; L clf represents the data set semantic category output loss function, L dc represents the field category output loss function, L jig represents the jigsaw arrangement category output loss function, L LMS represents the local maximum sensitivity, L clf , L dc and L jig are all based on cross-entropy loss calculation, and L LMS is based on Kullback-Leibler calculation; wherein the loss function refers to calculating the gap between the predicted label information of the field generalization network model for the input image data and the real label information; the real label information is saved in the data set when preprocessed in step 1); after obtaining the loss function, the model parameters are solved through the gradient descent algorithm, and finally the trained and optimized field generalization network model is obtained, which can complete the image classification task.

7. The image classification method based on domain generalization learning according to claim 6, characterized in that: In step 6), the trained domain generalization network model is directly generalized to the target domain dataset D T After that, the image classification task is completed on the target domain dataset D T The image classification accuracy is calculated, and the training and optimization of the domain generalization network model are not required. The image classification ability and generalization ability of the current domain generalization network model are evaluated by the image classification accuracy.

Images