Domain generalization method, device and computer equipment based on domain-invariant feature extraction

Abstract

The present application relates to a domain generalization method, device and computer equipment based on domain invariant feature extraction. The method includes: establishing a domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistent loss, including: a backbone network, a domain discriminator and a classifier based on a cross-spatial multi-scale information aggregation attention mechanism; training the domain generalization model according to the source domain data and the total loss function, randomly inserting a domain discriminator in the middle layer of the backbone network for each training batch, and learning domain invariant features based on the cross-spatial multi-scale information aggregation attention mechanism to identify the contribution of feature channels to the domain to improve the recognition ability of domain-specific features; using the trained domain generalization model to classify and predict the target domain data to obtain classification results. This method can learn features that are not related to a specific domain, effectively remove domain-specific features in the domain generalization problem, and allow predictions based on cross-domain invariant features.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence transfer learning technology, and in particular to a domain generalization method, apparatus and computer equipment based on domain-invariant feature extraction. Background Art

[0002] Convolutional neural networks have shown powerful performance in the field of machine learning, but most existing machine learning algorithms are based on the distribution invariance assumption, that is, the training and test data are independent and identically distributed. However, in the real world, trained models often need to face test data with different distributions, that is, the training and test data are in different fields, such as Figure 1 As shown in the figure, this assumption often does not hold. This type of problem is called domain drift, which hinders the further application of the model in various tasks. To solve this problem, people have conducted extensive research on domain adaptation. However, domain adaptation requires that the model can see part of the target domain data during training, which is not possible in some real-world scenarios. Therefore, domain generalization was proposed, aiming to enable the model to transfer the knowledge learned in the source domain to the unseen target domain under the premise of access to a single or multiple labeled source domains.

[0003] A typical method of existing domain generalization is to learn invariant features across domains. The core idea of ​​this type of method is based on the assumption of domain-specific features, that is, all domains have shared features, and each domain has its own unique domain-specific features. However, previous work has focused more on directly extracting domain-invariant features, while people usually choose to ignore domain-specific features that may affect the final prediction performance, hoping to eliminate domain-specific features through the final domain-invariant features. But the fact is that this method does not explicitly guide the model to remove domain-specific features in the network, which may cause the model to learn too many domain-related features. Studies have shown that in addition to domain-invariant features, some domain-specific features also provide useful information for the target domain. However, since the target domain is invisible, this cannot always be guaranteed. For example, some features describe texture features in an image, while others describe contour features. So when the model is trained in a domain where texture features are more obvious, its performance will decline in a domain dominated by line contours. Summary of the invention

[0004] Based on this, it is necessary to provide a domain generalization method, device and computer equipment based on domain-invariant feature extraction to address the above technical problems.

[0005] A domain generalization method based on domain-invariant feature extraction, the method comprising:

[0006] A domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistency loss is established; the domain generalization model includes: a backbone network, a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism, and a classifier; the domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism is used to aggregate cross-spatial information using multi-scale convolution operations, so that the network pays more attention to domain-invariant feature channels and suppresses domain-specific feature channels. On the basis of feature selection, the contribution of feature channels to domain recognition is scored.

[0007] Construct a total loss function; the total loss function includes domain prediction loss, reasoning consistency loss, and classification prediction loss; reasoning consistency loss is used to ensure that the prototype matrix of samples from the same category is consistent with their corresponding average value.

[0008] The domain generalization model is trained according to the source domain data and the total loss function to obtain a trained domain generalization model; wherein, in each training batch, a domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism is randomly inserted into the middle layer of the backbone network to identify the feature channels that contribute the most to a specific domain and delete them.

[0009] The trained domain generalization model is used to classify and predict the target domain data to obtain the classification results.

[0010] A domain generalization device based on domain-invariant feature extraction, the device comprising:

[0011] The domain generalization model construction module is used to establish a domain generalization model based on domain-invariant feature extraction; the domain generalization model includes: a backbone network, a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism, and a classifier; the domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism is used to aggregate cross-spatial information using multi-scale convolution operations, so that the network pays more attention to domain-invariant feature channels and suppresses domain-specific feature channels. On the basis of feature selection, the contribution of feature channels to domain recognition is scored.

[0012] The total loss function building module is used to construct the total loss function; the total loss function includes domain prediction loss, reasoning consistency loss and classification prediction loss; reasoning consistency loss is used to ensure that the prototype matrix of samples from the same category is consistent with their corresponding average value.

[0013] The domain generalization model training module is used to train the domain generalization model according to the source domain data and the total loss function to obtain a trained domain generalization model; wherein, in each training batch, a domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism is randomly inserted into the middle layer of the backbone network to identify the feature channels that contribute the most to a specific domain and delete them.

[0014] The domain generalization module is used to use the trained domain generalization model to classify and predict the target domain data to obtain the classification results.

[0015] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0016] The above-mentioned domain generalization method, device and computer equipment based on domain invariant feature extraction, the method includes: establishing a domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistent loss; the domain generalization model includes: a backbone network, a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism and a classifier; the domain discriminator improves the recognition ability of domain-specific features by learning domain-invariant features based on the cross-spatial multi-scale information aggregation attention mechanism to identify the contribution of feature channels to the domain; the domain generalization model is trained according to the source domain data and the total loss function to obtain a trained domain generalization model; wherein, in each training batch, a domain discriminator is randomly inserted into the middle layer of the backbone network to identify the feature channel with the largest contribution to the specific domain and delete it; the trained domain generalization model is used to classify and predict the target domain data to obtain a classification result. This method can learn features that are not related to a specific domain, effectively remove domain-specific features in the domain generalization problem, and allow it to make predictions based on cross-domain invariant features. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a domain generalization method based on domain-invariant feature extraction in one embodiment;

[0018] Figure 2 is an overall architecture diagram of a domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistent loss in another embodiment;

[0019] Figure 3 A schematic diagram of the structure of an attention mechanism for cross-spatial multi-scale information aggregation in another embodiment;

[0020] Figure 4 is a structural block diagram of a domain generalization device based on domain-invariant feature extraction in one embodiment;

[0021] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] Assume that Different source domains , for each field, , the field label is ,in Indicates In the field The data label pairs of samples, yes All source and target domains share the same class label space, and the goal of domain generalization is to train a model using only source domain data. , making it possible to Good performance.

[0024] In one embodiment, Figure 1 As shown, a domain generalization method based on domain invariant feature extraction is provided, and the method includes the following steps:

[0025] Step 100: Establish a domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistency loss; the domain generalization model includes: a backbone network, a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism, and a classifier; the domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism is used to aggregate cross-spatial information using multi-scale convolution operations, so that the network pays more attention to domain-invariant feature channels and suppresses domain-specific feature channels, and on the basis of feature selection, the contribution of feature channels to domain recognition is scored.

[0026] Specifically, the overall architecture of the domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistency loss is as follows: Figure 2 As shown in the figure. The Cross-Space Multi-Scale Attention Domain Discriminator (CMADD) based on the cross-space multi-scale information aggregation attention mechanism captures the short-range and long-range dependencies in the image through the cross-space information by designing a multi-scale parallel network, so that the network pays more attention to the domain-invariant feature channels and suppresses the domain-specific feature channels, which promotes the active removal of domain-specific features during model training. On the basis of feature selection, the domain discriminator based on the cross-space multi-scale information aggregation attention mechanism uses the feature channel to score the contribution of the domain recognition. The higher the score, the closer the connection between the feature channel and the domain, and the greater the probability of being removed.

[0027] The domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism learns domain-invariant features by identifying the contribution of feature channels to the domain.

[0028] The backbone network is used to extract features, and the backbone network can be but is not limited to ResNet-18 and ResNet-50.

[0029] The classifier is used to classify the extracted features, and the classifier is a fully connected layer.

[0030] Step 102: construct a total loss function; the total loss function includes domain prediction loss, reasoning consistency loss, and classification prediction loss; the reasoning consistency loss is used to ensure that the prototype matrix of samples from the same category is consistent with their corresponding average values.

[0031] Specifically, in order to ensure the extraction of more robust features and further improve the stability of feature channels to domain drift, a momentum-based inference consistency loss (MICL) is designed to keep each sample as close to the category center as possible during model training. With this, the model is able to learn features that are not related to a specific domain, allowing it to make predictions based on features that are invariant across domains.

[0032] Step 104: Train the domain generalization model according to the source domain data and the total loss function to obtain a trained domain generalization model; wherein, in each training batch, a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism is randomly inserted into the middle layer of the backbone network to identify and delete the feature channel that contributes most to a specific domain.

[0033] Specifically, in order to enable the domain generalization model to accurately remove domain-specific features during training, this method introduces a domain discriminator with an attention mechanism in multiple intermediate layers of the feature extraction network to locate domain-specific features. Input and its tags , this method first extracts Features generated by the intermediate layer ,in is the number of channels, and Represents the height and width dimensions respectively. Followed by the domain discriminator , to predict the domain label and calculate the discrimination loss. At the same time, this method uses a gradient reversal layer (GRL) before the domain discriminator to perform adversarial training, so that the backbone network can extract more general features.

[0034] The difference between the domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism (CMADD) and the traditional domain generalization method is that this method continuously filters out feature channels that contribute greatly to specific domain recognition to prevent the model from retaining too many domain-related features. By discarding domain-specific features during training, the method of this application reduces the instability of feature channels caused by domain shift and promotes the learning of domain-invariant features.

[0035] Step 106: Use the trained domain generalization model to perform classification prediction on the target domain data to obtain classification results.

[0036] The above-mentioned domain generalization method based on domain invariant feature extraction includes: establishing a domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistent loss; the domain generalization model includes: a backbone network, a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism, and a classifier; the domain discriminator improves the recognition ability of domain-specific features by learning domain-invariant features based on the cross-spatial multi-scale information aggregation attention mechanism to identify the contribution of feature channels to the domain; the domain generalization model is trained according to the source domain data and the total loss function to obtain a trained domain generalization model; wherein, in each training batch, a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism is randomly inserted into the middle layer of the backbone network to identify the feature channel with the largest contribution to the specific domain and delete it; the trained domain generalization model is used to classify and predict the target domain data to obtain the classification result. This method can learn features that are not related to a specific domain, effectively remove the domain-specific features in the domain generalization problem, and allow it to make predictions based on cross-domain invariant features.

[0037] In one embodiment, the backbone network in step 100 is a residual network.

[0038] In one embodiment, the intermediate layer is a residual block of a residual network.

[0039] In one embodiment, step 104 includes: in each round of training, randomly selecting a middle layer of the backbone network to introduce a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism to locate domain-specific features; inputting the input from the source domain data and the corresponding label into the backbone network to extract the domain-specific features generated by the first layer; lThe first intermediate feature is obtained by extracting the features generated by the intermediate layers; the first intermediate feature is input into the domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism, the contribution of the feature channel to the domain identification is scored, and the channel with the largest contribution to the domain identification is selected and deleted; according to the score of each feature channel, a binary feature mask is generated for the intermediate layer of the domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism; the binary feature mask is applied to the first intermediate feature to obtain a universal feature; the universal feature is extracted through the back-end part of the backbone network, and then the extracted features are used to generate the reasoning consistency loss, and the extracted features are input into the classifier to obtain the classification prediction result; according to the classification prediction result and the total loss function, the domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistency loss is reversely trained to obtain a trained domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistency loss.

[0040] Specifically, multiple intermediate layers means that the backbone network has four intermediate layers. Each time training, an intermediate layer is randomly selected and placed into a domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism. The feature mask is generated in the intermediate layer where the domain discriminator is placed.

[0041] In order to determine which features contain domain-specific information, the prediction results of the domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism in the intermediate layer are used as a measure of the importance of feature channels. Specifically, it is assumed that the channel that contributes most to domain prediction may contain more domain-specific information. First, the domain label prediction is performed:

[0042] (1)

[0043] in, For input, Characterized by is the domain classifier weight, It is the global average pooling. At the same time, the domain prediction loss is calculated:

[0044] (2)

[0045] in, is the cross entropy loss. Then the weighted activation of the correct domain prediction is calculated to quantify the Correlation between channels and domain-specific information:

[0046] (3)

[0047] in, For the domain discriminator corresponding to the domain The higher the weight value, the greater the contribution of the corresponding feature channel to domain prediction.

[0048] At the same time, the features extracted by convolutional neural networks represent different information at different layers of the network, and deep networks extract more semantic features. Recently, some research results have shown that features extracted from shallow layers show more obvious domain-specific characteristics, so shallow features are processed to alleviate domain gaps. However, previous methods are usually designed for specific shallow or deep networks, and such methods may not achieve the best domain generalization effect. Therefore, starting from considering the features of all network layers, it is recognized that the features extracted from each layer may contain domain-specific features. Specifically, all the intermediate layers of the backbone network (if the backbone network is a ResNet network, the intermediate layers are the residual blocks of ResNet) are selected as candidate sets. In each iteration, a residual block is randomly selected from the set and its feature channels are processed. After identifying the corresponding feature channels, in order to explicitly reduce the domain-specific information in the feature map, the channels that contribute most to domain identification will be selected and deleted during the training process. Specifically, the number of discarded feature channels is first calculated:

[0049] (4)

[0050] in, is the feature channel discarding rate. Subsequently, a binary feature mask is generated based on the score of each feature channel. , depending on whether the mask value is 1, it determines whether the corresponding channel is discarded. Specific feature mask The probability of being set to 1 is given by decision, with relatively high The feature channels are more likely to be discarded. In order to reduce the computational cost, the weighted random selection (WRS) algorithm is used to generate the binary mask Specifically, for the score No. channels, first generate a random number And calculate the key value Then, this method selects the one with the largest key value values ​​and mask Correspondence Set to 0:

[0051] (5)

[0052] in, Indicates the one with the largest key value Finally, the feature mask is applied to the original feature to obtain:

[0053] (6)

[0054] in, is the feature after mask processing.

[0055] In one embodiment, a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism includes an attention mechanism for cross-spatial multi-scale information aggregation and a domain classifier; inputting a first intermediate feature into a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism, selecting and deleting a channel that contributes most to domain identification, and scoring the contribution of feature channels to domain identification, including: inputting the first intermediate feature into an attention mechanism for cross-spatial multi-scale information aggregation, and performing convolution, X-direction maximum pooling, Y-direction maximum pooling, X-direction average pooling, and Y-direction average pooling, respectively, to obtain convolution features, X-direction maximum pooling results, Y-direction maximum pooling results, X-direction average pooling results, and Y-direction average pooling results; concatenating the X-direction maximum pooling results and the Y-direction maximum pooling results, and then performing convolution processing, and then activating with a Sigmoid function to obtain maximum pooling features; The average pooling result and the Y-direction average pooling result are concatenated and processed by convolution, and then activated by Sigmoid function to obtain the average pooling feature; the maximum pooling feature, the average pooling feature and the first intermediate feature are weighted summed, and then the obtained result is normalized to obtain the weighted normalized feature; the convolution feature is subjected to average pooling and Softmax activation, and then matrix multiplication operation is performed with the weighted normalized feature to obtain the first fusion feature; the weighted normalized feature is subjected to average pooling and Softmax activation, and then matrix multiplication operation is performed with the convolution feature to obtain the second fusion feature; the first fusion feature and the second fusion feature are added and fused, and then weighted with the first intermediate feature to obtain the domain-specific feature; the domain-specific feature is input into the domain classifier to obtain the domain label prediction; the first intermediate feature is subjected to global average pooling, and then element-by-element multiplication with the domain classifier weight is performed to obtain the score of the contribution of the feature channel to domain recognition.

[0056] Specifically, the domain classifier includes a fully connected layer.

[0057] The attention mechanism structure for cross-spatial multi-scale information aggregation is as follows: Figure 3 shown.

[0058] In one embodiment, generating a binary feature mask according to the score of each feature channel includes:

[0059] For the score No. The channels generate random numbers and calculate the key values; the key values ​​are:

[0060] (7)

[0061] in, The score is No. The key value of each channel, Indicates The score of each channel, Indicates the generated A random number between 0 and 1 for each channel;

[0062] Select the one with the largest key value key values ​​and will be the corresponding elements of the binary mask Set to 0; the corresponding element of the binary mask for:

[0063] (8)

[0064] in, Indicates the one with the largest key value value.

[0065] In one embodiment, step 102 includes: constructing a total loss function as:

[0066] (9)

[0067] in, represents the total loss, represents the inference consistency loss, represents the classification prediction loss, represents the domain prediction loss, , denote the predicted label and domain label respectively, , Representative The class labels predicted by the classifier in the domain, Indicates that it comes from the source domain The input label.

[0068] In one embodiment, the inference consistency loss is:

[0069] (10)

[0070] in, is the number of samples in a batch; Indicates The prototype matrix of samples; Indicates that the corresponding k The mean matrix of the prototype matrices of the class.

[0071] Update online with momentum at each iteration step ; The update formula is:

[0072] (11)

[0073] (12)

[0074] in, , Respectively represent the current moment and the previous moment corresponding to the k The average matrix of the prototype matrix of the class, represents the prototype matrix, represents the classifier weight, The output of the backbone network The characteristics of a channel.

[0075] Specifically, in order to obtain a more generalizable model, both features and classification weights need to be considered. If you only focus on features without fully considering the classifier weights corresponding to the features, the final classification results will be inaccurate. For example, a more important feature may correspond to a smaller classification weight. This biased estimate of feature importance is not desirable.

[0076] Compared with features, probability takes into account classification weights to a certain extent, alleviating the problem of focusing only on features. However, probability only provides a rough decision value and lacks a detailed explanation of the decision process. This limitation makes it challenging to understand the essential reasons for classification decisions. Therefore, relying solely on probability may not be able to control the decision process, resulting in an inability to effectively obtain reliable results.

[0077] In current deep models, the final output from the decision process involves passing through a feature extractor Get Features , and then through Applying the Classifier To calculate the final probability, expressed as Assume that categories, the output probability can be expressed as ,in yes The weight of .

[0078] To more deeply investigate the impact of features and classifier weights on the final classification, we can express the probability value generated by the sum of element-wise products between features and classifier weights as as follows:

[0079] (13)

[0080] In the above equation, k The probability of the class is all The accumulation of The element-by-element contribution of . Intuitively, Provides a piece of evidence to support a specific decision. Therefore, all As a matrix To represent the basic principle of classifying samples, it is called the prototype matrix. The formal definition of is shown in formula (12).

[0081] The prototype matrix overcomes the limitations of feature-based and probability-based analysis, represents the decision process from a more fine-grained perspective, and also takes into account classification weights. It provides a new tool for domain generalization or other related machine learning problems. The main goal of this application is to design a model that can maintain reliable output under different environments. The prototype matrix provides characteristics of how decisions are made, and new regularization can be introduced based on the concept of basic principles to control the decision process to approach the optimal design. Various methods can be used to design such regularization terms, but this method adopts a simple approach, which is to ensure that the prototype matrices of samples from the same category are consistent with their corresponding mean. This approach means that the decision to classify an object should be based on the same reasoning regardless of changes in samples or domains, because the causal factors of the decision are usually stable patterns that persist across samples and domains. Formally, an inference consistency loss term is imposed to force the prototype matrix of a sample to be close to its corresponding mean, expressed as , after the regularization process, the inference consistency loss is obtained as shown in formula (10).

[0082] However, calculation Directly involves averaging all samples, which takes a lot of computation time. Therefore, it is updated online in a momentum manner at each iteration step. , the update formula is shown in formula (11).

[0083] Then the total loss function can be expressed as the loss function shown in formula (9).

[0084] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0085] In a verification example, the method of the present application is evaluated based on three benchmark datasets: PACS, OfficeHome and VLCS: (1) PACS consists of images from 4 different fields: Art, Cartoon, Photo and Sketch, including 7 different categories and 9991 images. This example uses the official division provided by Reference 1 (Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. InCVPR, 2017) for training and verification. (2) OfficeHome consists of 4 different fields: Art, Clipart, Product and RealWorld, including 65 categories and about 15,500 images. Following the setting of reference 2 (Long, S., Zhou, Q., Ying, C., Ma, L. and Luo, Y., 2024. Rethinking domain generalization: Discriminability and generalizability. IEEE Transactions on Circuits and Systems for Video Technology), each source domain is randomly divided into 90% for training and 10% for validation. (3) VLCS (VLCS is a dataset proposed in reference 3 (Shi, Z., Ming, Y., Fan, Y., Sala, F., & Liang, Y. (2024, January). Domain generalization via nuclearnorm regularization. In Conference on Parsimony and Learning (pp. 179-201). PMLR)) consists of 10,729 images in 5 categories, covering 4 different domains: VOC 2007 (Pascal), LabelMe, Caltech and Sun.

[0086] (1) Experimental details

[0087] This example uses ImageNet pre-trained ResNet-18 and ResNet-50 as the backbone network for PACS and OfficeHome datasets. For VLCS, ResNet-18 is used as the backbone network of the method. The learning rate is set to 0.002 and the batch size is 64.

[0088] For ADD, the discard rate is set to 0.33, and the weight of the gradient reversal layer before the domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism is set to 0.25. For RCG, the momentum value m is set to 0.1, and the loss weight Set to 0.1. For all datasets, the leave-one-out strategy is used for evaluation, that is, one domain is used as the target domain in each dataset in turn, and the other domains are used as the source domain. The model of this method is trained using the training set of the source domain, and the hyperparameters are selected based on the validation set. The best model is selected on the validation set of all source domains and the Top-1 accuracy is returned. All experiments were completed using a single RTX4070 graphics card in the environment of Windows11.0, Pytorch2.2.0, and CUDA11.8.

[0089] (2) Evaluation on PACS

[0090] This example compares previous domain generalization works on the PACS dataset using ResNet-18 and ResNet-50 as the backbone networks, including methods based on domain-invariant feature extraction (I2 drop, CDG, DMG, MatchDG, Fisher) and other advanced methods, including methods based on data enhancement (i.e., MixStyle, EFDMix, StyleNeophile, ALOFT), methods based on feature enhancement (StableNet), etc.

[0091] As shown in Table 1, the framework of the model of this method achieves the highest average accuracy among all the compared methods on the two backbone networks. It is worth noting that the method of this application maintains the highest accuracy in the Cartoon and Sketch domains in ResNet-18. This is because the method of this application helps the domain discriminator to distinguish and delete domain-specific features in the network layer through the design of the attention mechanism. Although the method of this application only maintains the highest accuracy in the Sketch domain in ResNet-50, it achieves the highest average accuracy, which shows that the reasoning consistency generalization loss function added in the method of this application provides a robust decision-making basis for the model, so that the model shows good generalization ability in the face of a variety of different domains. Among the contrasting methods, I2-Drop penalizes the local texture of the model learning surface, but ignores the possibility of other forms of domain-specific features. DMG assumes that some domain-specific features provide useful information for the target domain, which may be because the target domain is not visible, so this assumption is not always guaranteed. The experimental results of RSC and CDG confirm its good performance, but the characteristics of only being applicable to single-layer networks hinder its further reduction of domain-specific information. The proposed method achieved the highest average accuracy among all the compared methods, which fully demonstrated its effectiveness in improving the generalization of the model.

[0092] Table 1. Accuracy of the framework of the model of this method on two backbone networks and all the comparison methods

[0093]

[0094] (3) Evaluation on OfficeHome

[0095] The prominent feature of the OfficeHome dataset is that it has a large number of categories and samples. As shown in Table 2, the method of this application was evaluated on the ResNet-18 and ResNet-50 backbone networks, and the effectiveness of the method of this application was proved by comparison with other SOTA methods. In the ResNet-18 backbone network, the method of this application maintained the highest accuracy in the two fields of Clipart and RealWorld; at the same time, in the ResNet-50 backbone network, the method of this application surpassed the latest methods SWAD-NU and DMDA, achieved an accuracy of 71.81%, and maintained the highest accuracy in the two fields of Clipart and RealWorld, further illustrating the role of the method of this application.

[0096] Table 2 Evaluation results of this method on ResNet-18 and ResNet-50 backbone networks

[0097]

[0098] (4) Evaluation on VLCS

[0099] As shown in Table 3, this method is evaluated on the VLCS dataset on the ResNet-18 and ResNet-50 backbone networks. From the experimental results, it can be seen that the proposed method surpasses the existing mainstream domain generalization methods, slightly surpasses DomainDrop in the ResNet-18 backbone network, and surpasses the latest method SWAD-NU by 0.37% in ResNet-50, which fully demonstrates the effectiveness of this method.

[0100] Table 3 Evaluation results of this method on the VLCS dataset on ResNet-18 and ResNet-50 backbone networks

[0101]

[0102] (5) Ablation experiment

[0103] This paper verifies the effectiveness of the two main modules of CMADD and MICL in the method of this paper by using the ResNet-18 backbone network to conduct ablation experiments on the PACS dataset. As shown in Table 4, the same parameters were used for training in this experiment. Compared with the baseline method ResNet-18, the final model of this embodiment improved the accuracy by 4.95%. When the two modules were added separately, the accuracy was improved by 3.67% and 3.97% respectively compared with the original model, which verified the effectiveness of these two modules in improving performance. The performance was best after both modules were added, which verified that the two modules complemented and promoted each other, and were indispensable for realizing the superior generalization ability of the model.

[0104] Table 4 Ablation experiment results

[0105]

[0106] In one embodiment, Figure 4 As shown, a domain generalization device based on domain invariant feature extraction is provided, including: a domain generalization model construction module, a total loss function construction module, a domain generalization model training module and a domain generalization module, wherein:

[0107] The domain generalization model building module is used to establish a domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistency loss; the domain generalization model includes: a backbone network, a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism, and a classifier; the domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism is used to aggregate cross-spatial information using multi-scale convolution operations, so that the network pays more attention to domain-invariant feature channels and suppresses domain-specific feature channels. On the basis of feature selection, the contribution of feature channels to domain recognition is scored.

[0108] The total loss function building module is used to construct the total loss function; the total loss function includes domain prediction loss, reasoning consistency loss and classification prediction loss; reasoning consistency loss is used to ensure that the prototype matrix of samples from the same category is consistent with their corresponding average value.

[0109] The domain generalization model training module is used to train the domain generalization model according to the source domain data and the total loss function to obtain a trained domain generalization model; wherein, in each training batch, a domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism is randomly inserted into the middle layer of the backbone network to identify the feature channels that contribute the most to a specific domain and delete them.

[0110] The domain generalization module is used to use the trained domain generalization model to classify and predict the target domain data to obtain the classification results.

[0111] In one of the embodiments, the backbone network in the domain generalization model construction module is a residual network.

[0112] In one of the embodiments, the middle layer in the backbone network in the domain generalization model training module is a residual block of the residual network.

[0113] In one of the embodiments, the domain generalization model training module is further used to introduce a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism in a randomly selected intermediate layer of the backbone network in each round of training to locate domain-specific features; input the input from the source domain data and the corresponding label into the backbone network to extract the domain-specific features from the first intermediate layer; lThe first intermediate feature is obtained by extracting the features generated by the intermediate layers; the first intermediate feature is input into the domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism, the contribution of the feature channel to the domain identification is scored, and the channel with the largest contribution to the domain identification is selected and deleted; according to the score of each feature channel, a binary feature mask is generated for the intermediate layer of the domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism; the binary feature mask is applied to the first intermediate feature to obtain a universal feature; the universal feature is extracted through the back-end part of the backbone network, and then the extracted features are used to generate the reasoning consistency loss, and the extracted features are input into the classifier to obtain the classification prediction result; according to the classification prediction result and the total loss function, the domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistency loss is reversely trained to obtain a trained domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistency loss.

[0114] In one embodiment, a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism includes an attention mechanism for cross-spatial multi-scale information aggregation and a domain classifier; a domain generalization model training module is further used to input the first intermediate feature into the attention mechanism for cross-spatial multi-scale information aggregation, and respectively undergo convolution, X-direction maximum pooling, Y-direction maximum pooling, X-direction average pooling, and Y-direction average pooling to obtain convolution features, X-direction maximum pooling results, Y-direction maximum pooling results, X-direction average pooling results, and Y-direction average pooling results; the X-direction maximum pooling results and the Y-direction maximum pooling results are concatenated and then subjected to convolution processing, and then activated using a Sigmoid function to obtain maximum pooling features; the X-direction average pooling results and the Y-direction average pooling results are concatenated and then subjected to convolution processing, Then, the Sigmoid function is used for activation to obtain the average pooling feature; the maximum pooling feature, the average pooling feature and the first intermediate feature are weighted and summed, and then the obtained result is normalized to obtain the weighted normalized feature; the convolution feature is subjected to average pooling and Softmax activation, and then matrix multiplication operation is performed with the weighted normalized feature to obtain the first fusion feature; the weighted normalized feature is subjected to average pooling and Softmax activation, and then matrix multiplication operation is performed with the convolution feature to obtain the second fusion feature; the first fusion feature and the second fusion feature are added and fused, and then weighted with the first intermediate feature to obtain the domain-specific feature; the domain-specific feature is input into the domain classifier to obtain the domain label prediction; the first intermediate feature is subjected to global average pooling, and then element-by-element multiplication is performed with the domain classifier weight to obtain the score of the feature channel's contribution to domain recognition.

[0115] In one embodiment, the domain generalization model training module is also used for the score No. Generate random numbers for each channel and calculate the key value; the key value is shown in formula (7); Select the channel with the largest key value key values ​​and will be the corresponding elements of the binary mask Set to 0; the corresponding element of the binary mask As shown in formula (8).

[0116] In one embodiment, the total loss function construction module is also used to construct the total loss function as shown in formula (9).

[0117] In one embodiment, the inference consistency loss in the total loss function building module is as shown in formula (10); it is updated online in a momentum manner in each iteration step. ; The update formula is shown in formula (11).

[0118] For the specific definition of the domain generalization device based on domain invariant feature extraction, please refer to the definition of the domain generalization method based on domain invariant feature extraction above, which will not be repeated here. Each module in the above-mentioned domain generalization device based on domain invariant feature extraction can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0119] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a domain generalization method based on domain-invariant feature extraction is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0120] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in any of the above method embodiments when executing the computer program.

[0122] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A domain generalization method based on domain-invariant feature extraction, characterized in that: The method comprises: A domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistency loss is established; the domain generalization model includes: a backbone network, a domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism, and a classifier; the domain discriminator based on a cross-spatial multi-scale information aggregation attention mechanism is used to aggregate cross-spatial information using multi-scale convolution operations, so that the network pays more attention to domain-invariant feature channels and suppresses domain-specific feature channels, and on the basis of feature selection, the contribution of feature channels to domain recognition is scored; Constructing a total loss function; the total loss function includes domain prediction loss, reasoning consistency loss and classification prediction loss; the reasoning consistency loss is used to ensure that the prototype matrix of samples from the same category is consistent with their corresponding average values; The domain generalization model is trained according to the source domain data and the total loss function to obtain a trained domain generalization model; wherein, in each training batch, the domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism is randomly inserted into the middle layer of the backbone network to identify the feature channel with the largest contribution to a specific domain and delete it; The trained domain generalization model is used to classify and predict the target domain data to obtain the classification results.

2. The method according to claim 1, characterized in that The backbone network is a residual network.

3. The method according to claim 2, characterized in that The intermediate layer is a residual block of the residual network.

4. The method according to claim 1, characterized in that The domain generalization model is trained according to the source domain data and the total loss function to obtain a trained domain generalization model, including: In each round of training, a domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism is introduced into a randomly selected intermediate layer of the backbone network to locate domain-specific features; The input from the source domain data and the corresponding labels are input into the backbone network to extract l The features generated by the intermediate layers are used to obtain the first intermediate features; Inputting the first intermediate feature into the domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism, scoring the contribution of feature channels to domain identification, and selecting and deleting the channel that contributes the most to domain identification; Generate a binary feature mask based on the score of each feature channel for the intermediate layer of the domain discriminator introduced based on the cross-spatial multi-scale information aggregation attention mechanism; Applying the binary feature mask to the first intermediate feature to obtain a common feature; The common features are extracted through the back-end part of the backbone network, and the extracted features are used to generate reasoning consistency loss, and the extracted features are input into the classifier to obtain classification prediction results; According to the classification prediction result and the total loss function, the domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistent loss is reversely trained to obtain a trained domain generalization model based on cross-spatial multi-scale information aggregation and reasoning consistent loss.

5. The method according to claim 4, characterized in that The domain discriminator based on the attention mechanism of cross-spatial multi-scale information aggregation includes an attention mechanism of cross-spatial multi-scale information aggregation and a domain classifier; Inputting the first intermediate feature into the domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism, selecting and deleting the channel that contributes most to domain recognition, and scoring the contribution of the feature channel to domain recognition, including: Inputting the first intermediate feature into the attention mechanism of cross-spatial multi-scale information aggregation, and performing convolution, maximum pooling in the X direction, maximum pooling in the Y direction, average pooling in the X direction, and average pooling in the Y direction, respectively, to obtain convolution features, maximum pooling results in the X direction, maximum pooling results in the Y direction, average pooling results in the X direction, and average pooling results in the Y direction; The X-direction maximum pooling result and the Y-direction maximum pooling result are concatenated and then subjected to convolution processing, and then activated by a Sigmoid function to obtain a maximum pooling feature; The X-direction average pooling result and the Y-direction average pooling result are concatenated and then subjected to convolution processing, and then activated by a Sigmoid function to obtain an average pooling feature; Performing weighted summation on the maximum pooling feature, the average pooling feature, and the first intermediate feature, and then normalizing the obtained result to obtain a weighted normalized feature; The convolution feature is average pooled and Softmax activated, and then matrix multiplied with the weighted normalized feature to obtain a first fusion feature; The weighted normalized feature is average pooled and Softmax activated, and then matrix multiplied with the convolution feature to obtain a second fusion feature; The first fused feature and the second fused feature are added and fused, and then weighted with the first intermediate feature to obtain a domain-specific feature; Inputting the domain-specific features into the domain classifier to obtain domain label prediction; The first intermediate feature is subjected to global average pooling and then element-wise multiplied with the domain classifier weight to obtain a score of the contribution of the feature channel to domain recognition.

6. The method according to claim 4, characterized in that Generate a binary feature mask based on the score of each feature channel, including: For the score No. The channels generate random numbers and calculate key values; the key values ​​are: ； in, The score is No. The key value of each channel, Indicates The score of each channel, Indicates the generated A random number between 0 and 1 for each channel; Select the one with the largest key value key values ​​and will be the corresponding elements of the binary mask Set to 0; the corresponding element of the binary mask for: ； in, Indicates the one with the largest key value value.

7. The method according to claim 1, characterized in that The total loss function is constructed as: ； in, represents the total loss, represents the reasoning consistency loss, represents the classification prediction loss, represents the domain prediction loss, , denote the predicted label and domain label respectively, , Representative The class labels predicted by the classifier in the domain, Indicates that it comes from the source domain The input label.

8. The method according to claim 1, characterized in that The reasoning consistency loss is: ； in, is the number of samples in a batch; Indicates The prototype matrix of samples; Indicates that the corresponding The average matrix of the prototype matrices of the class; Update online in momentum at each iteration step ; The update formula is: ； ； in, , Respectively represent the current moment and the previous moment corresponding to the The average matrix of the prototype matrix of the class, represents the prototype matrix, represents the classifier weight, The output of the backbone network The characteristics of a channel.

9. A domain generalization device based on domain-invariant feature extraction, characterized in that: The device comprises: A domain generalization model construction module is used to establish a domain generalization model based on cross-space multi-scale information aggregation and reasoning consistency loss; the domain generalization model includes: a backbone network, a domain discriminator based on a cross-space multi-scale information aggregation attention mechanism, and a classifier; the domain discriminator based on a cross-space multi-scale information aggregation attention mechanism is used to aggregate cross-space information using multi-scale convolution operations, so that the network pays more attention to domain-invariant feature channels and suppresses domain-specific feature channels, and on the basis of feature selection, the contribution of feature channels to domain recognition is scored; A total loss function construction module, used to construct a total loss function; the total loss function includes domain prediction loss, reasoning consistency loss and classification prediction loss; the reasoning consistency loss is used to ensure that the prototype matrix of samples from the same category is consistent with their corresponding average values; A domain generalization model training module is used to train the domain generalization model according to the source domain data and the total loss function to obtain a trained domain generalization model; wherein, in each training batch, the domain discriminator based on the cross-spatial multi-scale information aggregation attention mechanism is randomly inserted into the middle layer of the backbone network to identify the feature channel with the largest contribution to a specific domain and delete it; The domain generalization module is used to use the trained domain generalization model to classify and predict the target domain data to obtain the classification results.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the domain generalization method based on domain-invariant feature extraction according to any one of claims 1 to 8 is implemented.

Images