Field generalization method based on specific knowledge of perception field

Abstract

The invention belongs to the technical field of transfer learning, and discloses a domain generalization method based on sensing domain specific knowledge, which comprises the following steps: step 1, acquiring a data set, dividing training data and test data, and constructing a domain knowledge fusion network module, a multi-head domain specific knowledge sensing network module and an OVA domain discriminator network module according to the training data; step 2, calculating loss of the domain knowledge fusion network module, the multi-head domain specific knowledge perception network module and the OVA domain discriminator network module; and step 3, testing by using the test data through the loss optimization model. The domain knowledge fusion network module provided by the invention can well integrate and adjust domain invariant knowledge and domain specific knowledge through adversarial learning with the OVA domain discriminator network module, so as to better summarize unseen target domains, thereby improving generalization.

Description

Technical Field

[0001] The present invention belongs to the technical field of transfer learning, and specifically relates to a domain generalization method based on domain-specific knowledge in the perception field. Background Art

[0002] Deep Neural Network (DNN) has achieved remarkable success in many applications. However, DNN is often limited to scenarios where the training and test data are independent and identically distributed, and this condition is often difficult to meet in practical applications. For example, in autonomous driving tasks, it is usually difficult to obtain a large amount of road construction data from various cities for training. Therefore, Domain Generalization (DG) technology aims to learn models from multiple source domains and generalize them to unknown domains with distribution shifts to make up for the deficiencies of DNN.

[0003] Current domain generalization methods mainly focus on three aspects: data augmentation, learning strategies, and representation learning. First, data augmentation methods mainly expand the original data set through operations such as image cropping and rotation to assist the model in learning the general features of the training data. Learning strategies improve the generalization ability of the model through means such as ensemble learning and meta-learning. Representation learning aims to learn the common representations between data in multiple domains, and these common representations can usually help the model understand unknown test data. These methods all use the general features of multiple domains for classification, but in practical applications, the target domains are diverse and unpredictable, and the specific features of each domain may have a beneficial effect on some target domains, and these specific features are ignored in current methods. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a domain generalization method based on domain-specific knowledge in the perception field. This method can retain a large amount of useful domain-specific knowledge. The proposed domain knowledge fusion network module can well integrate and adjust domain-invariant knowledge and domain-specific knowledge through adversarial learning with the OVA domain discriminator network module, so as to better generalize the unseen target domain, thereby improving the generalization ability.

[0005] To achieve the above object, the present invention is realized by the following technical solutions:

[0006] The present invention is a domain generalization method based on domain-specific knowledge in the perception field, and the method specifically includes the following steps:

[0007] Step 1: Obtain a dataset, divide it into training data and test data, and based on the training data, construct a domain knowledge fusion network module, a multi-head domain-specific knowledge perception network module, and an OVA domain discriminator network module. Among them, the domain knowledge fusion network module includes a Resnet50 residual network, an encoder, and a fully connected layer. The multi-head domain-specific knowledge perception network module includes N S feature extractors, and the OVA domain discriminator network module includes N S binary classifiers, where N is the number of source domains;

[0008] Step 2: Calculate the losses of the domain knowledge fusion network module, the multi-head domain-specific knowledge perception network module, and the OVA domain discriminator network module constructed in Step 1, that is, the domain perception loss of the multi-head domain-specific knowledge perception network module the loss l of the OVA domain discriminator network module sim the contrastive loss l contraSt the classification loss l CE the consistency loss l con ;

[0009] Step 3: Optimize the domain knowledge fusion network module, the multi-head domain-specific knowledge perception network module, and the OVA domain discriminator network module through the losses in Step 2, and use the test data for testing.

[0010] A further improvement of the present invention is that: Step 1 specifically includes the following steps:

[0011] Step 1.1: Obtain a dataset D, and divide the dataset D into a training dataset D S and test data D T , where

[0012] Step 1.2: According to the domain labels, divide the training dataset into where represents N k sample sets that satisfy the same data distribution, that is, share a domain label k, represents the i-th sample in the k-th domain;

[0013] Step 1.3: Use the Resnet50 residual network as the feature extraction backbone network Use an encoder to extract the features into codes, and use the fully connected layer as a linear classifier The Resnet50 residual network, the encoder, and the fully connected layer constitute the domain knowledge fusion network module;

[0014] Step 1.4. Construct knowledge perceptrons for each domain according to the number of domains in the training quantity, jointly form a multi-head domain-specific knowledge perception network module, and construct an OVA domain discriminator network.

[0015] A further improvement of the present invention lies in: in step 1.4, the multi-head domain-specific knowledge perception network module includes N S feature extractors f (k) (·; θ k ), k ∈ (1, N S ), N S is the number of source domains, and each of the feature extractors has the same structure and is a Resnet50 residual network structure.

[0016] A further improvement of the present invention lies in: in the step 1.4, the OVA domain discriminator network module O(·; σ) includes N s binary classifiers o (k) (·; σ k ), k ∈ (1, N S ), N S is the number of source domains, and the output of each binary classifier o (k) represents the matching degree estimation between the sample and the k-th domain.

[0017] A further improvement of the present invention lies in: the specific steps of the step 2 are as follows:

[0018] Step 2.1. Take a feature extractor f (k) (·; θ k ), k ∈ (1, N S ), use the sample as the input, and obtain the feature Set an encoder for each feature extractor Obtain the feature encoding of

[0019] The taken feature extractor f (k) (·; θ k ), k ∈ (1, N S ) domain perception loss The calculation method is:

[0020]

[0021] where represents the sample of the same class as the encoding , N k is the total number of samples in the k-th domain;

[0022] Other feature extractors f(r) (·; θ r ) Calculate the loss of the data using D r by the same method.

[0023] Step 2.2, after the N S feature extractors in the multi-head domain-specific knowledge perception network module are optimized, freeze the parameters of the multi-head domain-specific knowledge perception network module, and select the k-th feature extractor f (k) (·; σ k ) and the k-th binary classifier o (k) (·; σ k ) in the OVA domain discriminator network module. Using the samples belonging to any domain r as the input, obtain which is a two-dimensional vector, and each dimension represents the dissimilarity and similarity of the samples in the r-th domain with the k-th domain respectively. Let represent the output probability of all samples in the r-th domain. Then the loss of the k-th binary classifier o (k) (·; σ k ) in the OVA domain discriminator network module is:

[0024]

[0025] Finally, the loss l sim of the OVA domain discriminator network module is:

[0026]

[0027] Step 2.3, take the domain knowledge fusion network module, use all training data as the input, and obtain the encoding of the samples k ∈ (1, N S ), i ∈ (1, N k ), and the classification confidence Calculate the contrast loss e contrast and the classification loss l CE :

[0028]

[0029] where N is the total number of samples in the training dataset;

[0030] Step 2.4, the domain knowledge fusion network module needs to learn the domain-specific knowledge perceived in the multi-head domain-specific knowledge perception network module, and with the help of the OVA domain discriminator network module, calculate the consistency loss l con :

[0031]

[0032] The total loss l of the domain knowledge fusion network module is as follows:

[0033] l = l CE + α·l contrast + λ·l con。

[0034] A further improvement of the present invention lies in that: Step 3 specifically includes the following steps:

[0035] Step 3.1. Optimize the backpropagation of the k-th feature extractor of the multi-head domain-specific knowledge perception network module according to the perception loss calculated in Step 2.1, and the loss l of the OVA domain discriminator network module calculated in Step 2.2 and the consistency loss l calculated in Step 4 sim to optimize the backpropagation of the OVA domain discriminator network module, and optimize the backpropagation of the e to the domain knowledge fusion network module calculated in Step 2.4; con Step 3.2. Use the domain knowledge fusion network module to test the test data, and use the domain knowledge fusion network module to classify and predict the test data D

[0036] A further improvement of the present invention lies in that: Step 3.1 is specifically: T Step 3.1.1 Optimize each feature extractor in the multi-head domain-specific knowledge perception network module according to

[0037] Step 3.1.2. The domain knowledge fusion network module uses the idea of adversarial learning to compete with the OVA domain discriminator network module, and optimizes the domain knowledge fusion network module with the total loss l of the domain knowledge fusion network module;

[0038] Step 3.1.3. Freeze all the parameters of the multi-head domain-specific knowledge perception network module, so that l does not contain the gradient information of the multi-head domain-specific knowledge network module. The OVA domain discriminator network module uses the idea of adversarial learning to compete with the domain knowledge fusion network module through l

[0039] Step 3.1.4. The OVA domain discriminator network module uses l

[0040] = l sim - λ·l con for optimization learning. The domain knowledge fusion network module, the multi-head domain-specific knowledge perception network module, and the OVA domain discriminator network module share an Adam optimizer, and the learning rate is set to 5×10 disc sim con -5 con -5 -5 .

[0041] A further improvement of the present invention lies in that: in step 1.3, the encoder includes a fully connected layer that maps 2048 dimensions to 512 dimensions, a batch normalization layer, an activation function, a Dropout layer, a fully connected layer that maps 512 dimensions to 256 dimensions, and a fully connected layer that maps the data from 256 dimensions to 256 dimensions.

[0042] Advantages of the present invention: The multi-head domain-specific knowledge perception network module proposed in the present invention can retain a large amount of useful domain-specific knowledge.

[0043] The domain knowledge fusion network module proposed in the present invention can well integrate and adjust domain-invariant knowledge and domain-specific knowledge through adversarial learning with the OVA domain discriminator network module, so as to better generalize the unseen target domain, thereby improving the generalization ability.

[0044] The method proposed in the present invention is significantly better than the corresponding baseline models on multiple datasets for domain generalization. Description of the Drawings

[0045] Figure 1 is a flowchart of the method of the present invention.

[0046] Figure 2 is a network model diagram of the present invention. Detailed Embodiments

[0047] The following will disclose the embodiments of the present invention with diagrams. For the sake of clarity, many practical details will be described together in the following narrative. However, it should be understood that these practical details are not used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary. In addition, for the purpose of simplifying the diagrams, some conventional structures and components will be shown in a simple schematic manner in the diagrams.

[0048] As Figure 1 shown, the present invention is a domain generalization method based on perceiving domain-specific knowledge, and the domain generalization method specifically includes the following steps:

[0049] Step 1. Use domain generalization datasets such as the PACS dataset, TerraIncognita, and OfficeHome to obtain the datasets. The PACS dataset consists of four domains: Photo, Art painting, Cartoon, and Sketch. Use the data of one domain as the test data, and the other three domains as the training data. In the experimental data, the accuracy of A as the test domain, the accuracy of C as the test data, the accuracy of P as the test domain, and the accuracy of S as the test domain are respectively represented. Divide the training data and the test data, and construct a domain knowledge fusion network module, a multi-head domain-specific knowledge perception network module, and an OVA domain discriminator network module based on the training data. Among them, the domain knowledge fusion network module includes a Resnet50 residual network, an encoder, and a fully connected layer. The multi-head domain-specific knowledge perception network module includes N S feature extractors, and the OVA domain discriminator network module includes N S binary classifiers, where N S is the number of source domains.

[0050] Specifically, it includes the following steps:

[0051] Step 1.1. Obtain the dataset D, and divide the dataset D into a training dataset D S and a test data D T , where

[0052] Step 1.2. According to the domain labels, divide the training dataset into where represents N k sample sets that satisfy the same data distribution, that is, share a domain label k, represents the i-th sample in the k-th domain;

[0053] Step 1.3. Use the Resnet50 residual network as the feature extraction backbone network Use an encoder to extract the features into codes, and use a fully connected layer as a linear classifier The Resnet50 residual network, the encoder, and the fully connected layer constitute the domain knowledge fusion network module.

[0054] The encoder in this step includes a fully connected layer that maps 2048 dimensions to 512 dimensions, a batch normalization layer, an activation function, a Dropout layer, a fully connected layer that maps 512 dimensions to 256 dimensions, and a fully connected layer that maps the data from 256 dimensions to 256 dimensions, acting as a projection head to better perform the contrast learning task.

[0055] Step 1.4: Construct knowledge perceptrons for each domain according to the number of domains in the training quantity, jointly form a multi-head domain-specific knowledge perception network module, and construct an OVA domain discriminator network.

[0056] The multi-head domain-specific knowledge perception network module includes N S feature extractors f (k) (·; θ k ), k ∈ (1, N S ), N is the number of source domains, and each of the feature extractors has the same structure and is a Resnet50 residual network structure.

[0057] The OVA domain discriminator network module O(·; σ) includes N S binary classifiers o (k) (·; σ k ), k ∈ (1, N S ), N is the number of source domains, and the output of each binary classifier o (k) represents the matching degree estimation of the sample with the k-th domain.

[0058] Step 2: Calculate the loss of the domain knowledge fusion network module constructed in Step 1, the loss of the multi-head domain-specific knowledge perception network module, and the loss of the OVA domain discriminator network module, that is, the domain perception loss of the multi-head domain-specific knowledge perception network module The loss l sim of the OVA domain discriminator network module, the contrast loss l contrast , the classification loss l CE , and the consistency loss l con . The loss calculation specifically includes the following steps:

[0059] Step 2.1: Take a feature extractor f (k) (·; θ k ) of the multi-head domain-specific knowledge perception network module, k ∈ (1, N s ), use the sample as the input to obtain the feature Set an encoder for each feature extractor to obtain the encoding of the feature

[0060] The domain perception loss (k) (·; θ k ) of the taken feature extractor f S is calculated as follows: The calculation method is:

[0061]

[0062] Where Denote samples of the same category as the encoded k ; N is the total number of samples in the k-th domain;

[0063] Other feature extractor f (r) (·; θ r ) calculates the loss using D r 's data in the same way:

[0064]

[0065] Step 2.2. After optimizing the N s feature extractors in the multi-head domain-specific knowledge perception network module, freeze the parameters of the multi-head domain-specific knowledge perception network module, and select the k-th feature extractor f (k) (·; θ k ) and the k-th binary classifier o (k) (·; σ k ) in the OVA domain discriminator network module. Using samples belonging to any domain r as input, obtain which is a two-dimensional vector. Each dimension represents the dissimilarity and similarity of samples in the r-th domain with those in the k-th domain. Let represent the output probability of all samples in the r-th domain. Then the loss (k) (·; σ k ) of the k-th binary classifier o in the OVA domain discriminator network module is:

[0066]

[0067] Finally, the loss l sim of the OVA domain discriminator network module is:

[0068]

[0069] Step 2.3. Take the domain knowledge fusion network module, use all training data as input, and obtain the encoding of the samples k ∈ (1, N S ), i ∈ (1, N k ), and the classification confidence Calculate the contrast loss l contrast and the classification loss l CE :

[0070]

[0071] where N is the total number of samples in the training dataset;

[0072] Step 2.4: The domain knowledge fusion network module needs to learn the domain-specific knowledge perceived by the multi-head domain-specific knowledge perception network module, and with the help of the OVA domain discriminator network module, calculate the consistency loss l con :

[0073]

[0074] The total loss l of the domain knowledge fusion network module is as follows:

[0075] l = l CE + α·l contrast + λ·l con 。

[0076] Step 3: Optimize the domain knowledge fusion network module, the multi-head domain-specific knowledge perception network module, and the OVA domain discriminator network module through the loss in Step 2, and use the test data for testing, which specifically includes the following steps:

[0077] Step 3.1: According to the perception loss calculated in Step 2.1 Optimize the k-th feature extractor of the multi-head domain-specific knowledge perception network module by backpropagation, and the loss l of the OVA domain discriminator network module calculated in Step 2.2 sim And the consistency loss l calculated in Step 4 con Optimize the OVA domain discriminator network module by backpropagation, and optimize the l calculated in Step 2.4 for the domain knowledge fusion network module by backpropagation. Specifically:

[0078] Step 3.1.1 Optimize each feature extractor in the multi-head domain-specific knowledge perception network module according to Optimize;

[0079] Step 3.1.2: The domain knowledge fusion network module uses the idea of adversarial learning to compete with the OVA domain discriminator network module, and optimizes the domain knowledge fusion network module with the total loss l of the domain knowledge fusion network module;

[0080] Step 3.1.3: Freeze all the parameters of the multi-head domain-specific knowledge perception network module, so that l sim Does not contain the gradient information of the multi-head domain-specific knowledge network module. The OVA domain discriminator network module uses the idea of adversarial learning to compete with the domain knowledge fusion network module through l con And the OVA domain discriminator network module uses l disc = l sim - λ·l conFor optimization learning, the domain knowledge fusion network module, the multi-head domain-specific knowledge perception network module, and the OVA domain discriminator network module share an Adam optimizer, and the learning rate is set to 5×10 -5 .

[0081] Step 3.2: Use the domain knowledge fusion network module to test the test data. Utilize the domain knowledge fusion network module to classify and predict the test data D T .

[0082] Table 1 below shows the result comparison data of the present invention with other algorithms on the PACS dataset. Bold indicates the optimal accuracy, and underlines indicate the sub-optimal ones.

[0083] Table 1

[0084]

[0085] As can be seen from Table 1 above, the present application has obtained the highest accuracy of 88.9% in terms of average accuracy compared with some current advanced methods. Benefiting from the use of the specific knowledge of all domains, the present application has achieved the best result of 83.1% in the Sketch domain, which has the largest data distribution difference from other domains in the PACS dataset. It also performs optimally in the Art Painting domain and the Cartoon domain, which are 89.7% and 54.9% respectively, improving by 0.4% and 0.5% compared with the sub-optimal methods.

[0086] The above are only the embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. A domain generalization method based on perceptual domain-specific knowledge, characterized by: The domain generalization method specifically includes the following steps: Step 1: Obtain a data set, divide it into training data and test data, and build a domain knowledge fusion network module, a multi-head domain-specific knowledge perception network module and an OVA domain discriminator network module based on the training data. The domain knowledge fusion network module includes a Resnet50 residual network, an encoder and a fully connected layer, and the multi-head domain-specific knowledge perception network module includes N S feature extractors, the OVA domain discriminator network module includes N s binary classifiers, N is the number of source domains; Step 2: Calculate the loss of the domain knowledge fusion network module constructed in step 1, the loss of the multi-head domain-specific knowledge perception network module, and the loss of the OVA domain discriminator network module; Step 3: Use the test data to test the domain knowledge fusion network module, multi-head domain-specific knowledge perception network module and OVA domain discriminator network module through the loss optimization of step 2.

2. The domain generalization method based on perceptual domain-specific knowledge according to claim 1, characterized in that: The step 1 specifically includes the following steps: Step 1.1: Get data set D and divide it into training data set D s And the test data D T ,in Step 1.2: Divide the training data set into in Indicates N that satisfies the same data distribution k A set of samples, that is, sharing a domain label k, represents the i-th sample of the k-th domain; Step 1.3: Use Resnet50 residual network as the feature extraction backbone network Using an encoder Extract features as encoding to use fully connected layers as linear classifiers The Resnet50 residual network, encoder and fully connected layer constitute the domain knowledge fusion network module; Step 1.4: Build knowledge sensors for each field according to the number of training fields, together form a multi-head field-specific knowledge perception network module, and build an OVA field discriminator network.

3. The domain generalization method based on perceptual domain-specific knowledge according to claim 2, characterized in that: In step 1.4, the multi-head domain-specific knowledge perception network module includes N S feature extractor f (k) (·;θ k ), k∈(1, N S ), N S is the number of source domains, and each feature extractor has the same structure and is a Resnet50 residual network structure.

4. The domain generalization method based on perceptual domain-specific knowledge according to claim 2, characterized in that: In step 1.4, the OVA domain discriminator network module O(·;σ) includes N S A binary classifier o (k) (·;σ k ), k∈(1, N S ), N S is the number of source domains, each binary classifier o (k) The output of represents the matching estimate of the sample with the kth domain.

5. The domain generalization method based on perceptual domain-specific knowledge according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1: Take a feature extractor f of the multi-head domain-specific knowledge perception network module (k) (·;θ k ), k∈(1, N S ), with sample As input, we get the features Set up an encoder for each feature extractor Get Features Encoding The feature extractor f (k) (·;θ k ), k∈(1, N S ) domain-aware loss The calculation method is: in Representation and encoding Samples of the same type, N k is the total number of samples in the kth domain; Step 2.2: N in the multi-head domain-specific knowledge perception network module S After the optimization of feature extractors is completed, the parameters of the multi-head domain-specific knowledge perception network module are frozen and the kth feature extractor f is selected. (k) (·;θ k ) and the kth binary classifier o in the OVA domain discriminator network module (k) (·;σ k ), with samples belonging to any domain r As input, we get is a two-dimensional vector, each dimension represents the sample of the rth field The dissimilarity and similarity with the k-th field, let represents the output probability of all samples in the rth domain, then the kth binary classifier o in the OVA domain discriminator network module (k) (·;σ k ) for: The loss of the final OVA domain discriminator network module is l sim for: Step 2.3: Take the domain knowledge fusion network module, take all training data as input, and get the encoding of the sample k∈(1,N S ), i∈(1, N k ), and classification confidence Calculate the contrast loss l contrast and classification loss l CE : Where N is the total number of samples in the training data set; Step 2.4: The domain knowledge fusion network module needs to learn the domain-specific knowledge perceived by the multi-head domain-specific knowledge perception network module, and calculate the consistency loss l with the help of the OVA domain discriminator network module. con : The total loss l of the domain knowledge fusion network module is: l=l CE +a·l contrast +λ·l con。 6. The domain generalization method based on perceptual domain-specific knowledge according to claim 5, characterized in that: The step 3 specifically includes the following steps: Step 3.1: Perceptual loss calculated according to step 2.1 Optimize the back propagation of the kth feature extractor of the multi-head domain-specific knowledge perception network module, and the loss l of the OVA domain discriminator network module calculated in step 2.2 sim and the consistency loss l calculated in step 4 con Optimize the back propagation of the OVA domain discriminator network module, and optimize the back propagation of the domain knowledge fusion network module calculated in step 2.4; Step 3.2: Use the domain knowledge fusion network module to test the test data. T Make classification predictions.

7. The domain generalization method based on perceptual domain-specific knowledge according to claim 6, characterized in that: The step 3.1 is specifically as follows: Step 3.1.1: Each feature extractor in the multi-head domain-specific knowledge perception network module is optimization; Step 3.1.2, the domain knowledge fusion network module uses the idea of ​​adversarial learning to compete with the OVA domain discriminator network module, and the total loss l of the domain knowledge fusion network module is used to optimize the domain knowledge fusion network module; Step 3.1.3: Freeze all parameters of the multi-head domain-specific knowledge perception network module so that l sim Without the gradient information of the multi-head domain-specific knowledge network module, the OVA domain discriminator network module uses the idea of ​​adversarial learning. con To compete with the domain knowledge fusion network module, the OVA domain discriminator network module uses l disc = l sim -λ·l con For optimization learning, the domain knowledge fusion network module, the multi-head domain-specific knowledge perception network module, and the OVA domain discriminator network module share an Adam optimizer, and the learning rate is set to 5×10 -5 .

8. The domain generalization method based on perceptual domain difference knowledge according to claim 1, characterized in that: The encoder in step 1.3 includes a fully connected layer that maps 2048 dimensions to 512 dimensions, a batch normalization layer, an activation function, a Dropout layer, a fully connected layer that maps 512 dimensions to 256 dimensions, and a fully connected layer that maps data from 256 dimensions to 256 dimensions.

Images