Passive domain adaptive model training method, system and device and storage medium
By using cross-training of the basic network model and the verification network model in the passive domain adaptive model, the problems of low accuracy and data dependence in the passive domain adaptive model are solved, and the accuracy of the model is improved and Robustness avoids data transfer and privacy issues.
Patent Information
- Application Number
- CN202410890880.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-03
- Publication Date
- 2024-11-26
AI Technical Summary
The accuracy of the existing passive domain adaptive model is difficult to verify, the model prediction results cannot be evaluated, and the source domain data is required, which has difficulty in data transmission and privacy issues.
By acquiring the target domain image, constructing information maximization loss function and correlation matrix consistency loss function, using cross-consistency constraints to cross-train the basic network model and verification network model to reduce the unreliability and instability of pseudo-labels , improve accuracy and robustness.
The accuracy and robustness of the passive domain adaptive model is improved, the dependence on source domain data is avoided, and data transmission and privacy issues are solved.
Smart Images

Figure CN119027708A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of model training for image classification, and in particular to a passive domain adaptive model training method, system, device and storage medium. Background Art
[0002] Although deep neural networks have achieved great success in various computer vision tasks, they come at the cost of large labeled training datasets and extensive computing resources. Their performance tends to degrade significantly when the training and test samples come from different environments, which is often referred to as the domain drift problem. Unsupervised domain adaptation is a set of transfer learning methods that aims to transfer knowledge from a labeled dataset (source domain) to a new unlabeled dataset (target domain).
[0003] At present, unsupervised domain adaptation is divided into two categories, namely active domain adaptation methods and passive domain adaptation methods. Active domain adaptation methods include statistical distribution matching and generative adversarial network methods to minimize domain differences, but active domain adaptation methods must rely on labeled data in the source domain. In practical applications, due to data privacy issues or transmission difficulties, it is impossible to access source domain samples. Passive domain adaptation methods directly perform unsupervised transfer learning in the target domain without the need for labeled data in the source domain.
[0004] In the transfer learning image classification task, the performance of the model trained in the source domain is often greatly reduced when it is transferred to the target domain. Most of the existing transfer learning domain adaptation methods require access to the training data of the source domain, which is not applicable in some scenarios where data transmission is difficult or privacy is sensitive. In order to solve this data transmission or privacy problem, passive domain adaptation that does not require access to source domain data has been proposed. However, the accuracy of the current passive domain adaptation method is average. The core challenge is that in the absence of annotated target domains, the accuracy of the neural network model of passive domain adaptation transfer is often difficult to verify, and there is a lack of an effective method to evaluate the prediction results of the network model adapted in the target domain.
[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the invention
[0006] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a passive domain adaptive model training method, system, device and storage medium are provided, aiming to solve the problem in the prior art that the accuracy of the passive domain adaptive model is difficult to verify and the model prediction results cannot be evaluated.
[0007] The technical solution adopted by the present invention to solve the problem is as follows:
[0008] In a first aspect, an embodiment of the present invention provides a passive domain adaptive model training method, wherein the method comprises:
[0009] Acquire a target domain image, and input the target domain image into a basic network model to obtain the probability of each category of the target domain image and output features of the basic network model;
[0010] Determine an information maximization loss function based on the probability of each category of the target domain image;
[0011] Constructing a verification network training data set based on the probability of each category of the target domain image, and inputting the verification network training data set into a verification network model to obtain output features of the verification network model;
[0012] Determine a correlation matrix consistency loss function based on the output features of the basic network model and the output features of the verification network model;
[0013] Obtaining sample attention weights of the verification network training data set, and determining an attention consistency loss function based on the sample attention weights;
[0014] Based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, a basic network model update loss function is constructed, and based on the basic network model update loss function, a model training is performed on the basic network model to obtain a new output result of the basic network model, and the step of constructing a new verification network training data set based on the new output result of the basic network model is continued to perform model training on the verification network model until the verification network model converges.
[0015] In one implementation, the determining of the information maximization loss function based on the probability of each category of the target domain image includes:
[0016] The probability distribution of each category is obtained based on the probability calculation of each category of the target domain image;
[0017] Calculating the probability average of the target domain image based on the probability distribution of each category;
[0018] An information maximization loss function is constructed based on the probability distribution of each category and the probability average.
[0019] In one implementation, constructing a verification network training data set based on the probability of each category of the target domain image, and inputting the verification network training data set into a verification network model to obtain output features of the verification network model includes:
[0020] Determining sample confidence based on the probability of each category of the target domain image, and constructing a verification network training data set based on the sample confidence;
[0021] The verification network training data set is input into the verification network model to obtain the verification network model output features.
[0022] In one implementation, determining the sample confidence based on the probability of each category of the target domain image, and constructing a verification network training data set based on the sample confidence includes:
[0023] Calculating the information entropy corresponding to the probability of each category of the target domain image, and using the information entropy as the sample confidence;
[0024] Sort the confidence levels of the samples from large to small;
[0025] Selecting a target domain image corresponding to the sample confidence level greater than a preset confidence level to determine a training image;
[0026] Obtaining sample labels corresponding to the training images to obtain training image labels;
[0027] A validation network training data set is constructed based on the training images and training image labels.
[0028] In one implementation, determining the correlation matrix consistency loss function based on the output features of the basic network model and the output features of the verification network model includes:
[0029] Calculate the cosine similarity based on the output features of the basic network model;
[0030] Calculate a correlation matrix based on the output features of the verification network model;
[0031] A correlation matrix consistency loss function is constructed based on the cosine similarity and the correlation matrix.
[0032] In one implementation, obtaining the sample attention weights of the verification network training data set and determining the attention consistency loss function based on the sample attention weights includes:
[0033] Obtaining a sample attention weight of the basic network model for a verification network training data set to obtain a first sample attention weight;
[0034] Obtaining a sample attention weight of the verification network model for the verification network training data set to obtain a second sample attention weight;
[0035] An attention consistency loss function is determined based on the first sample attention weight and the second sample attention weight.
[0036] In one implementation, before constructing the basic network model update loss function based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, the method further includes:
[0037] A first balancing hyperparameter and a second balancing hyperparameter are obtained, wherein the first balancing hyperparameter is used to correct the correlation matrix consistency loss function, and the second balancing hyperparameter is used to correct the attention consistency loss function.
[0038] In a second aspect, an embodiment of the present invention further provides a passive domain adaptive model training system, wherein the system comprises:
[0039] A target image acquisition module is used to acquire a target domain image and input the target domain image into a basic network model to obtain the probability of each category of the target domain image and output features of the basic network model;
[0040] An information maximization loss function determination module, used to determine the information maximization loss function based on the probability of each category of the target domain image;
[0041] A verification network model training module is used to construct a verification network training data set based on the probability of each category of the target domain image, and input the verification network training data set into the verification network model to obtain the verification network model output features;
[0042] A correlation matrix consistency loss function determination module, used to determine a correlation matrix consistency loss function based on the output features of the basic network model and the output features of the verification network model;
[0043] An attention consistency loss function determination module, used to obtain the sample attention weights of the verification network training data set and determine the attention consistency loss function based on the sample attention weights;
[0044] The model iteration training module is used to construct a basic network model update loss function based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, and perform model training on the basic network model based on the basic network model update loss function to obtain a new output result of the basic network model, and continue to execute the step of constructing a new verification network training data set based on the new output result of the basic network model to perform model training on the verification network model until the verification network model converges.
[0045] In one implementation, the information maximization loss function determination module includes:
[0046] A probability distribution calculation unit, configured to calculate the probability distribution of each category based on the probability of each category of the target domain image;
[0047] A probability average calculation unit, used for calculating the probability average of the target domain image based on the probability distribution of each category;
[0048] The information maximization loss function determination unit is used to construct the information maximization loss function based on the probability distribution of each category and the probability average.
[0049] In one implementation, the verification network model training module includes:
[0050] A training data set construction unit, used to determine the sample confidence based on the probability of each category of the target domain image, and to construct a verification network training data set based on the sample confidence;
[0051] The verification network output feature acquisition unit is used to input the verification network training data set into the verification network model to obtain the verification network model output feature.
[0052] In one implementation, the training data set construction unit includes:
[0053] A confidence calculation subunit, used to calculate the information entropy corresponding to the probability of each category of the target domain image, and use the information entropy as the sample confidence;
[0054] A sorting subunit, used to sort the sample confidences from large to small;
[0055] A training image determination subunit, used to select a target domain image corresponding to the sample confidence level greater than a preset confidence level, and determine a training image;
[0056] The label acquisition subunit is used to acquire the sample label corresponding to the training image to obtain the training image label;
[0057] The training data set construction subunit is used to construct a verification network training data set based on the training images and training image labels.
[0058] In one implementation, the correlation matrix consistency loss function determination module includes:
[0059] A cosine similarity calculation unit, used to calculate the cosine similarity based on the output features of the basic network model;
[0060] A correlation matrix calculation unit, used for calculating a correlation matrix based on the output features of the verification network model;
[0061] A correlation matrix consistency loss function construction unit is used to construct a correlation matrix consistency loss function based on the cosine similarity and the correlation matrix.
[0062] In one implementation, the attention consistency loss function determination module includes:
[0063] A first sample attention weight acquisition unit is used to acquire the sample attention weight of the basic network model to the verification network training data set to obtain a first sample attention weight;
[0064] A second sample attention weight acquisition unit is used to acquire the sample attention weight of the verification network model to the verification network training data set to obtain a second sample attention weight;
[0065] An attention consistency loss function determining unit is used to determine an attention consistency loss function based on the first sample attention weight and the second sample attention weight.
[0066] In one implementation, the system further includes:
[0067] A loss function correction module is used to obtain a first balance hyperparameter and a second balance hyperparameter, wherein the first balance hyperparameter is used to correct the correlation matrix consistency loss function, and the second balance hyperparameter is used to correct the attention consistency loss function.
[0068] In a third aspect, the present invention also provides a terminal device, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include a method for executing a passive domain adaptive model training method as described in any one of the above.
[0069] In a fourth aspect, an embodiment of the present invention further provides a non-temporary computer-readable storage medium, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the passive domain adaptive model training method as described in any one of the above.
[0070] Beneficial effects of the present invention: Compared with the prior art, the present invention provides a passive domain adaptive model training method, system, device and storage medium. The present invention first obtains a target domain image, and inputs the target domain image into a basic network model to obtain the probabilities of each category of the target domain image and the output features of the basic network model; then determines an information maximization loss function based on the probabilities of each category of the target domain image; constructs a verification network training data set based on the probabilities of each category of the target domain image, and inputs the verification network training data set into the verification network model to obtain the output features of the verification network model; determines a correlation matrix consistency loss function based on the output features of the basic network model and the output features of the verification network model; obtains the sample attention weights of the verification network training data set, and determines an attention consistency loss function based on the sample attention weights; constructs a basic network model update loss function based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, and performs model training on the basic network model based on the basic network model update loss function to obtain a new output result of the basic network model, and continues to execute the step of constructing a new verification network training data set based on the new output result of the basic network model to perform model training on the verification network model until the verification network model converges. This application uses the cross-consistency constraints of the classification task and the relevance ranking task to cross-train the basic network model and the verification network model, and cooperate with each other, thereby reducing the unreliability and instability of the pseudo-labels of the existing domain adaptation method and greatly improving the accuracy and robustness of passive domain adaptation. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0072] Figure 1 It is a flowchart of a passive domain adaptive model training method provided by an embodiment of the present invention.
[0073] Figure 2 It is a framework illustration of the passive domain adaptive model training method provided by an embodiment of the present invention.
[0074] Figure 3 It is a schematic diagram of the correlation matrix consistency and attention consistency of the passive domain adaptive model training method provided by an embodiment of the present invention.
[0075] Figure 4It is an overall flow chart of model training of the passive domain adaptive model training method provided by an embodiment of the present invention.
[0076] Figure 5 It is a principle block diagram of a passive domain adaptive model training system provided by an embodiment of the present invention.
[0077] Figure 6 It is a block diagram of the internal structure principle of the terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0078] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0079] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0080] In the prior art, the neural network model for passive domain adaptive migration has low accuracy and poor robustness.
[0081] In order to solve the problems of the prior art, this embodiment provides a passive domain adaptation model training method, through which the accuracy and robustness of the passive domain adaptation model in image classification can be improved. In the specific implementation, first, the target domain image is obtained, and the target domain image is input into the basic network model to obtain the probabilities of each category of the target domain image and the output features of the basic network model; then, the information maximization loss function is determined based on the probabilities of each category of the target domain image; further, a verification network training data set is constructed based on the probabilities of each category of the target domain image, and the verification network training data set is input into the verification network model to obtain the output features of the verification network model; then, the correlation matrix consistency loss function is determined based on the output features of the basic network model and the output features of the verification network model; then, the sample attention weights of the verification network training data set are obtained, and the attention consistency loss function is determined based on the sample attention weights; finally, the basic network model update loss function is constructed based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, and the basic network model is model-trained based on the basic network model update loss function to obtain a new output result of the basic network model, and the step of constructing a new verification network training data set based on the new output result of the basic network model is continued to perform model training on the verification network model until the verification network model converges. Therefore, in the present invention, by utilizing the cross-consistency constraints of the classification task and the relevance ranking task, the basic network model and the verification network model are cross-trained and cooperate with each other, thereby reducing the unreliability and instability of the pseudo-labels of the existing domain adaptation method and greatly improving the accuracy and robustness of passive domain adaptation.
[0082] For example, first obtain the target domain image, and then input it into the basic network model for classification, so that the probability of each category of the group of images and the output characteristics of the basic network model can be obtained. The information maximization loss function of the basic network model can be determined using the probability of each category; then, based on the probability of each category of the target domain image, the target image is screened out to construct a verification network training data set. After the verification network training data set is constructed, it is input into the verification network model for model output to obtain the output characteristics of the verification network model; further, the correlation matrix consistency loss function is constructed based on the output characteristics of the basic network model and the output characteristics of the verification network model; the sample attention of the verification network training data set is obtained. Weights, an attention consistency loss function can be constructed; finally, the information maximization loss function determined according to the output result of the basic model, as well as the constructed correlation matrix consistency loss function and attention consistency loss function, reconstruct the loss function of the basic network model, and based on the new loss function, the basic network model is trained, and the step of constructing a new verification network training data set based on the new output result of the basic network model is continued to be performed to train the verification network model, and the step is repeated to cross-train the basic network model and the verification network model until the basic network model and the verification network model converge, and the trained basic network model and verification network model can be obtained. Therefore, in the present invention, by utilizing the cross-consistency constraints of the classification task and the correlation sorting task, the basic network model and the verification network model are cross-trained and cooperate with each other, thereby reducing the unreliability and instability of the pseudo-labels of the existing domain adaptation method, and greatly improving the accuracy and robustness of passive domain adaptation.
[0083] Exemplary Methods
[0084] The embodiment of the present invention provides a passive domain adaptive model training method, which can be applied to a terminal device. Figure 1 As shown, the method includes:
[0085] Step S100: Acquire a target domain image, and input the target domain image into a basic network model to obtain the probabilities of each category of the target domain image and output features of the basic network model.
[0086] In this embodiment, if Figure 2 As shown in the figure, we first need to obtain the target domain image, then input the target domain image into the basic network model, and train the basic network model. The basic network model can output the probability value of the target domain image being divided into various categories and the output features of the basic network model. In the specific implementation, refer to the unsupervised domain adaptation setting of transfer learning, first train the source domain model in the source domain, and then perform unsupervised domain adaptation on the source domain model in the target domain to obtain the basic network model.
[0087] In one implementation, ResNet50 / 101 pre-trained with the ImageNet dataset is used as the feature extractor of the basic network model, and the original classifier head is replaced with a normalization layer (BN) and a fully connected layer with 256 hidden neurons. Smooth cross entropy training is used in the source domain as the source domain basic network model, and the source domain basic network model is copied and the classification head is replaced with a binary classification as the verification network model. When migrating to the target domain, stochastic gradient descent with a learning rate of 0.001 is used to update the weights of the two network models (basic network model and verification network model), and all training and testing processes are completed on a single Tesla A100 GPU.
[0088] Step S200: determining an information maximization loss function based on the probabilities of each category of the target domain image.
[0089] In this embodiment, the information maximization loss function is determined based on the probability of each category of the target domain image, including: calculating the probability distribution of each category based on the probability of each category of the target domain image; calculating the probability average of the target domain image based on the probability distribution of each category; and constructing the information maximization loss function based on the probability distribution of each category and the probability average. The information maximization loss function will make the model predict the output probability close to the one-hot encoding, increase the confidence, and improve the accuracy of the model on the test set.
[0090] In the specific implementation, when the basic network model of step S100 is trained, the maximum loss function is obtained based on the probability of each category and the average probability, as shown in formula (1):
[0091]
[0092] in, represents the information maximization loss function, x t represents the target domain image, K represents the total number of categories, k represents the kth category, δ represents the softmax function, c represents the classifier, and f represents the feature extractor. The probability distribution of each category of the target domain image output by the basic network model is expressed as: δ k [c s (f t (x t ))], the probability average of the target domain image output is expressed as: Thus, the information maximization loss function of the basic network model when trained on the target domain can be obtained.
[0093] Step S300: construct a verification network training data set based on the probability of each category of the target domain image, and input the verification network training data set into the verification network model to obtain the output features of the verification network model.
[0094] In this embodiment, the verification network training data set is constructed based on the probability of each category of the target domain image, and the verification network training data set is input into the verification network model to obtain the output features of the verification network model, including: determining the sample confidence based on the probability of each category of the target domain image, and constructing the verification network training data set based on the sample confidence; inputting the verification network training data set into the verification network model to obtain the output features of the verification network model. The accuracy of the verification network model can be improved by selecting samples with high confidence to construct the verification network training data set and using the data set to train the verification network model.
[0095] In one implementation, the method of determining the sample confidence based on the probability of each category of the target domain image and constructing a verification network training data set based on the sample confidence includes: calculating the information entropy corresponding to the probability of each category of the target domain image, and using the information entropy as the sample confidence; sorting the sample confidences from large to small; selecting the target domain image corresponding to the sample confidence greater than a preset confidence to determine the training image; obtaining the sample label corresponding to the training image to obtain the training image label; and constructing a verification network training data set based on the training image and the training image label.
[0096] In the specific implementation, after the basic network model is trained, the verification network model needs to be trained, and the verification network training data set is constructed according to the output results of the basic network model. First, the prediction results of the trained basic network model are sorted according to the confidence, and then the samples with high confidence and their labels are selected to construct the training data set. The information entropy of the probability of each category of the target domain image is calculated and used as the confidence. The confidence calculated for each category probability is sorted from large to small, and samples with confidence greater than the preset confidence are selected to prepare for the construction of the training data set, that is, target domain images with high confidence are selected to determine them as training images for the verification network model. When the initial round of model training is performed, the preset sample confidence is set to 50%. As the epoch increases during the alternating training of the model, the preset sample confidence gradually increases to 100%. After the training image is determined, the sample label corresponding to the training image is further obtained to obtain the training image label; finally, the training image and the training image label constitute the verification network training data set.
[0097] In one implementation, the cross entropy loss function of the verification network model is expressed as formula (2):
[0098]
[0099] Among them, F a 、F b and Fc Represent different images, q k represents the kth element value of the one-hot vector label, and Φ represents the validation network. The training data set is input into the validation network model. For example, three pictures (F a 、F b 、F c ), where F a and F b In the same category, the verification model outputs the binary classification result, F c and F a 、F b If they are of the same category, the output is 1, otherwise it is 0.
[0100] Step S400: determining a correlation matrix consistency loss function based on the output features of the basic network model and the output features of the verification network model.
[0101] In this embodiment, the determination of the correlation matrix consistency loss function based on the output features of the basic network model and the output features of the verification network model includes: calculating cosine similarity based on the output features of the basic network model; calculating a correlation matrix based on the output features of the verification network model; and constructing a correlation matrix consistency loss function based on the cosine similarity and the correlation matrix. Constructing a correlation matrix consistency loss function allows the verification network model to provide guidance for the training of the basic network model, thereby enhancing the accuracy of the basic network model.
[0102] In specific implementation, once the verification network model is trained, it can be used to provide external guidance for the adaptation of the basic network model, which can be achieved by constructing correlation matrix consistency and attention consistency. The correlation matrix consistency loss function is shown in formula (3):
[0103]
[0104] Among them, C Γ (F a ,F b ) is the cosine similarity calculated from the output features of the basic network model, C Φ (F a ,F b ) is the correlation matrix output by the verification network model. Figure 3 Shown are the constructed correlation matrix consistency and attention consistency, Figure 3 (Left) is the correlation matrix. The first row is the correlation matrix of the basic network model, and the second row is the correlation matrix of the verification network. Figure 3 (Right) is the attention consistency. The first row is the attention distribution map of the basic network model, and the second row is the attention distribution map of the verification network.
[0105] Step S500: Obtain sample attention weights of the verification network training data set, and determine an attention consistency loss function based on the sample attention weights.
[0106] In this embodiment, the method of obtaining the sample attention weights of the verification network training data set and determining the attention consistency loss function based on the sample attention weights includes: obtaining the sample attention weights of the basic network model for the verification network training data set to obtain a first sample attention weight; obtaining the sample attention weights of the verification network model for the verification network training data set to obtain a second sample attention weight; and determining the attention consistency loss function based on the first sample attention weight and the second sample attention weight. Constructing the attention consistency loss function allows the verification network model to provide guidance for the training of the basic network model, thereby enhancing the accuracy of the basic network model.
[0107] In the specific implementation, the attention weights of the basic network model and the verification network model for the same sample in the training data set are obtained respectively, and then the attention consistency loss function is constructed based on the sample attention weights. The sample attention loss function is shown in formula (4):
[0108]
[0109] in, represents the attention weight of the basic network model on sample F, Represents the attention weight of the verification network model on sample F.
[0110] Step S600, constructing a basic network model update loss function based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, and performing model training on the basic network model based on the basic network model update loss function to obtain a new output result of the basic network model, and continuing to execute the step of constructing a new verification network training data set based on the new output result of the basic network model to perform model training on the verification network model until the verification network model converges.
[0111] In this embodiment, before constructing the basic network model update loss function based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, it also includes: obtaining a first balance hyperparameter and a second balance hyperparameter, the first balance hyperparameter is used to correct the correlation matrix consistency loss function, and the second balance hyperparameter is used to correct the attention consistency loss function.
[0112] In a specific implementation, before updating the loss function of the basic network model, it is necessary to first obtain the first balance hyperparameter and the second balance hyperparameter, and use the first balance hyperparameter and the second balance hyperparameter to perform balance modulation on it, as shown in formula (5):
[0113]
[0114] Wherein, λ1 represents the first balancing hyperparameter, and λ2 represents the second balancing hyperparameter.
[0115] In the specific implementation, the loss function of the basic network model is updated based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, and the basic network model is continued to be trained using the updated loss function. Then, the verification network training data set is continued to be updated according to the training results, and then the verification network model is trained and updated. The basic network model and the verification network model are trained alternately and cyclically until both the basic network model and the verification network model converge. Figure 4 shown.
[0116] The results of the target domain directly predicted by the base network are not accurate enough, especially when samples with large domain differences have low confidence. There is a lack of an effective method to evaluate the prediction results of the network model adapted in the target domain. To address this problem, the present invention proposes that when adjusting the network model to predict sample labels based on encoded features, these prediction results are used to construct new training samples with pseudo labels to learn a new verification check network that performs different but compatible tasks in the target. The assumption is that if the prediction results are accurate, the two networks should have highly consistent reasoning processes. Specifically, first, the base network model performs image classification, and the task of the verification network model is to relatively rank the samples, and the training labels of these samples are constructed based on the confidence of the prediction results of the base network model. Then two similarity metrics are developed, namely cross-network correlation matrix consistency and attention consistency, which provide important guidance for the unsupervised domain adaptation process and greatly optimize and improve the accuracy and robustness of the model.
[0117] The present invention utilizes the cross-consistency constraints (i.e., correlation matrix consistency and attention consistency constraints) of different tasks (i.e., classification task and correlation ranking task), and the two network models are cross-trained and cooperate with each other, which reduces the unreliability and instability of pseudo-labels of existing domain adaptation methods and greatly improves the accuracy and robustness of passive domain adaptation.
[0118] Different from the existing domain adaptation methods, the present invention does not need to access the source domain data. A verification network is developed using cross-constrained validation learning. For the cosine similarity of the two sample features output by the base network, the verification network directly outputs the correlation between the two samples, constructs the consistency of the correlation matrix, and pays attention to the same samples of the two networks. The entire cross-validation network architecture has greatly optimized the passive domain adaptive transfer learning to improve the accuracy and robustness of the entire model.
[0119] Exemplary Systems
[0120] Based on the above embodiments, the present invention also provides a passive domain adaptive model training system, such as Figure 5 As shown, the system in this embodiment includes: a target image acquisition module 10, an information maximization loss function determination module 20, a verification network model training module 30, a correlation matrix consistency loss function determination module 40, an attention consistency loss function determination module 50, and a model iteration training module 60. Among them, the target image acquisition module 10 is used to acquire the target domain image, and input the target domain image into the basic network model to obtain the probabilities of each category of the target domain image and the output features of the basic network model; the information maximization loss function determination module 20 is used to determine the information maximization loss function based on the probabilities of each category of the target domain image; the verification network model training module 30 is used to construct a verification network training data set based on the probabilities of each category of the target domain image, and input the verification network training data set into the verification network model to obtain the output features of the verification network model; the correlation matrix consistency loss function determination module 40 is used to determine the correlation matrix consistency based on the output features of the basic network model and the output features of the verification network model. loss function; the attention consistency loss function determination module 50 is used to obtain the sample attention weights of the verification network training data set, and determine the attention consistency loss function based on the sample attention weights; the model iteration training module 60 is used to construct a basic network model update loss function based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, and perform model training on the basic network model based on the basic network model update loss function to obtain a new output result of the basic network model, and continue to execute the step of constructing a new verification network training data set based on the new output result of the basic network model to perform model training on the verification network model until the verification network model converges.
[0121] In one implementation, the information maximization loss function determination module includes:
[0122] A probability distribution calculation unit, configured to calculate the probability distribution of each category based on the probability of each category of the target domain image;
[0123] A probability average calculation unit, used for calculating the probability average of the target domain image based on the probability distribution of each category;
[0124] The information maximization loss function determination unit is used to construct the information maximization loss function based on the probability distribution of each category and the probability average.
[0125] In one implementation, the verification network model training module includes:
[0126] A training data set construction unit, used to determine the sample confidence based on the probability of each category of the target domain image, and to construct a verification network training data set based on the sample confidence;
[0127] The verification network output feature acquisition unit is used to input the verification network training data set into the verification network model to obtain the verification network model output feature.
[0128] In one implementation, the training data set construction unit includes:
[0129] A confidence calculation subunit, used to calculate the information entropy corresponding to the probability of each category of the target domain image, and use the information entropy as the sample confidence;
[0130] A sorting subunit, used to sort the sample confidences from large to small;
[0131] A training image determination subunit, used to select a target domain image corresponding to the sample confidence level greater than a preset confidence level, and determine a training image;
[0132] The label acquisition subunit is used to acquire the sample label corresponding to the training image to obtain the training image label;
[0133] The training data set construction subunit is used to construct a verification network training data set based on the training images and training image labels.
[0134] In one implementation, the correlation matrix consistency loss function determination module includes:
[0135] A cosine similarity calculation unit, used to calculate the cosine similarity based on the output features of the basic network model;
[0136] A correlation matrix calculation unit, used for calculating a correlation matrix based on the output features of the verification network model;
[0137] A correlation matrix consistency loss function construction unit is used to construct a correlation matrix consistency loss function based on the cosine similarity and the correlation matrix.
[0138] In one implementation, the attention consistency loss function determination module includes:
[0139] A first sample attention weight acquisition unit is used to acquire the sample attention weight of the basic network model to the verification network training data set to obtain a first sample attention weight;
[0140] A second sample attention weight acquisition unit is used to acquire the sample attention weight of the verification network model to the verification network training data set to obtain a second sample attention weight;
[0141] An attention consistency loss function determining unit is used to determine an attention consistency loss function based on the first sample attention weight and the second sample attention weight.
[0142] In one implementation, the system further includes:
[0143] A loss function correction module is used to obtain a first balance hyperparameter and a second balance hyperparameter, wherein the first balance hyperparameter is used to correct the correlation matrix consistency loss function, and the second balance hyperparameter is used to correct the attention consistency loss function.
[0144] Based on the above embodiments, the present invention further provides a terminal device, the principle block diagram of the terminal device can be as follows: Figure 6 The terminal device is the host computer in the above embodiment, such as an ATM machine. The terminal device may include one or more processors 100 ( Figure 6 Only one is shown), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a program for a passive domain adaptive model training method. When one or more processors 100 execute the computer program 102, the various steps in the passive domain adaptive model training method embodiment can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of each template / unit in the passive domain adaptive model training system embodiment can be implemented, which is not limited here.
[0145] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0146] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal device. The memory 101 may also be used to temporarily store data that has been output or is to be output.
[0147] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal device to which the scheme of the present invention is applied. The specific terminal device may include more or less components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0148] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods, wherein any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0149] In summary, the present invention discloses a passive domain adaptive model training method, system, device and storage medium, the method comprising, first, obtaining a target domain image, and inputting the target domain image into a basic network model to obtain the probability of each category of the target domain image and the output feature of the basic network model; then, determining an information maximization loss function based on the probability of each category of the target domain image; further, constructing a verification network training data set based on the probability of each category of the target domain image, and inputting the verification network training data set into the verification network model to obtain the output feature of the verification network model; then determining a correlation matrix consistency loss function based on the output feature of the basic network model and the output feature of the verification network model; then obtaining the sample attention weight of the verification network training data set, and determining an attention consistency loss function based on the sample attention weight; finally, constructing a basic network model update loss function based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, and performing model training on the basic network model based on the basic network model update loss function to obtain a new output result of the basic network model, and continuing to execute the step of constructing a new verification network training data set based on the new output result of the basic network model to perform model training on the verification network model until the verification network model converges. Therefore, in the present invention, by utilizing the cross-consistency constraints of the classification task and the relevance ranking task, the basic network model and the verification network model are cross-trained and cooperate with each other, thereby reducing the unreliability and instability of the pseudo-labels of the existing domain adaptation method and greatly improving the accuracy and robustness of passive domain adaptation.
[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A passive domain adaptive model training method, characterized in that: The method comprises: Acquire a target domain image, and input the target domain image into a basic network model to obtain the probability of each category of the target domain image and output features of the basic network model; Determine an information maximization loss function based on the probability of each category of the target domain image; Constructing a verification network training data set based on the probability of each category of the target domain image, and inputting the verification network training data set into a verification network model to obtain output features of the verification network model; Determine a correlation matrix consistency loss function based on the output features of the basic network model and the output features of the verification network model; Obtaining sample attention weights of the verification network training data set, and determining an attention consistency loss function based on the sample attention weights; Based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, a basic network model update loss function is constructed, and based on the basic network model update loss function, a model training is performed on the basic network model to obtain a new output result of the basic network model, and the step of constructing a new verification network training data set based on the new output result of the basic network model is continued to perform model training on the verification network model until the verification network model converges.
2. The passive domain adaptive model training method according to claim 1, characterized in that: The determining of the information maximization loss function based on the probability of each category of the target domain image includes: The probability distribution of each category is obtained based on the probability calculation of each category of the target domain image; Calculating the probability average of the target domain image based on the probability distribution of each category; An information maximization loss function is constructed based on the probability distribution of each category and the probability average.
3. The passive domain adaptive model training method according to claim 1, characterized in that: The method of constructing a verification network training data set based on the probability of each category of the target domain image, and inputting the verification network training data set into a verification network model to obtain output features of the verification network model includes: Determining sample confidence based on the probability of each category of the target domain image, and constructing a verification network training data set based on the sample confidence; The verification network training data set is input into the verification network model to obtain the verification network model output features.
4. The passive domain adaptive model training method according to claim 3, characterized in that: The step of determining the sample confidence based on the probability of each category of the target domain image, and constructing a verification network training data set based on the sample confidence includes: Calculating the information entropy corresponding to the probability of each category of the target domain image, and using the information entropy as the sample confidence; Sort the confidence levels of the samples from large to small; Selecting a target domain image corresponding to the sample confidence level greater than a preset confidence level to determine a training image; Obtaining sample labels corresponding to the training images to obtain training image labels; A validation network training data set is constructed based on the training images and training image labels.
5. The passive domain adaptive model training method according to claim 1, characterized in that: Determining the correlation matrix consistency loss function based on the output features of the basic network model and the output features of the verification network model includes: Calculate the cosine similarity based on the output features of the basic network model; Calculate a correlation matrix based on the output features of the verification network model; A correlation matrix consistency loss function is constructed based on the cosine similarity and the correlation matrix.
6. The passive domain adaptive model training method according to claim 1, characterized in that: The obtaining of the sample attention weights of the verification network training data set and determining the attention consistency loss function based on the sample attention weights includes: Obtaining a sample attention weight of the basic network model for a verification network training data set to obtain a first sample attention weight; Obtaining a sample attention weight of the verification network model for the verification network training data set to obtain a second sample attention weight; An attention consistency loss function is determined based on the first sample attention weight and the second sample attention weight.
7. The passive domain adaptive model training method according to claim 1, characterized in that: Before constructing the basic network model update loss function based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, the method further includes: A first balancing hyperparameter and a second balancing hyperparameter are obtained, wherein the first balancing hyperparameter is used to correct the correlation matrix consistency loss function, and the second balancing hyperparameter is used to correct the attention consistency loss function.
8. A passive domain adaptive model training system, characterized in that: The system comprises: A target image acquisition module is used to acquire a target domain image and input the target domain image into a basic network model to obtain the probability of each category of the target domain image and output features of the basic network model; An information maximization loss function determination module, used to determine the information maximization loss function based on the probability of each category of the target domain image; A verification network model training module is used to construct a verification network training data set based on the probability of each category of the target domain image, and input the verification network training data set into the verification network model to obtain the verification network model output features; A correlation matrix consistency loss function determination module, used to determine a correlation matrix consistency loss function based on the output features of the basic network model and the output features of the verification network model; An attention consistency loss function determination module, used to obtain the sample attention weights of the verification network training data set and determine the attention consistency loss function based on the sample attention weights; The model iteration training module is used to construct a basic network model update loss function based on the information maximization loss function, the correlation matrix consistency loss function and the attention consistency loss function, and perform model training on the basic network model based on the basic network model update loss function to obtain a new output result of the basic network model, and continue to execute the step of constructing a new verification network training data set based on the new output result of the basic network model to perform model training on the verification network model until the verification network model converges.
9. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a program of a passive domain adaptive model training method stored in the memory and executable on the processor. When the processor executes the program of the passive domain adaptive model training method, the steps of the passive domain adaptive model training method as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program of a passive domain adaptation model training method. When the program of the passive domain adaptation model training method is executed by a processor, the steps of the passive domain adaptation model training method as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Passive active domain adaptive model training method and device
CN115186773A
Image classification method, system and device based on active domain self-adaption and medium
CN116630708A
Passive domain adaptive image classification method based on class guide meta-learning
CN117523295A
