Domain adaptation-oriented intra-class self-supervised learning method and device

Abstract

The invention relates to the technical field of domain adaptation, and provides an intra-class self-supervised learning method and device oriented to domain adaptation. The method comprises the following steps: acquiring a source domain feature vector and a target domain feature vector through a feature extractor of a to-be-migrated model, and determining distances between all samples and clustering centers in corresponding domains according to the source domain feature vector and the target domain feature vector to obtain a first loss function value; through a classifier of the to-be-migrated model, mutual information of all samples and corresponding feature vectors is determined according to the source domain feature vector and the target domain feature vector to obtain a second loss function value, and classification accuracy of the classifier is determined to obtain a third loss function value; and iteratively optimizing the parameters of the feature extractor and the classifier according to the first loss function value, the second loss function value and the third loss function value to obtain the trained to-be-migrated model, thereby solving the problem of poor decision output accuracy of the deep learning model caused by inconsistent probability distribution of training data and test data.

Description

Technical Field

[0001] The present invention relates to the field of domain adaptation technology, and in particular to a method and device for intra-class self-supervised learning oriented to domain adaptation. Background Art

[0002] In recent years, deep learning methods have been widely used in various fields due to their excellent performance. However, most of these deep learning methods rely on a large amount of independent and identically distributed training data. But in practical applications, the sources of training data and test data, training conditions and training environments are often quite different. Especially in some fields, there are problems such as small data sample size and difficulty in data collection and annotation, which often lead to large distribution differences between training data and test data.

[0003] When deep learning models face real-world scenarios, the probability distributions of training data and test data are inconsistent, that is, there is a data domain offset problem, which seriously affects the accuracy of the deep learning model's decision output, making the use of deep learning methods in practical applications less effective. Applying deep learning models to real-world scenarios poses greater security risks and poor practicality.

[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a domain-adaptive intra-class self-supervised learning method and device, the purpose of which is to introduce an intra-class self-supervised learning mechanism, and design a corresponding loss function based on the distance between all samples in the source domain and the target domain and the cluster center in the corresponding domain, the mutual information between all samples in the source domain and the target domain and the corresponding feature vectors, and the classification accuracy of all samples in the source domain and the target domain, so as to iteratively optimize the parameters of the deep learning model, improve the domain adaptation learning effect of the deep learning model, and enhance the generalization ability of the deep learning model when facing domain offset data; solve the problem in the prior art that the deep learning model has poor accuracy in decision output when facing real scenarios due to the inconsistent probability distribution of training data and test data.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a domain-adaptive intra-class self-supervised learning method, comprising:

[0008] Obtaining a source domain feature vector and a target domain feature vector through a feature extractor of the model to be migrated; determining the distances between all samples in the source domain and the target domain and the cluster centers in the corresponding domains according to the source domain feature vector and the target domain feature vector, and obtaining a first loss function value;

[0009] Determine, by a classifier of the model to be migrated, the mutual information between all samples in the source domain and the target domain and the corresponding feature vector according to the source domain feature vector and the target domain feature vector, and obtain a second loss function value;

[0010] Determining, by the classifier, classification accuracy of all samples in the source domain and the target domain according to the source domain feature vector and the target domain feature vector, and obtaining a third loss function value;

[0011] According to the first loss function value, the second loss function value and the third loss function value, the parameters of the feature extractor and the parameters of the classifier are iteratively optimized to obtain a trained model to be migrated.

[0012] Furthermore, the feature extractor of the model to be migrated is used to obtain a source domain feature vector and a target domain feature vector; according to the source domain feature vector and the target domain feature vector, the distance between all samples in the source domain and the target domain and the cluster center in the corresponding domain is determined to obtain the first loss function value, which includes:

[0013] The feature extractor embeds the features of the samples in the source domain into a space of preset dimensions to obtain a source domain original vector; normalizes the source domain original vector to obtain a source domain feature vector; the feature extractor embeds the features of the samples in the target domain into a space of preset dimensions to obtain a target domain original vector; normalizes the target domain original vector to obtain a target domain feature vector;

[0014] Determine the source domain cluster center of all source domain feature vectors on the source domain; determine the target domain cluster center of all target domain feature vectors on the target domain;

[0015] Determining a source domain center vector based on the source domain feature vector; determining a source domain similarity distribution vector based on the source domain cluster center and the source domain center vector;

[0016] Determining a target domain center vector based on the target domain feature vector; determining a target domain similarity distribution vector based on the target domain cluster center and the target domain center vector;

[0017] The distances between the source domain similarity distribution vector and all samples in the source domain are summed to obtain a first intermediate value; the distances between the target domain similarity distribution vector and all samples in the target domain are summed to obtain a second intermediate value; and the sum of the first intermediate value and the second intermediate value is determined as a first loss function value.

[0018] Furthermore, the calculation formula of the source domain center vector is:

[0019]

[0020] Where F(·) represents the feature extraction function, represents the i-th source domain feature vector;

[0021] The calculation formula of the target domain center vector is:

[0022]

[0023] in, represents the i-th target domain feature vector;

[0024] The calculation formula of the source domain similarity distribution vector is:

[0025]

[0026] in, represents the source domain cluster center, k represents the total number of source domain categories, represents the cluster center of the source domain category of the current batch, φ represents the learnable parameter, and j represents the source domain category of the current batch;

[0027] The calculation formula of the target domain similarity distribution vector is:

[0028]

[0029] in, represents the natural exponential function.

[0030] Furthermore, the calculation formula of the first loss function value is:

[0031]

[0032] in, represents the source domain similarity distribution vector, c s (·) represents samples in the source domain, represents the target domain similarity distribution vector, c t (·) represents samples in the target domain, represents the number of source domain feature vectors, represents the number of target domain feature vectors, i represents the sample number of the source domain feature vector or the target domain feature vector, represents the cross entropy function, and |·| represents the cardinality of the set.

[0033] Furthermore, the calculation formula of the second loss function value is:

[0034]

[0035] Among them, x represents the characteristics of the sample, y represents the label of the sample, represents the source domain sample space, represents the target domain sample space, θ represents the feature extraction network parameters of the feature extractor, represents mathematical expectation, p(y|x; θ) represents the probability that the output label is y given an input sample x when the feature extraction network parameter is θ. represents the cross entropy function.

[0036] Furthermore, the calculation formula of the third loss function value is:

[0037]

[0038] in, represents the source domain sample space, (x, y) represents the sample and the corresponding label, F(·) represents the feature extraction function, C(·) represents the cosine similarity classification function, σ(·) represents the Softmax function, represents the cross entropy function.

[0039] Further, the iteratively optimizing the parameters of the feature extractor and the parameters of the classifier according to the first loss function value, the second loss function value, and the third loss function value includes:

[0040] Calculate an overall optimization target value based on the first loss function value, the second loss function value, and the third loss function value;

[0041] Based on the first feature extractor formula, calculate the first feature extractor gradient of the first loss function value with respect to the parameters of the feature extractor; based on the second feature extractor formula, calculate the second feature extractor gradient of the second loss function value with respect to the parameters of the feature extractor; based on the third feature extractor formula, calculate the third feature extractor gradient of the third loss function value with respect to the parameters of the feature extractor; update the parameters of the feature extractor based on the first feature extractor gradient, the second feature extractor gradient and the third feature extractor gradient;

[0042] Calculating a first classification gradient of a second loss function value with respect to a parameter of the classifier based on a second classifier formula; calculating a third classifier gradient of a third loss function value with respect to a parameter of the classifier based on a third classifier formula; and updating the parameters of the classifier based on the second classifier gradient and the third classifier gradient;

[0043] The parameters of the feature extractor and the parameters of the classifier are iteratively optimized until the overall optimization target value converges to a preset value, and a trained model to be migrated is obtained based on the converged parameters of the feature extractor and the parameters of the classifier.

[0044] Furthermore, the calculation formula of the overall optimization target value is:

[0045]

[0046] Among them, θ F represents the parameters of the feature extractor, θ C represents the parameters of the classifier, represents the optimal θ that satisfies the overall optimization objective value F and θ C , represents the first loss function value, represents the second loss function value, represents the third loss function value, λ1 represents the first balance coefficient, λ2 represents the second balance coefficient, represents the θ that minimizes the overall optimization term F and θ C ;

[0047] The first feature extractor formula is: The second feature extractor formula is: The third feature extractor formula is: The second classifier formula is: The third classifier formula is:

[0048] In a second aspect, the present invention further provides a domain-adaptive intra-class self-supervised learning device, comprising:

[0049] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the domain adaptation-oriented intra-class self-supervised learning method described in the first aspect.

[0050] In a third aspect, the present invention further provides a non-volatile computer storage medium storing computer executable instructions, which are executed by one or more processors to complete the domain adaptation-oriented intra-class self-supervised learning method described in the first aspect.

[0051] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in a memory, and executing the domain adaptation-oriented intra-class self-supervised learning method as in the first aspect.

[0052] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, enables the computer or the processor to execute the domain-adaptive in-class self-supervised learning method as described in the first to fourth aspects and any one of them.

[0053] In the sixth aspect, the present invention also provides an intra-class self-supervised learning system for domain adaptation, including the intra-class self-supervised learning device for domain adaptation as in the second aspect, and using the intra-class self-supervised learning method for domain adaptation as described in the first aspect to complete the interaction with the intra-class self-supervised learning device for domain adaptation of the second aspect.

[0054] Different from the prior art, the present invention has at least the following beneficial effects:

[0055] The present invention designs corresponding loss functions based on the distances between all samples in the source domain and the target domain and the cluster centers in the corresponding domains, the mutual information between all samples in the source domain and the target domain and the corresponding feature vectors, and the classification accuracy of all samples in the source domain and the target domain, and iteratively optimizes the parameters of the model to be migrated by combining various loss functions; the present invention uses the loss function to select samples that are most helpful in improving the performance of the model to be migrated, so as to align the distribution of samples in the source domain and samples in the target domain through iterative optimization, reduce the data distribution difference, and migrate the ability of the feature extractor to extract source domain feature vectors in the source domain and the ability of the classifier to predict the corresponding output to extract target domain features in the target domain. The method improves the domain adaptation learning effect of the model to be transferred and its generalization ability in new environments or new fields, and enhances the robustness of the model to be transferred, so that it can more robustly cope with various input and data distribution changes, thereby improving the overall performance stability of the model to be transferred; it greatly reduces the obstacles of data domain offset problems faced by deep learning models in real scenarios, so that deep learning models can be safely applied in real scenarios, and solves the problem of poor accuracy of deep learning model decision outputs in the prior art due to inconsistent probability distributions of training data and test data when deep learning models face real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 It is a flowchart of a method for self-supervised learning within a class for domain adaptation provided by an embodiment of the present invention;

[0058] Figure 2 It is a flowchart of another domain-adaptive intra-class self-supervised learning method provided by an embodiment of the present invention;

[0059] Figure 3 is a flow chart of step 10 provided in an embodiment of the present invention;

[0060] Figure 4 is a flow chart of step 40 provided in an embodiment of the present invention;

[0061] Figure 5 is a structural diagram of a domain-adaptive intra-class self-supervised learning device provided by an embodiment of the present invention;

[0062] Figure 6 It is a schematic diagram of the architecture of another domain-adaptive intra-class self-supervised learning device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0065] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0066] In the description of the present invention, it is necessary to understand that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present disclosure.

[0067] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0068] When describing some embodiments, the expressions "coupling", "coupling" and "connection" and their derivatives may be used. For example, when describing some embodiments, the term "connection" may be used to indicate that two or more components are in direct physical or electrical contact with each other. For another example, when describing some embodiments, the term "coupling" may be used to indicate that two or more components are in direct physical or electrical contact. However, the terms "connection" or "coupling" may also refer to two or more components that are not in direct contact with each other, but still cooperate or interact with each other, such as "optical path coupling", "wireless connection", etc. The embodiments disclosed herein are not necessarily limited to the contents of the present invention.

[0069] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0070] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0071] Embodiment 1:

[0072] In order to solve the above problems, Figure 1 As shown, the embodiment of the present invention provides a domain-adaptive intra-class self-supervised learning method, including:

[0073] Step 10: Obtain a source domain feature vector and a target domain feature vector through a feature extractor of the model to be migrated; determine the distance between all samples in the source domain and the target domain and the cluster center in the corresponding domain according to the source domain feature vector and the target domain feature vector, and obtain a first loss function value.

[0074] The application scenarios of the embodiments of the present invention are described below:

[0075] The source domain (equivalent to training data) and the target domain (equivalent to test data) refer to different data distributions or data sets. The source domain refers to the data set or data distribution used for pre-training of the deep learning model; there are a large number of data samples (i.e., data sets or data distributions) in the source domain for learning, and these samples are used to train and build the original model to complete the target task. For example, in the image classification task, the ImageNet data set can be regarded as a source domain, which contains a large number of images and corresponding labels for training deep learning models. The target domain refers to the new data set or new data distribution to which the deep learning model will be applied; in the target domain, there are usually only a few samples available for learning, so the deep learning model needs to transfer the knowledge and features learned from the source domain to the target domain in order to obtain good performance on the target task in the target domain. For example, when applying the image classification model to the medical image data set, the medical image data set is the target domain. Through the knowledge transfer from the source domain to the target domain, the deep learning model can better adapt to the characteristics and data distribution of the target domain, thereby improving the performance on the target task.

[0076] The objectives of the embodiments of the present invention are: to use the model to be transferred that has been trained in the source domain to achieve good generalization performance in the target domain, that is, to be able to accurately classify or predict unlabeled new samples in the target task and make the output accuracy of the model to be transferred higher; to improve the adaptability of the model to be transferred in the target domain so that it can be safely applied to real scenarios in the target domain, thereby improving the practicality of using deep learning methods in practical applications.

[0077] Among them, the model to be migrated is a deep learning model; the model to be migrated includes a feature extractor and a classifier, the feature extractor is used to extract the features of the samples input into the model to be migrated, and the classifier is used to perform classification prediction for the target task according to the features. The embodiment of the present invention determines the most representative sample among all the samples in the source domain as the source domain cluster center; determines the most representative sample among all the samples in the target domain as the target domain cluster center; the distance between all samples in the source domain and the target domain and the cluster center in the corresponding domain refers to: the distance between each sample in the source domain and the source domain cluster center; the distance between each sample in the target domain and the target domain cluster center; the embodiment of the present invention obtains the distance between all samples in the source domain and the target domain and the cluster center in the corresponding domain by calculating the value of the first loss function.

[0078] Step 20: Determine the mutual information between all samples in the source domain and the target domain and the corresponding feature vectors according to the source domain feature vector and the target domain feature vector through the classifier of the model to be transferred, and obtain a second loss function value.

[0079] Among them, after the classifier obtains the source domain feature vector or the target domain feature vector extracted by the feature extractor, it determines the mutual information between the sample in the source domain and the source domain feature vector corresponding to the sample in the source domain, or determines the mutual information between the sample in the target domain and the target domain feature vector corresponding to the sample in the target domain.

[0080] Mutual information is used to measure the statistical correlation or dependence between two random variables (e.g., a sample in a source domain and a source domain feature vector corresponding to the sample in the source domain), and represents the amount of information contained in one random variable about another random variable. The embodiment of the present invention obtains the mutual information by calculating the value of the second loss function, so as to learn more effective and richer feature representations of samples in the source domain and samples in the target domain in iterations.

[0081] Step 30: Determine, by the classifier, the classification accuracy of all samples in the source domain and the target domain according to the source domain feature vector and the target domain feature vector, and obtain a third loss function value.

[0082] The embodiment of the present invention measures the classification accuracy of the classifier for all samples in the source domain and the target domain through the third loss function value.

[0083] Step 40: According to the first loss function value, the second loss function value and the third loss function value, iteratively optimize the parameters of the feature extractor and the parameters of the classifier to obtain a trained model to be migrated.

[0084] The present invention designs corresponding loss functions based on the distances between all samples in the source domain and the target domain and the cluster centers in the corresponding domains, the mutual information between all samples in the source domain and the target domain and the corresponding feature vectors, and the classification accuracy of all samples in the source domain and the target domain, and iteratively optimizes the parameters of the model to be migrated by combining various loss functions; the present invention uses the loss function to select samples that are most helpful in improving the performance of the model to be migrated, so as to align the distribution of samples in the source domain and samples in the target domain through iterative optimization, reduce the data distribution difference, and migrate the ability of the feature extractor to extract source domain feature vectors in the source domain and the ability of the classifier to predict the corresponding output to extract target domain features in the target domain. The method improves the domain adaptation learning effect of the model to be transferred and its generalization ability in new environments or new fields, and enhances the robustness of the model to be transferred, so that it can more robustly cope with various input and data distribution changes, thereby improving the overall performance stability of the model to be transferred; it greatly reduces the obstacles of data domain offset problems faced by deep learning models in real scenarios, so that deep learning models can be safely applied in real scenarios, and solves the problem of poor accuracy of deep learning model decision outputs in the prior art due to inconsistent probability distributions of training data and test data when deep learning models face real scenarios.

[0085] The overall process of iteratively optimizing the parameters of the feature extractor and the classifier in the embodiment of the present invention is as follows: Figure 2 shown.

[0086] In order to illustrate the process of determining the value of the first loss function, as shown in Figure 3 As shown, the step 10 includes:

[0087] Step 101: The feature extractor embeds the features of the samples in the source domain into a space of preset dimensions to obtain a source domain original vector; normalizes the source domain original vector to obtain a source domain feature vector; the feature extractor embeds the features of the samples in the target domain into a space of preset dimensions to obtain a target domain original vector; normalizes the target domain original vector to obtain a target domain feature vector.

[0088] The preset dimension is selected by those skilled in the art according to a specific usage scenario; in an optional embodiment, the preset dimension may be 512.

[0089] In one embodiment, the method for normalizing the original vector of the source domain may be: each value in the original vector of the source domain is divided by the sum of the squares of the original vector of the source domain to obtain an intermediate value, and the square root of the intermediate value is taken, and the result after the square root is taken as the corresponding source domain feature vector; the method for normalizing the original vector of the target domain is similar.

[0090] Step 102: determining the source domain clustering center of all source domain feature vectors in the source domain; and determining the target domain clustering center of all target domain feature vectors in the target domain.

[0091] Among them, the method of calculating the source domain cluster center and the target domain cluster center is selected by technical personnel in this field according to the specific usage scenario; in an optional embodiment, the k-means algorithm can be used to calculate the source domain cluster center and the target domain cluster center respectively.

[0092] Step 103: determining a source domain center vector based on the source domain feature vector; and determining a source domain similarity distribution vector according to the source domain cluster center and the source domain center vector.

[0093] The embodiment of the present invention calculates the source domain similarity distribution vector of the current batch through the source domain cluster center and the source domain center vector.

[0094] In an optional embodiment, the calculation formula of the source domain center vector is:

[0095]

[0096] Where F(·) represents the feature extraction function, Denotes the ith source domain feature vector. The embodiment of the present invention performs a nonlinear transformation on the features of the input source domain feature vector to obtain a source domain center vector.

[0097] In an optional embodiment, the calculation formula of the source domain similarity distribution vector is:

[0098]

[0099] in, represents the source domain cluster center, k represents the total number of source domain categories, represents the cluster center of the source domain category of the current batch, φ represents the learnable parameter, and j represents the source domain category of the current batch. s That is, the cluster center of a specific category of data in the source domain in the feature space.

[0100] Step 104: determining a target domain center vector based on the target domain feature vector; and determining a target domain similarity distribution vector according to the target domain cluster center and the target domain center vector.

[0101] In an optional embodiment, the target domain center vector is calculated as follows:

[0102]

[0103] in, represents the i-th target domain feature vector.

[0104] In an optional embodiment, the target domain similarity distribution vector is calculated as follows:

[0105]

[0106] in, exp(·) represents the natural exponential function.

[0107] Step 105: summing the distances between the source domain similarity distribution vector and all samples in the source domain to obtain a first intermediate value; summing the distances between the target domain similarity distribution vector and all samples in the target domain to obtain a second intermediate value; and determining the sum of the first intermediate value and the second intermediate value as a first loss function value.

[0108] In an optional embodiment, the calculation formula of the first loss function value is:

[0109]

[0110] in, represents the source domain similarity distribution vector, c s (·) represents samples in the source domain, represents the target domain similarity distribution vector, c t (·) represents samples in the target domain, represents the number of source domain feature vectors, represents the number of target domain feature vectors, i represents the sample number of the source domain feature vector or the target domain feature vector, represents the cross entropy function, and |·| represents the cardinality of the set.

[0111] The specific implementation of the cross entropy function is determined by those skilled in the art based on specific usage scenarios and experience, and is not limited here.

[0112] The first loss function calculates the cross entropy of the similarity between the source domain feature vector and the target domain feature vector to the corresponding cluster center and the sample label, and uses the cross entropy as the target for model optimization, which improves the cohesion of intra-class data and increases the vector distance between inter-class data. The feature extractor trained in this way has a strong ability to distinguish feature data of different categories.

[0113] The embodiment of the present invention expects that the model to be migrated after training can output high confidence and diversified prediction results. In order to maximize the mutual information between the input of the model to be migrated and the output of the model to be migrated, a calculation formula for the second loss function value is designed; the goals expected to be achieved by the calculation formula are divided into two items: maximizing the entropy of the mathematical expectation of the network prediction of the model to be migrated, and minimizing the entropy of the network output. Specifically, the calculation formula of the second loss function value in step 20 is:

[0114]

[0115] Among them, x represents the characteristics of the sample, y represents the label of the sample, represents the source domain sample space, represents the target domain sample space, θ represents the feature extraction network parameters of the feature extractor, represents mathematical expectation, p(y|x; θ) represents the probability that the output label is y given an input sample x when the feature extraction network parameter is θ. represents the cross entropy function.

[0116] The embodiment of the present invention optimizes the value of the second loss function so that the entropy of the mathematical expectation of the network prediction of the model to be migrated is as large as possible, and the entropy of the network output is as small as possible; at this time, the model to be migrated after training can output high-confidence and diversified prediction results, which significantly increases the mutual information between the input of the model to be migrated and the output of the model to be migrated, thereby helping to improve the effect of the model in the target domain.

[0117] The embodiment of the present invention designs a calculation formula for the third loss function value to annotate the category prediction of samples in the source domain and the target domain through a classifier. Specifically, the calculation formula for the third loss function value in step 30 is:

[0118]

[0119] in, represents the source domain sample space, (x, y) represents the sample and the corresponding label, F(·) represents the feature extraction function, C(·) represents the cosine similarity classification function, σ(·) represents the Softmax function, represents the cross entropy function.

[0120] The embodiment of the present invention introduces the cosine similarity calculation of the feature vector and the category center into the third loss function value, so as to better capture the directional difference between the source domain feature vector and the target domain feature vector, reduce the interference of irrelevant dimensions in the high-dimensional vector space, stabilize the sample similarity measurement, make the classification boundary of the trained model to be migrated clearer, and improve the classification accuracy.

[0121] In order to illustrate the process of iterative optimization of the parameters of the feature extractor and classifier, Figure 4 As shown, the step 40 includes:

[0122] Step 401: Calculate an overall optimization target value based on the first loss function value, the second loss function value and the third loss function value.

[0123] In an optional embodiment, the calculation formula of the overall optimization target value is:

[0124]

[0125] Among them, θ F represents the parameters of the feature extractor, θ C represents the parameters of the classifier, represents the optimal θ that satisfies the overall optimization objective value F and θ C , represents the first loss function value, represents the second loss function value, represents the third loss function value, λ1 represents the first balance coefficient, λ2 represents the second balance coefficient, represents the θ that minimizes the overall optimization term F and θ C .

[0126] In an embodiment of the present invention, the first balance coefficient and the second balance coefficient will change dynamically with the iteration rounds to reduce the interference of the gradient of the first loss function in the initial training stage on the performance improvement of the model to be migrated.

[0127] Step 402: Based on the first feature extractor formula, calculate the first feature extractor gradient of the first loss function value with respect to the parameters of the feature extractor; based on the second feature extractor formula, calculate the second feature extractor gradient of the second loss function value with respect to the parameters of the feature extractor; based on the third feature extractor formula, calculate the third feature extractor gradient of the third loss function value with respect to the parameters of the feature extractor; update the parameters of the feature extractor based on the first feature extractor gradient, the second feature extractor gradient and the third feature extractor gradient.

[0128] In an optional embodiment, the first feature extractor formula is: The second feature extractor formula is: The third feature extractor formula is:

[0129] Step 403: Calculate a first classification gradient of a second loss function value with respect to the parameters of the classifier based on the second classifier formula; calculate a third classifier gradient of a third loss function value with respect to the parameters of the classifier based on the third classifier formula; and update the parameters of the classifier based on the second classifier gradient and the third classifier gradient.

[0130] In an optional embodiment, the second classifier formula is: The third classifier formula is:

[0131] After updating the parameters of the feature extractor and the classifier, the back propagation of this round of iteration is completed.

[0132] Step 404: iteratively optimize the parameters of the feature extractor and the parameters of the classifier until the overall optimization target value converges to a preset value, and obtain a trained model to be migrated based on the converged parameters of the feature extractor and the parameters of the classifier.

[0133] Among them, the preset value is selected by those skilled in the art according to the specific usage scenario and is not limited here. After the back propagation is completed, a new round of iterative training will be started. After the iteration based on the overall optimization target value in the embodiment of the present invention (that is, the training is completed), a model to be migrated with good performance in the target domain is obtained, and the parameters of the feature extractor and classifier of the trained model to be migrated are used to predict new test data in the real scenario of the target domain, and an output with higher accuracy can be obtained.

[0134] It should be noted that the domain adaptation-oriented intra-class self-supervised learning method of the embodiment of the present invention is not only applicable to classic neural network models represented by convolutional neural networks, but also to visual Transformer models.

[0135] In order to verify that the intra-class self-supervised learning method for domain adaptation of the embodiment of the present invention can achieve effective domain adaptation in various domain offset scenarios, a specific example of training on the DomainNet dataset is provided below; when the models to be migrated are the BiT model, the DeiT-B / 16 model, the DeiT-S / 16 model and the VGG-16 model, the effects of using the intra-class self-supervised learning method for domain adaptation of the embodiment of the present invention on different types of models to be migrated are compared, as shown in the following table:

[0136]

[0137] As indicated by the numerical values ​​in columns 3 to 9 in the above table: the generalization ability of domain offset data of the model to be migrated when migrating from different source domains to different target domains. As indicated by column 10 in the above table: the average value of the generalization ability of domain offset data of the model to be migrated in the scenario of migrating from the source domain to the target domain. It can be seen from the data in the above table that the in-class self-supervised learning method for domain adaptation of the embodiment of the present invention has certain effects on the BiT model, DeiT-B / 16 model, DeiT-S / 16 model and VGG-16 model, and improves the generalization ability of domain offset data of the corresponding model to be migrated by 5%, 4.2%, 8.2% and 5.9% respectively on average.

[0138] Embodiment 2:

[0139] In an optional embodiment, if Figure 5 , which is a schematic diagram of the structure of a domain-adaptive intra-class self-supervised learning device according to an embodiment of the present invention; the domain-adaptive intra-class self-supervised learning device according to an embodiment of the present invention comprises a feature extractor F, a feature memory Vs, a feature memory Vt, a classifier C, a self-supervisor and a back propagator, wherein:

[0140] Feature extractor F: used to extract source domain feature vectors of all samples from the source domain and target domain feature vectors of all samples from the target domain; in each round of iteration, the parameters of the feature extractor are adjusted through back propagation.

[0141] Feature memory Vs: used to store the source domain feature vectors of all samples in the source domain; the source domain feature vectors in the feature memory Vs will be updated after each round of iteration.

[0142] Feature memory Vt: used to store the target domain feature vectors of all samples in the target domain; the target domain feature vectors in the feature memory Vt will be updated after each round of iteration.

[0143] Classifier C: obtains the source domain feature vector from the feature memory Vs, obtains the target domain feature vector from the feature memory Vt, and calculates the second loss function value and the third loss function value according to the source domain feature vector and the target domain feature vector respectively; in each round of iteration, the parameters of classifier C are adjusted by back propagation.

[0144] Self-supervisor: Obtain the source domain feature vector from the feature memory Vs, obtain the target domain feature vector from the feature memory Vt, and calculate the first loss function value.

[0145] Back propagator: Calculate the overall optimization target value based on the first loss function value, the second loss function value, and the third loss function value, and adjust the parameters of the feature extractor and the parameters of the classifier C through back propagation.

[0146] In another optional embodiment, if Figure 6 , which is a schematic diagram of the architecture of a domain-adaptive intra-class self-supervised learning device according to an embodiment of the present invention. The domain-adaptive intra-class self-supervised learning device according to this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0147] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0148] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the in-class self-supervised learning method for domain adaptation in this embodiment. The processor 21 executes the in-class self-supervised learning method for domain adaptation by running the non-volatile software programs and instructions stored in the memory 22.

[0149] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0150] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the intra-class self-supervised learning method for domain adaptation in the above-mentioned embodiment is executed, for example, each step of the intra-class self-supervised learning method for domain adaptation in the above-described embodiment of the present invention is executed.

[0151] An embodiment of the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors, such as Figure 6 A processor 21 can enable the above one or more processors to execute the in-class self-supervised learning method for domain adaptation in the specific implementation of the present invention, for example, to execute each step of the in-class self-supervised learning method for domain adaptation in the embodiment of the present invention described above; it can also realize Figure 6 The modules and units described above; or executing the domain adaptation-oriented intra-class self-supervised learning method in the specific implementation mode of the present invention, for example, executing the steps of the domain adaptation-oriented intra-class self-supervised learning method in the above-described embodiment of the present invention; it can also be realized Figure 6 The various modules and units described.

[0152] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0153] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0154] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for intra-class self-supervised learning for domain adaptation, characterized in that: include: Obtaining a source domain feature vector and a target domain feature vector through a feature extractor of the model to be migrated; determining the distances between all samples in the source domain and the target domain and the cluster centers in the corresponding domains according to the source domain feature vector and the target domain feature vector, and obtaining a first loss function value; Determine, by a classifier of the model to be migrated, the mutual information between all samples in the source domain and the target domain and the corresponding feature vector according to the source domain feature vector and the target domain feature vector, and obtain a second loss function value; Determining, by the classifier, classification accuracy of all samples in the source domain and the target domain according to the source domain feature vector and the target domain feature vector, and obtaining a third loss function value; According to the first loss function value, the second loss function value and the third loss function value, the parameters of the feature extractor and the parameters of the classifier are iteratively optimized to obtain a trained model to be migrated.

2. The domain adaptation-oriented intra-class self-supervised learning method according to claim 1, characterized in that: The step of obtaining a source domain feature vector and a target domain feature vector through a feature extractor of the model to be migrated; determining the distances between all samples in the source domain and the target domain and the cluster centers in the corresponding domains according to the source domain feature vector and the target domain feature vector, and obtaining the first loss function value includes: The feature extractor embeds the features of the samples in the source domain into a space of preset dimensions to obtain a source domain original vector; normalizes the source domain original vector to obtain a source domain feature vector; the feature extractor embeds the features of the samples in the target domain into a space of preset dimensions to obtain a target domain original vector; normalizes the target domain original vector to obtain a target domain feature vector; Determine the source domain cluster center of all source domain feature vectors on the source domain; determine the target domain cluster center of all target domain feature vectors on the target domain; Determining a source domain center vector based on the source domain feature vector; determining a source domain similarity distribution vector based on the source domain cluster center and the source domain center vector; Determining a target domain center vector based on the target domain feature vector; determining a target domain similarity distribution vector based on the target domain cluster center and the target domain center vector; The distances between the source domain similarity distribution vector and all samples in the source domain are summed to obtain a first intermediate value; the distances between the target domain similarity distribution vector and all samples in the target domain are summed to obtain a second intermediate value; and the sum of the first intermediate value and the second intermediate value is determined as a first loss function value.

3. The domain adaptation-oriented intra-class self-supervised learning method according to claim 2, characterized in that: The calculation formula of the source domain center vector is: Where F(·) represents the feature extraction function, represents the i-th source domain feature vector; The calculation formula of the target domain center vector is: in, represents the i-th target domain feature vector; The calculation formula of the source domain similarity distribution vector is: in, represents the source domain cluster center, k represents the total number of source domain categories, represents the cluster center of the source domain category of the current batch, φ represents the learnable parameter, and j represents the source domain category of the current batch; The calculation formula of the target domain similarity distribution vector is: in, exp(·) represents the natural exponential function.

4. The domain adaptation-oriented intra-class self-supervised learning method according to claim 2, characterized in that: The calculation formula of the first loss function value is: in, represents the source domain similarity distribution vector, c s (·) represents samples in the source domain, represents the target domain similarity distribution vector, c t (·) represents samples in the target domain, represents the number of source domain feature vectors, represents the number of target domain feature vectors, i represents the sample number of the source domain feature vector or the target domain feature vector, represents the cross entropy function, and |·| represents the cardinality of the set.

5. The domain adaptation-oriented intra-class self-supervised learning method according to claim 1, characterized in that: The calculation formula of the second loss function value is: Among them, x represents the characteristics of the sample, y represents the label of the sample, represents the source domain sample space, represents the target domain sample space, θ represents the feature extraction network parameters of the feature extractor, represents mathematical expectation, p(y|x; θ) represents the probability that the output label is y given an input sample x when the feature extraction network parameter is θ. represents the cross entropy function.

6. The domain adaptation-oriented intra-class self-supervised learning method according to claim 1, characterized in that: The calculation formula of the third loss function value is: in, represents the source domain sample space, (x, y) represents the sample and the corresponding label, F(·) represents the feature extraction function, C(·) represents the cosine similarity classification function, σ(·) represents the Softmax function, represents the cross entropy function.

7. The domain adaptation-oriented intra-class self-supervised learning method according to claim 1, characterized in that: The iteratively optimizing the parameters of the feature extractor and the parameters of the classifier according to the first loss function value, the second loss function value, and the third loss function value comprises: Calculate an overall optimization target value based on the first loss function value, the second loss function value, and the third loss function value; Based on the first feature extractor formula, calculate the first feature extractor gradient of the first loss function value with respect to the parameters of the feature extractor; based on the second feature extractor formula, calculate the second feature extractor gradient of the second loss function value with respect to the parameters of the feature extractor; based on the third feature extractor formula, calculate the third feature extractor gradient of the third loss function value with respect to the parameters of the feature extractor; update the parameters of the feature extractor based on the first feature extractor gradient, the second feature extractor gradient and the third feature extractor gradient; Calculating a first classification gradient of a second loss function value with respect to a parameter of the classifier based on a second classifier formula; calculating a third classifier gradient of a third loss function value with respect to a parameter of the classifier based on a third classifier formula; and updating the parameters of the classifier based on the second classifier gradient and the third classifier gradient; The parameters of the feature extractor and the parameters of the classifier are iteratively optimized until the overall optimization target value converges to a preset value, and a trained model to be migrated is obtained based on the converged parameters of the feature extractor and the parameters of the classifier.

8. The domain adaptation-oriented intra-class self-supervised learning method according to claim 7, characterized in that: The calculation formula of the overall optimization target value is: Among them, θ F represents the parameters of the feature extractor, θ C represents the parameters of the classifier, represents the optimal θ that satisfies the overall optimization objective value F and θ C , represents the first loss function value, represents the second loss function value, represents the third loss function value, λ1 represents the first balance coefficient, λ2 represents the second balance coefficient, represents the θ that minimizes the overall optimization term F and θ C ; The first feature extractor formula is: The second feature extractor formula is: The third feature extractor formula is: The second classifier formula is: The third classifier formula is:

9. A domain-adaptive intra-class self-supervised learning device, characterized in that: The intra-class self-supervised learning device for domain adaptation includes at least one processor and a memory, and the at least one processor and the memory are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to implement the intra-class self-supervised learning method for domain adaptation described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the domain adaptation-oriented intra-class self-supervised learning method described in any one of claims 1-8.

Images