A multi-view based low-rank domain adaptation method and system

Abstract

The application discloses a low-rank field self-adaptive method based on multi-view, comprising the following steps: acquiring source domain sample images and target domain sample images, performing at least two different enhanced mask operations, inputting the masked source domain sample images and target domain sample images and the source domain labels into a student network for training, calculating the loss generated by the training to update the network weight, and obtaining at least two networks; performing multi-view feature extraction on the source domain sample images and the target domain sample images to obtain combined features, capturing the global and local structures of the combined features by constructing a similarity graph and a matrix; integrating the similarity graph and the matrix to obtain a high-order graph, converting the high-order graph into a joint optimization of a target function, optimizing a term to obtain an optimal representation, summing the absolute values of the slices in the high-order graph to obtain an affinity matrix; obtaining a projection matrix by using local preserving projection, projecting to a latent space to obtain projection features and calculate a target function, and performing category prediction on target samples, which is more superior than a single-view method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unsupervised domain adaptation technology of transfer learning, in particular to a low-rank domain adaptation method and system based on multi-view. BACKGROUND

[0002] Deep neural networks have shown considerable progress in various computer vision tasks, significantly enhancing the capabilities of image recognition, object detection, and image generation. These advances have catalyzed transformative applications across various fields, including facial recognition, medical imaging, and autonomous driving. However, despite the strong performance of deep neural networks, it requires a large labeled dataset and often struggles when applied across different domains. Unsupervised domain adaptation addresses these limitations by facilitating knowledge transfer from a labeled source domain to an unlabeled target domain, adjusting the feature distribution to improve performance on the target data without adding labels to the target domain. The field of unsupervised domain adaptation involves adjusting a model trained on a labeled source domain to an unlabeled target domain, with three main approaches: discrepancy-based methods, adversarial learning methods, and self-supervised learning methods. Discrepancy-based methods aim to minimize the distribution discrepancy between the source and target domains by aligning their feature representations, but due to inherent limitations, discrepancy-based methods face challenges in generating explicit features and matching feature distributions between domains near class boundaries. Adversarial training methods involve training a model to learn domain-invariant features, achieving an adversarial goal where a discriminator distinguishes between source and target domain data, while a feature extractor confuses the discriminator, but domain adversarial methods often encounter decision boundary bias towards the source domain, inaccurate classification of target domain samples, leading to overfitting and poor generalization to the target domain. Self-supervised learning methods utilize automatically generated pseudo-labels or pretext tasks within the target domain to learn useful representations, facilitating better alignment and knowledge transfer from the source domain, but self-supervised learning methods struggle to align semantic structures between domains, as instance learning typically captures low-level features without encoding high-level semantic information, in addition, the generation of noisy labels and reliance on pseudo-labels introduce errors that accumulate during training, affecting the model's ability to accurately adapt to the target domain. In the field of unsupervised domain adaptation, inaccurate pseudo-labeling leads to catastrophic error accumulation during the learning process, and the Selective Pseudo-Labeling (SPL) method is a structured prediction-based selective pseudo-labeling strategy that enhances domain adaptation by leveraging structural information in unlabeled target samples. The idea of structured prediction is inspired by the fact that samples in the target domain are well-clustered in the deep feature space, so unsupervised clustering analysis can be used to facilitate accurate pseudo-labeling. Liang et al. proposed Source Hypothesis Transfer (SHOT), which freezes the classifier module of the source model and learns a target-specific feature extraction module by leveraging information maximization and self-supervised pseudo-labeling to implicitly align the representation of the target domain with the source hypothesis. Lukas et al. introduced the MIC module, which most previous unsupervised domain adaptation methods struggle to handle classes with similar visual appearances on the target domain, as there is no underlying truth that can be used to learn subtle appearance differences. To address this issue, unsupervised domain adaptation is enhanced by learning spatial contextual relationships in the target domain, enabling robust visual recognition.Yang et al.proposed a unified framework, i.e., transferable visual transformer (TVT), to fully exploit the portability of ViT for domain adaptation to address the problem that directly using adaptive strategies based on convolutional neural networks cannot fully exploit the role of ViT in knowledge transfer. In addition, by utilizing discriminative clustering to enhance feature diversity and separability, which is destroyed in the adversarial domain alignment process.

[0003] On the basis of the above method, the context-enhanced domain adaptation method based on tensor multi-view low-rank high-order graph learning can solve the problem of distinguishing classes with similar visual attributes. SUMMARY

[0004] In view of the above problems, the present application is proposed.

[0005] Therefore, the technical problem solved by the present application is that the present application solves the problem of distinguishing classes with similar visual attributes.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a multi-view based low-rank domain adaptation method, comprising:

[0008] Obtaining source domain sample images and target domain sample images, performing at least two different enhancement mask operations on the source domain sample images and target domain sample images, inputting the masked source domain sample images and target domain sample images and source domain labels into a student network for training, calculating the loss generated by training using at least two loss functions to update the network weights, and obtaining at least two networks;

[0009] Using the at least two networks to perform multi-view feature extraction on the source domain sample images and target domain sample images to obtain combined features of at least two groups of source domain sample image features and target domain sample image features, and capturing the global and local structures of the combined features by constructing a similarity graph and a matrix;

[0010] Integrating the similarity graph and the matrix to obtain a high-order graph, converting the high-order graph into a joint optimization of a target function, optimizing the terms of the target function to obtain an optimal representation based on the high-order graph, summing the absolute values of the slices in the high-order graph to obtain an affinity matrix;

[0011] Using local preserving projection based on the affinity matrix to obtain a projection matrix, projecting the source domain sample image features and target domain sample image features to a latent space according to the projection matrix to obtain projection features of the source domain sample images and target sample images, calculating a target function for the projection features, and predicting the categories of the target samples according to the target function.

[0012] As a preferred scheme of the low-rank domain adaptation method based on multi-view of the application, wherein: the loss update network weight produced by training is calculated by using at least two loss functions, including:

[0013] The student network learns from the source domain through a supervised loss, and the loss function is expressed as:

[0014]

[0015] Wherein, is a supervised loss, is the jth source domain sample of the student model The probability assigned to the kth class, m is the number of labeled data in the source domain, and C is the total number of samples, is the kth class label of the jth sample .

[0016] Adopting adversarial training, introducing a discriminator combined with a feature extractor and a classifier to reduce the performance caused by the change of each domain distribution data, and obtaining an adversarial loss function expressed as:

[0017]

[0018] Wherein, x s is a source domain sample image, x t is a target domain sample image, S ψ is a combination of a feature extractor and a classifier, D δ is a discriminator, is the expected value of the source domain sample , is the expected value of the target domain sample .

[0019] Introducing a target class alignment and consistency loss, obtaining a class boundary in the target feature space by using the context relationship, and obtaining a loss function expressed as:

[0020]

[0021] Wherein, is the classifier output of the target sample k class, is the target domain sample image x t Introduced augmented mask, is an importance matrix based on entropy adjusting the influence of each sample, ′ and ·k are the feature vectors related to the extracted class k ′ and k, k'' is all classes, C is the total number of samples, and T φ is a teacher network.

[0022] As a preferred scheme of the low-rank multi-view domain adaptation method, the method further comprises:

[0023] The student model training is controlled by an optimization strategy characterized by a minimax game, and a loss function is represented as:

[0024]

[0025] Wherein, ψ, φ are weight parameters of the teacher and student networks, λ adv and λ tcacl are weighting factors.

[0026] As a preferred scheme of the low-rank multi-view domain adaptation method, the method further comprises:

[0027] The local structure is captured by constructing a similarity graph;

[0028] The global structure is obtained by constructing a Laplacian matrix and combining the normalized relationship between data points;

[0029] The Laplacian matrix is represented as:

[0030]

[0031] Wherein, W i is a similarity matrix, D i is a degree matrix of the similarity matrix, is the degree of the lth node, is a semi-inverse matrix of the degree matrix, j is a node, is the degree of the lth row and jth column node in the similarity matrix.

[0032] As a preferred scheme of the low-rank multi-view domain adaptation method, the method further comprises:

[0033] F is updated by fixing all variables in the objective function except F, and is represented as:

[0034]

[0035] All other variables are fixed, and the variable is updated, and the calculation and processing are performed by using ADMM, and is represented as:

[0036]

[0037] Add i=1,2,3, as different The derivative of each variable of the objective function is taken, and the optimal The updated expression is:

[0038]

[0039]

[0040] is the Lagrange multiplier, and the updated Lagrange multiplier is is expressed as:

[0041] where F i is the feature transformation matrix of the i-th view, is the corresponding transpose matrix, and tr(·) is the trace of the matrix, is the i-th slice of the matrix in the third dimension, and L i is the Laplacian matrix of the i-th view, λ is the regularization parameter, ε is the tensor, and ||·|| F is the Frobenius norm, I is the identity matrix, β is the weight parameter, and ||·||2 is the 2-norm, is the i-th slice of the matrix in the first dimension, is the j-th slice of the matrix in the second dimension, and ||·|| * is the kernel norm, and ||·|| 2,1 is the matrix norm, μ is the parameter of the Lagrange multiplier term, σ is the singular value, and U and V are unitary matrices, is the matrix used in the kernel norm update.

[0042] As a preferred solution of the multi-view based low-rank domain adaptation method described in the application, wherein the affinity matrix and the projection matrix include:

[0043] The absolute values of the slices in the high-order graph are summed to obtain the affinity matrix, which is expressed as:

[0044]

[0045] The projection matrix is expressed as:

[0046]

[0047] where diag(·) is a diagonal matrix created from a vector, is the i-th slice of the matrix in the third dimension, is a diagonal matrix, is the sum of the weights of the i-th row of the matrix S, is a Laplace matrix, is a view, Π is a projection matrix, Π T is a corresponding transpose matrix, is a diagonal matrix.

[0048] As a preferred solution of the low-rank domain adaptation method based on multi-view of the application, wherein: the class prediction of the target sample according to the objective function comprises:

[0049] The source domain sample image and the target domain sample image in the view are projected in the latent space, the prototype vector is initialized by using the K-means algorithm, the conditional probability of the class to which the target projection feature belongs to the prototype vector and the class centroid is calculated, the class probability is obtained from the maximum value of the conditional probability, and the class of the combined prediction sample is predicted.

[0050] In a second aspect, the application provides a system for low-rank domain adaptation based on multi-view, comprising:

[0051] A loss calculation module is configured to obtain source domain sample images and target domain sample images, perform at least two different enhanced mask operations on the source domain sample images and the target domain sample images, input the masked source domain sample images and target domain sample images and source domain labels into a student network for training, calculate the loss generated by the training by using at least two loss functions, update the network weights, and obtain at least two networks.

[0052] A feature extraction module is configured to perform multi-view feature extraction on the source domain sample images and the target domain sample images by using the at least two networks, obtain combined features of at least two groups of source domain sample image features and target domain sample image features, and capture the global and local structures of the combined features by constructing a similarity graph and a matrix.

[0053] An integration module is configured to integrate the similarity graph and the matrix to obtain a high-order graph, convert the high-order graph into a joint optimization of an objective function, optimize the terms of the objective function to obtain an optimal representation based on the high-order graph, and sum the absolute values of the slices in the high-order graph to obtain an affinity matrix.

[0054] A prediction module is configured to obtain a projection matrix by using a local preserving projection based on the affinity matrix, project the source domain sample image features and the target domain sample image features into a latent space according to the projection matrix to obtain projection features of the source domain sample images and the target sample images, calculate an objective function for the projection features, and perform class prediction of the target sample according to the objective function.

[0055] In a third aspect, the application provides a computing device, comprising:

[0056] a memory and a processor;

[0057] The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the low-rank domain adaptation method based on multiple views.

[0058] In a fourth aspect, the present application provides a computer readable storage medium storing computer executable instructions, and the computer executable instructions, when executed by a processor, implement the steps of the low-rank domain adaptation method based on multiple views.

[0059] The low-rank domain adaptation method based on multiple views has the advantages of superior domain alignment and enhanced model generalization ability, and is superior to the current most advanced method in various unsupervised domain adaptation tasks, and has strong adaptability and effectiveness in dealing with complex domain adaptation challenges. By effectively utilizing the complementary advantages of class separability and view consistency, the classification performance is significantly improved. The consistency improvement of different data sets emphasizes the robustness and generalization of the method. At the same time, it is found that the average accuracy of ViT backbone network in various methods including the present method is significantly improved, which shows the potential of ViT backbone in improving the performance of unsupervised domain adaptation. In addition, it shows robustness to hyperparameter changes and indicates that the multi-view method is superior to the single-view method in improving classification accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor. Among them:

[0061] Figure 1 The present application provides a low-rank domain adaptation method based on multiple views. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0063] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0064] It should also be noted that, as used herein, "one embodiment" or "an embodiment" refers to a particular aspect of the present application, which can include all options or modifications that are compatible with this particular aspect, including expanded or restrictive options. Moreover, as used herein, "exemplary" or "illustrative" means "serving as an example, instance, or illustration," and should not necessarily be construed as preferred or advantageous over other embodiments.

[0065] The present application is described in detail below in connection with the attached drawings, which are meant to be exemplary and not limiting. In the description below, like reference numerals are used to describe the same, similar or corresponding components throughout the several views of the drawings, in which:

[0066] In the description of the present application, it should be noted that the terms "upper and lower, inside and outside" and the like indicate the positional or directional relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore should not be construed as limiting the present application. In addition, the terms "first, second or third" are only for descriptive purposes and should not be construed as indicating or implying relative importance.

[0067] Unless otherwise defined and limited in the specification, the terms "mounting, connecting, and connecting" should be broadly understood, for example: it can be fixedly connected, detachably connected or integrally connected; it can also be mechanically connected, electrically connected or directly connected; it can also be indirectly connected through an intermediate medium; it can also be the internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0068] Embodiment 1

[0069] Reference Figure 1 For an embodiment of the present application, a low-rank domain adaptive method based on multi-view is provided, comprising:

[0070] S100: Obtain source domain sample images and target domain sample images, perform at least two different enhancement mask operations on the source domain sample images and the target domain sample images, input the masked source domain sample images and target domain sample images and the source domain label into the student network for training, calculate the loss generated by the training using at least two loss functions to update the network weight, and obtain at least two networks;

[0071] In the embodiment of the present application, the student network learns from the source domain through a supervised loss, and the loss function is represented as:

[0072]

[0073] wherein, is a supervised loss, is the jth source domain sample of the student model is the probability assigned to the kth class, m is the number of labeled data in the source domain, and C is the total number of samples, is the kth class label of the jth sample .

[0074] Adversarial training is adopted, and a combination of a discriminator and a feature extractor and a classifier is introduced to reduce the performance caused by the change of the distribution of data in each field, and the obtained adversarial loss function is represented as:

[0075]

[0076] wherein, x s is a source domain sample image, x t is a target domain sample image, S ψ is a combination of a feature extractor and a classifier, D δ is a discriminator, is an expected value of the source domain sample , is an expected value of the target domain sample .

[0077] Target class alignment and consistency loss are introduced, and the class boundary is obtained in the target feature space by utilizing the context relationship, and the obtained loss function is represented as:

[0078]

[0079] wherein, is the classifier output of the target sample kth class, is the target domain sample image x t introduced augmented mask, is an importance matrix based on entropy to adjust the influence of each sample, ·k ′ and ·k are the feature vectors related to the extracted kth class k ′ and k, k" is all classes, C is the total number of samples, and T φ is a teacher network.

[0080] Specifically, the feature space is regarded as a d-dimensional Euclidean space, which is represented as An output space representing a set of labels is defined, denoted as Y = *1,2,...,C+; the source domain Ds The m labeled data Target domain D t The n unlabeled data Set the samples from different domains share a common output space.

[0081] Generate three different feature spaces using different augmentation mask techniques, at least two different mask operations can be color jitter, Gaussian blur, random erasing, etc., through random mask image blocks, the model identifies the context clues;

[0082] The image processed by the mask is transmitted to the student model for training, and the pseudo-label predicted by the exponential moving average teacher model is used, in the training process, the feature extraction part is divided into three modules, namely the teacher network, the student network and the discriminator, and a suitable loss function needs to be selected to calculate the loss generated by the training;

[0083] The student network is trained on the source domain, and the supervised source loss is used to calculate the cross entropy between the predicted label and the real label, which can be used to evaluate the prediction confidence of the ith sample, and the greater the value, the greater the confidence of the model in predicting the sample.

[0084] By adopting adversarial training, the discriminator is introduced into the combination of the feature extractor and the classifier to reduce the performance reduction caused by the change of the data distribution of each field, wherein the target of the student network is to minimize the domain difference, and the target of the discriminator is to maximize its ability to distinguish the domain.

[0085] The application preferably uses target class alignment and consistency loss (TCACL), and W is an importance matrix for adjusting the influence of each sample based on entropy, and W adjusts the weight of each output in the calculation process to enhance the consistency of the class distribution, that is, the weighted classifier output between different classes k and k ′ is standardized and compared to promote clearer class distinction. The MCC loss function can also be used.

[0086] It should be noted that the MCC loss function reduces cross-class confusion less than the target class alignment and consistency loss, integrates the pair-wise class confusion between the source classifier and the target classifier into the MCC loss function, and gives higher weights to the samples that the model is more confident about. The target class alignment and consistency loss also introduce various augmented masks for the target image, the teacher network generates pseudo-labels for the target samples and compares them with the predictions of the masked images, the teacher network uses the EMA method to update its weights, which can better reduce cross-class confusion and achieve more obvious and separable class boundaries.

[0087] In the embodiment of the application, the student model training is controlled by an optimization strategy characterized by minimax game, and the loss function is represented as:

[0088]

[0089] wherein ψ, φ are the weight parameters of the teacher and student networks, λ adv and λ tcacl are the weighting factors.

[0090] Specifically, the weight parameters representing the teacher and student networks are respectively fine-tuned to minimize the overall loss, the parameters of the discriminator are denoted as δ, the adversarial loss is adjusted to be maximum, and the weighting factor is used to control the influence of the adversarial loss and the target class alignment and consistency loss.

[0091] It should be noted that by minimizing the loss of the student model and maximizing the loss of the discriminator, balanced optimization in the model training process can be achieved, the generalization ability of the model is improved, the adversarial loss is introduced, the robustness of the model to input data can be enhanced through the adversarial process with the discriminator, the weight parameters of the teacher and student networks can be adjusted to more finely control the learning process of the model, and the weighting factor can be used to flexibly control the proportion of different loss terms in the total loss to adapt to different training requirements.

[0092] S200: performing multi-view feature extraction on the source domain sample images and the target domain sample images using at least two networks to obtain combined features of at least two groups of source domain sample image features and target domain sample image features, and capturing global and local structures of the combined features by constructing a similarity graph and a matrix;

[0093] In the embodiments of the present application, the local structure is captured by constructing a similarity graph.

[0094] The global structure is obtained by constructing a Laplacian matrix and merging the normalized relationship between data points.

[0095] The Laplacian matrix is represented as:

[0096]

[0097] wherein W i is a similarity matrix, D i is a degree matrix of the similarity matrix, is the degree of the lth node, is a semi-inverse matrix of the degree matrix, j is a node, is the degree of the lth row and jth column node in the similarity matrix.

[0098] Specifically, the model is trained using various augmented mask techniques, features are extracted from the source domain and the target domain, thereby generating a combined feature space, a similarity graph is constructed to capture the local structure, and a Laplacian matrix is constructed to obtain the global structure of the data by merging the normalized relationship between data points.

[0099] It should be noted that by the combined use of the similarity graph and the Laplacian matrix, the model can simultaneously consider the local neighborhood structure and the global structure of the data, improving the recognition ability of complex data patterns. The acquisition of normalized relationships and global structures helps to optimize the performance of the model on specific tasks. In particular, in the case of distribution differences between the source domain and the target domain, the adversarial training enhances the robustness of the model to input perturbations. Through the adversarial process with the discriminator, the model can learn more robust feature representations.

[0100] S300: Integrate the similarity graph and the matrix to obtain a high-order graph, convert the high-order graph into a joint optimization of the objective function, optimize the terms of the objective function to obtain an optimal representation based on the high-order graph, and sum the absolute values of the slices in the high-order graph to obtain an affinity matrix.

[0101] In the embodiments of the present application, all variables in the objective function except F are fixed to update F, denoted as:

[0102]

[0103] Fix all other variables and update the variable using ADMM for calculation and processing, denoted as:

[0104]

[0105] Add i = 1, 2, 3, as different variables of the objective function are differentiated, and the optimal The updated expression is:

[0106]

[0107] is the Lagrange multiplier, and the updated Lagrange multiplier is denoted as:

[0108] where F i is the feature transformation matrix of the i-th view, is the corresponding transpose matrix, tr(〃) is the trace of the matrix, is the i-th slice of the matrix in the third dimension, L i is the Laplacian matrix of the i-th view, λ is the regularization parameter, ε is the tensor, and ||·|| F is the Frobenius norm, I is the identity matrix, β is the weight parameter, and ||·||2 is the 2-norm, is the i-th slice of the matrix in the first dimension, is the j-th slice of matrix in the second dimension, ||·|| is the Frobenius norm, * is the nuclear norm, ||·|| is the Frobenius norm, 2,1 is the matrix norm, μ is the parameter of the Lagrange multiplier term, σ is the singular value, U and V are the unitary matrices, is the matrix used in the nuclear norm update.

[0109] Specifically, instead of

[0110] Specifically, the high-order graph is learned to achieve the inter-view consistency and inter-class separability and the graph captures the structure of the tensor , where the tensor ε is the superposition of similar graphs constructed by the normalized spectral embedding matrices, the high-order learning form is converted into the subsequent joint optimization problem covering all v views of , and the objective function is expressed as:

[0111]

[0112] where the first term maximizes the sum of the trajectories L i F i of all views, effectively enhancing the representation of data points to ensure that they are well separated according to the graph structure. The constraint L i F i ensures that the feature transformation is standard orthogonal, maintains the distance and maintains the effective orthogonal basis of the transformed features. The second term minimizes the difference between the stacked tensor and the learned graph , and λ is the regularization parameter controlling the weight of the term . The third term is used to improve the discriminative performance of multi-view clustering, where represents all elements along the first and second dimensions at the v-th index of the third dimension, and by using the nuclear norm on , the sum of the singular values of the contribution matrix is clustered to similar instances while separating dissimilar instances. The bidirectional relationship between the nodes of the adjacency matrix of the undirected graph is symmetric, so the constraint ensures that all horizontal slices are symmetric. The fourth term promotes the low-rank and sparse structure in , which is also subject to the symmetry constraint, where represents ​the jth vertical slice of the jth view of the jth high-order graph. The weight parameter β is used to adjust the balance of the competition between the third and fourth terms of the objective function.

[0113] It should be noted that by jointly optimizing these terms, an optimal representation based on high-order graphs is obtained, thereby improving the domain alignment performance.

[0114] S400: Obtain a projection matrix based on the affinity matrix using local preserving projection, project the source domain sample image features and the target domain sample image features into the latent space according to the projection matrix, obtain the projection features of the source domain sample image and the target sample image, calculate an objective function based on the projection features, and perform class prediction on the target sample according to the objective function;

[0115] In the embodiments of the present application, the absolute values of the slices in the high-order graph are summed to obtain an affinity matrix, which is represented as:

[0116]

[0117] The projection matrix is obtained by using LPP to project the formal objective function, and is represented as:

[0118]

[0119] wherein diag(·) is a diagonal matrix created from a vector, is the ith slice of the matrix in the third dimension, is a diagonal matrix, is the sum of the weights of the ith row of the matrix S, is a Laplacian matrix, is a view, Π is a projection matrix, and Π T is a corresponding transpose matrix, is a diagonal matrix.

[0120] Specifically, after obtaining the high-order graph, the absolute values of the slices are summed to obtain an affinity matrix that integrates multiple view information and captures comprehensive relationships, and then an LPP is used to project the formal objective function to determine the projection matrix, so as to ensure that the projected samples in the same class are closely clustered.

[0121] It should be noted that in the calculation of the projection matrix, the penalty function Π T is incorporated into the objective function to prevent the projection matrix from assuming extreme values, thereby improving the stability and generalization of the projection matrix.

[0122] In the embodiments of the present application, the source domain sample images and the target domain sample images in the view are projected in the latent space, the prototype vectors are initialized by using the K-means algorithm, the conditional probability of the target projection feature belonging to the class of the prototype vector and the class centroid is calculated, the class probability is obtained from the maximum value of the conditional probability, and the class of the combined prediction sample is predicted.

[0123] Specifically, after the source domain sample images and the target domain sample images in the view are projected in the latent space, the distance between each target domain sample and the prototype vector of the source domain sample, and the center point of the target domain sample image are used for class prediction.

[0124] Let be the projection feature space of view i, denoted as The space contains the combined projection feature and the projection target feature The prototype of the class of view i is denoted as:

[0125]

[0126] K-means clustering is performed on to utilize the structural information of the target samples. The objective function of K-means aims to reduce the sum of squared distances between each sample and the center point to which it is assigned, with the constraint that each sample can only belong to one cluster; each cluster must contain at least one sample, and the cluster center of the class is denoted as

[0127] For each feature extracted from different views i∈{1,…,v+}, the conditional probability of the target projection feature belonging to the class y of the prototype vector and the class centroid is calculated and denoted as:

[0128]

[0129] wherein, is the projection feature space of the source domain view i, is the prototype feature vector of the class y in the source domain view i, is all the projection feature vectors of the class y in the source domain view j, is the value counted by all the samples of the class y in the source domain view j, is the projection feature of the target domain view i, is the prototype projection feature vector of the class y in the source domain view i, is the prototype projection feature vector of the class y in the target domain view i; based on the three views and The class probability is obtained by taking the maximum value of the conditional probability.

[0130] It should be noted that by considering the conditional probability, the model can more accurately predict the class of the sample, and initializing the prototype vector using the K-means algorithm helps to discover the inherent structure of the data, thereby improving the generalization ability of the model to new data. In a class-imbalanced dataset, by using methods such as weighted sampling, the minority class samples can be ensured to be adequately represented, thereby improving the recognition ability of the classifier for the minority class.

[0131] The above is a schematic scheme of the multi-view based low-rank domain adaptation method of the embodiment. It should be noted that the technical scheme of the multi-view based low-rank domain adaptation device belongs to the same concept as the technical scheme of the multi-view based low-rank domain adaptation method described above. The technical scheme of the multi-view based low-rank domain adaptation device in the embodiment is not described in detail, and can be referred to the description of the technical scheme of the multi-view based low-rank domain adaptation method described above.

[0132] The multi-view based low-rank domain adaptation device in the embodiment comprises:

[0133] The loss calculation module is configured to obtain source domain sample images and target domain sample images, perform at least two different enhancement mask operations on the source domain sample images and the target domain sample images, input the masked source domain sample images and target domain sample images and the source domain labels into the student network for training, calculate the loss generated by the training using at least two loss functions to update the network weights, and obtain at least two networks.

[0134] The feature extraction module is configured to perform multi-view feature extraction on the source domain sample images and the target domain sample images using the at least two networks to obtain combined features of at least two groups of source domain sample image features and target domain sample image features, and capture the global and local structures of the combined features by constructing a similarity graph and a matrix.

[0135] The integration module is configured to integrate the similarity graph and the matrix to obtain a high-order graph, convert the high-order graph into a joint optimization of a target function, optimize the terms of the target function to obtain an optimal representation based on the high-order graph, and sum the absolute values of the slices in the high-order graph to obtain an affinity matrix.

[0136] The prediction module is configured to obtain a projection matrix based on the affinity matrix using local preservation projection, project the source domain sample image features and the target domain sample image features into a latent space according to the projection matrix to obtain projection features of the source domain sample images and the target sample images, calculate a target function based on the projection features, and predict the class of the target sample based on the target function.

[0137] The embodiment also provides a computing device suitable for multi-view based low-rank domain adaptation, comprising:

[0138] a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the multi-view based low-rank domain adaptation method proposed in the above embodiment.

[0139] The embodiment also provides a storage medium having a computer program stored thereon, and the computer program is executed by a processor to realize the multi-view based low-rank domain adaptation method proposed in the above embodiment.

[0140] The storage medium proposed in the embodiment and the multi-view based low-rank domain adaptation method proposed in the above embodiment belong to the same inventive concept, and the technical details not described in the embodiment can be referred to the above embodiment, and the embodiment has the same beneficial effects as the above embodiment.

[0141] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary universal hardware, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a FLASH, a hard disk or an optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of various embodiments of the present application.

[0142] Embodiment 2

[0143] Referring to Table 1, for an embodiment of the present application, a multi-view based low-rank domain adaptation method is provided, and the technical effects used in the method are verified and explained.

[0144] Taking the data set Office-Home as an example, 15588 pictures are used, and 65 categories are included;

[0145] Firstly, three mask operations are performed on the source sample pictures, which are: color jitter combined with blur operation, denoted as j b ; color jitter combined with gray mask operation, denoted as j g ; color jitter combined with random erasing, denoted as j e .

[0146] The source sample, the target sample without masking, and the target sample with masking are input into the student network training to obtain a supervised loss L src The target image without masking is input into the teacher network training to obtain a TCACL loss L tcacl The pseudo-label predicted by the teacher network is updated using the exponential moving average of the results of the student model training, so that the teacher network can generate robust pseudo-labels using context and local information and compare them with the prediction of the masked image. After training is completed, features are extracted from three independent feature spaces.

[0147] After obtaining the feature space, a similarity graph is constructed to capture the local structure of the data points. After obtaining the high-order graph, the constructed affinity matrix integrates the information of multiple views and captures the overall relationship. Then, the matrix is applied to local preservation projection to obtain the internal structure information of the subspace, and low-rank subspace learning is applied to align the conditional distributions of the source sample and the target sample under different views.

[0148] Accuracy of unsupervised domain adaptation classification using ResNet-50 and ViT features of Office-Home as shown in Table 1

[0149]

[0150] Where A is Art, C is Clipart, P is Product, and R is Real-World, representing four different domains, such as A->C, the model of domain A performs unsupervised classification on data in domain C through domain adaptation methods.

[0151] As shown in the table, under the ResNet-50 backbone network, MLRGL achieves 63.7% and 88.3% on A->C and P->R, respectively, exceeding the 57.1% and 82.2% of other methods such as SHOT. In the ViT backbone scenario, MLRGL performs well on tasks such as A->P and R->P, achieving 90.9% and 93.0%, respectively, better than TVT's 85.6% on A->P and MIC's 91.4% on R->P. In addition, the overall average accuracy of the method is 1.2% higher than MIC's 86.2%.

[0152] It shows excellent performance in different domain adaptation scenarios, further consolidating its advantages in unsupervised domain adaptation tasks.

[0153] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced, without departing from the spirit and scope of the technical solutions of the present application, and all of them should be covered in the scope of the claims of the present application.

Claims

1. A low-rank neighborhood adaptive method based on multiple views, characterized in that, include: Obtain source domain sample images and target domain sample images, perform at least two different enhancement masking operations on the source domain sample images and target domain sample images, input the masked source domain sample images and target domain sample images and source domain labels into the student network for training, calculate the training loss using at least two loss functions to update the network weights, and obtain at least two networks. The source domain sample image and the target domain sample image are extracted using the at least two networks to obtain at least two sets of combined features of source domain sample image features and target domain sample image features. The global and local structures of the combined features are captured by constructing similarity graphs and matrices. The similarity graph and matrix are integrated to obtain a higher-order graph. The higher-order graph is then converted into a joint optimization of the objective function. The terms of the objective function are optimized to obtain the optimal representation based on the higher-order graph. The absolute values ​​of the slices in the higher-order graph are summed to obtain the affinity matrix. Based on the affinity matrix, a projection matrix is ​​obtained by local preservation projection. The source domain sample image features and target domain sample image features are projected onto the latent space according to the projection matrix to obtain the projection features of the source domain sample image and the target sample image. An objective function is calculated on the projection features, and the category prediction of the target sample is performed according to the objective function.

2. The low-rank neighborhood adaptive method based on multiple views as described in claim 1, characterized in that, Updating network weights by calculating the training loss using at least two loss functions includes: The student network learns from the source domain using supervised loss, the loss function of which is expressed as: in, To monitor losses, For the j-th source domain sample of the student model The probability assigned to class k, where m is the number of labeled data in the source domain, and C is the total number of samples. For the j-th sample The tag of the kth class; Adversarial training is employed, introducing a combination of discriminator, feature extractor, and classifier to mitigate performance issues caused by variations in domain-specific data distribution. The resulting adversarial loss function is expressed as: Where, x s For the source domain sample image, x t For the target domain sample image, S ψ D is a combination of feature extractor and classifier. δ For the discriminator, For source domain samples Expected value For target domain samples Expected value; By introducing target class alignment and consistency loss, and utilizing contextual relationships to obtain class boundaries within the target feature space, the resulting loss function is expressed as: in, The output of the classifier for the target sample of class k is... For the target domain sample image x t The introduced augmentation mask, To adjust the importance matrix of each sample's influence based on entropy, ·k ′ And ·k represents the extracted category k. ′ The feature vectors related to k, where k″ represents all classes, C is the total number of samples, and T is the total number of samples. φ For teachers' networks.

3. The low-rank neighborhood adaptive method based on multiple views as described in claim 1 or 2, characterized in that, Also includes: The student model training is controlled by an optimization strategy characterized by a mini-maximum game, and the loss function is expressed as: Where ψ,φ are the weight parameters of the teacher and student networks, and λ adv and λ tcacl This is the weighting factor.

4. The low-rank neighborhood adaptive method based on multiple views as described in claim 3, characterized in that, The global and local structures of combined features are captured by constructing similarity graphs and matrices, including: Capture local structure by constructing similarity graphs; The global structure is obtained by constructing the Laplace matrix and merging the normalized relationships between data points; The Laplace matrix is ​​expressed as: Among them, W i Let D be a similarity matrix. i Let be the degree matrix of the similarity matrix. Let l be the degree of the l-th node. Let j be the semi-inverse of the degree matrix, and j be the node. Let be the degree of the node in the l-th row and j-th column of the similarity matrix.

5. The low-rank neighborhood adaptive method based on multiple views as described in claim 4, characterized in that, Optimizing the terms of the objective function to obtain the optimal representation based on a higher-order graph includes: To update F while fixing all variables in the objective function except F, it can be expressed as: Fix all other variables and apply the variable The update is performed using ADMM calculations, as shown below: Add to As different By taking the derivative of the objective function with respect to each variable, the optimal value is obtained by minimizing the given objective function. The updated expression is: For Lagrange multipliers, updated Lagrange multipliers Represented as: Among them, F i Let be the feature transformation matrix of the i-th view. Let tr(·) be the transpose of the matrix, and tr(·) be the trace of the matrix. Let L be the i-th slice of the matrix along the third dimension. i Let be the Laplacian matrix of the i-th view, λ be the regularization parameter, and v be the tensor, ||·|| F Let I be the Frobenius norm, β be the identity matrix, β be the weight parameter, and ||·||2 be the 2-norm. This represents the i-th slice of the matrix along its first dimension. For the j-th slice of the matrix along the second dimension, ||·|| * For the nuclear norm, ||·|| 2,1 Let l2 be the l2-l1-l1 norm of the matrix, μ be the parameter of the Lagrange multiplier, σ be the singular value, and U and V be unitary matrices. This is the matrix used in the nuclear norm update.

6. The low-rank neighborhood adaptive method based on multiple views as described in claim 5, characterized in that, The affinity matrix and projection matrix include: For higher-order graphs Summing the absolute values ​​of the slices in the matrix yields the affinity matrix, represented as: Using the LPP projection to formalize the objective function, the projection matrix is ​​represented as follows: Where, diag(·) creates a diagonal matrix from a vector. This represents the i-th slice of the matrix along its third dimension. It is a diagonal matrix. The sum of the weights in the i-th row of matrix S. For Laplace matrix, Π is the view, and Π is the projection matrix. T This is the corresponding transpose matrix. It is a diagonal matrix.

7. The low-rank neighborhood adaptive method based on multiple views as described in claim 6, characterized in that, Predicting the category of a target sample based on the objective function includes: The source domain sample image and the target domain sample image in the view are projected into the latent space. The prototype vector is initialized using the K-means algorithm. The conditional probability of the target projected feature belonging to the category of the prototype vector and the category centroid is calculated. The category probability is obtained from the maximum value of the conditional probability and then merged to predict the category of the sample.

8. A system based on a low-rank neighborhood adaptive method using multiple views, characterized in that, include: The loss calculation module is used to acquire source domain sample images and target domain sample images, perform at least two different enhancement masking operations on the source domain sample images and target domain sample images, input the masked source domain sample images and target domain sample images and source domain labels into the student network for training, calculate the training loss using at least two loss functions to update the network weights, and obtain at least two networks. The feature extraction module is used to perform multi-view feature extraction on the source domain sample image and the target domain sample image using the at least two networks, to obtain at least two sets of combined features of source domain sample image features and target domain sample image features, and to capture the global and local structure of the combined features by constructing similarity maps and matrices; An integration module is used to integrate the similarity graph and the matrix to obtain a higher-order graph, convert the higher-order graph into a joint optimization of the objective function, optimize the terms of the objective function to obtain the optimal representation based on the higher-order graph, and sum the absolute values ​​of the slices in the higher-order graph to obtain the affinity matrix. The prediction module is used to obtain a projection matrix based on the affinity matrix using local preservation projection, project the source domain sample image features and target domain sample image features onto the latent space according to the projection matrix to obtain the projection features of the source domain sample image and the target sample image, calculate the objective function on the projection features, and predict the category of the target sample according to the objective function.

9. An electronic device, characterized in that, The device includes: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images