Multi-view-based low-rank domain self-adaption method and system

Abstract

The invention discloses a multi-view-based low-rank domain self-adaption method, which comprises the following steps of: acquiring a source domain sample image and a target domain sample image, performing at least two different mask enhancement operations, transmitting the masked source domain sample image, the masked target domain sample image and a source domain label into a student network for training, and obtaining a multi-view-based low-rank domain self-adaption model; calculating loss generated by training and updating network weight to obtain at least two networks; performing multi-view feature extraction on the source domain sample image and the target domain sample image to obtain combined features, and capturing global and local structures of the combined features by constructing a similar graph and a matrix; integrating the similar graph and the matrix to obtain a high-order graph, converting the high-order graph into joint optimization of a target function, optimizing items to obtain optimal representation, and summing absolute values of slices in the high-order graph to obtain an affinity matrix; a projection matrix is obtained through local preserving projection and projected to a potential space, projection features are obtained, a target function is calculated, category prediction is conducted on a target sample, and compared with a single-view method, the method has more superiority.

Description

Technical Field

[0001] The present invention relates to the technical field of unsupervised domain adaptation of transfer learning, and specifically to a multi-view based low-rank domain adaptation method and system. Background Art

[0002] Deep neural networks have demonstrated considerable progress in various computer vision tasks, significantly enhancing the capabilities of image recognition, object detection, and image generation. These advances have catalyzed transformative applications in various fields, including facial recognition, medical imaging, and autonomous driving. However, despite their powerful performance, deep neural networks require large labeled datasets and often have problems when applied across different domains. Unsupervised domain adaptation addresses these limitations by facilitating the transfer of knowledge from a labeled source domain to an unlabeled target domain, adjusting the feature distribution to improve performance on the target data without adding labels to the target domain. The field of unsupervised domain adaptation involves adapting a model trained on a labeled source domain to an unlabeled target domain, and there are three main approaches: discrepancy-based methods, adversarial learning methods, and self-supervised learning methods. Discrepancy-based methods aim to minimize the distribution difference between the source and target domains by aligning their feature representations, but due to inherent limitations, discrepancy-based methods face challenges in generating explicit features near class boundaries and fully matching feature distributions between domains. Adversarial training methods involve training a model to learn domain-invariant features to achieve the adversarial goal, where the discriminator distinguishes between source and target domain data, while the feature extractor confuses the discriminator. However, domain adversarial methods often encounter decision boundary deviations for the source domain and inaccurate classification of samples in the target domain, leading to overfitting and poor generalization to the target domain. Self-supervised learning methods use pseudo-labels or pretext tasks automatically generated in the target domain to learn useful representations, promoting better alignment and knowledge transfer from the source domain. However, it is difficult for self-supervised learning methods to align semantic structures between domains because instance learning usually captures low-level features and cannot encode high-level semantic information. In addition, the generation of noisy labels and the reliance on pseudo-labels introduce errors that accumulate during training, affecting the model's ability to accurately adapt to the target domain. In the field of unsupervised domain adaptation, inaccurate pseudo-labeling can lead to catastrophic error accumulation during learning. The Selective Pseudo-Labeling (SPL) method is a selective pseudo-labeling strategy based on structured prediction that enhances domain adaptation by leveraging structural information in unlabeled target samples. The idea of ​​structured prediction is inspired by the fact that samples in the target domain are well clustered in the deep feature space, so unsupervised clustering analysis can be used to promote accurate pseudo-labeling. Liang et al. proposed Source Hypothesis Transfer (SHOT), which freezes the classifier module of the source model and learns a target-specific feature extraction module by leveraging information maximization and self-supervised pseudo-labeling to implicitly align the representation of the target domain with the source hypothesis. Lukas et al. introduced the MIC module. Most previous unsupervised domain adaptation methods have difficulty handling classes with similar visual appearance on the target domain because there is no ground truth available to learn slight appearance differences. To address this issue, unsupervised domain adaptation is enhanced by learning spatial contextual relationships in the target domain to achieve robust visual recognition.Yang et al. proposed a unified framework, namely Transferable Visual Transformer (TVT), to fully exploit the portability of ViT for domain adaptation to address the problem that directly using convolutional neural network-based adaptation strategies cannot fully utilize the role of ViT in knowledge transfer. In addition, discriminative clustering is utilized to enhance feature diversity and separability, which are destroyed in the adversarial domain alignment process.

[0003] Based on the above methods, the context-enhanced domain adaptation method based on tensor multi-view low-rank high-order graph learning can solve the problem of distinguishing categories with similar visual attributes. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: the present invention solves the problem of distinguishing categories with similar visual attributes.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a multi-view based low-rank domain adaptation method, comprising:

[0008] Obtaining a source domain sample image and a target domain sample image, performing at least two different enhancement mask operations on the source domain sample image and the target domain sample image, passing the masked source domain sample image and the target domain sample image and the source domain label into a student network for training, and using at least two loss functions to calculate the loss generated by the training to update the network weights, thereby obtaining at least two networks;

[0009] Using the at least two networks to perform multi-view feature extraction on the source domain sample image and the target domain sample image, obtain at least two groups of combined features of the source domain sample image features and the target domain sample image features, and capture the global and local structures of the combined features by constructing a similarity graph and a matrix;

[0010] Integrating the similarity graph and the matrix to obtain a high-order graph, converting the high-order graph into a joint optimization of an objective function, optimizing the terms of the objective function to obtain an optimal representation based on the high-order graph, and summing the absolute values ​​of the slices in the high-order graph to obtain an affinity matrix;

[0011] A projection matrix is ​​obtained by using local preservation projection based on the affinity matrix, and the source domain sample image features and the target domain sample image features are projected into the latent space according to the projection matrix to obtain the projection features of the source domain sample image and the target sample image, and an objective function is calculated for the projection features, and the category of the target sample is predicted according to the objective function.

[0012] As a preferred solution of the multi-view based low-rank domain adaptation method of the present invention, wherein: using at least two loss functions to calculate the loss generated by training to update the network weights includes:

[0013] The student network learns from the source domain through supervision loss, and the loss function is expressed as:

[0014]

[0015] in, To monitor losses, is the jth source domain sample of the student model The probability assigned to class k, m is the number of labeled data in the source domain, C is the total number of samples, is the jth sample The label of the kth category;

[0016] Using adversarial training, we introduce a combination of discriminator, feature extractor and classifier to reduce the performance caused by changes in the distribution data in various fields, and obtain the adversarial loss function expressed as:

[0017]

[0018] Among them, x s is the source domain sample image, x t is the sample image in the target domain, S ψ is the combination of feature extractor and classifier, D δ is the discriminator, The source domain sample The expected value of The target domain sample Expected value;

[0019] Introducing target class alignment and consistency loss, by utilizing contextual relationships to obtain class boundaries in the target feature space, the resulting loss function is expressed as:

[0020]

[0021] in, is the classifier output of target sample k, is the target domain sample image x t The augmentation mask introduced, is the importance matrix for adjusting the influence of each sample based on entropy, k ′ and k is the extracted category k ′ The feature vector associated with k, k″ is all categories, C is the total number of samples, T φ Network for teachers.

[0022] As a preferred solution of the multi-view based low-rank domain adaptation method of the present invention, it also includes:

[0023] The student model training is controlled by an optimization strategy characterized by a min-max game, and the loss function is expressed as:

[0024]

[0025] Among them, ψ, φ are the weight parameters of the teacher and student networks, λ adv and λ tcacl is the weighting factor.

[0026] As a preferred solution of the multi-view based low-rank domain adaptation method of the present invention, the global and local structures of the combined features are captured by constructing similarity graphs and matrices, including:

[0027] Capturing local structure by constructing similarity graphs;

[0028] The global structure is obtained by constructing the Laplacian matrix and incorporating the normalized relationships between data points;

[0029] The Laplacian matrix is ​​expressed as:

[0030]

[0031] Among them, W i is the similarity matrix, D i is the degree matrix of the similarity matrix, is the degree of the lth node, is the semi-inverse matrix of the degree matrix, j is the node, is the degree of the node in the lth row and jth column in the similarity matrix.

[0032] As a preferred solution of the multi-view based low-rank domain adaptation method described in the present invention, wherein: optimizing the terms of the objective function to obtain the optimal representation based on the high-order graph includes:

[0033] Fix all variables except F in the objective function to update F, expressed as:

[0034]

[0035] Fix all other variables and Update, and use ADMM to calculate and process as follows:

[0036]

[0037] Add to i=1,2,3, as different Derivative each variable of the objective function, and obtain the optimal solution by minimizing the given objective function. The updated expression is:

[0038]

[0039]

[0040] is the Lagrange multiplier, and the updated Lagrange multiplier is It is expressed as:

[0041] Among them, F i is the feature transformation matrix of the i-th view, is the corresponding transposed matrix, tr(〃) is the trace of the matrix, is the i-th slice of the matrix in the third dimension, L i is the Laplacian matrix of the i-th view, λ is the regularization parameter, ε is a tensor, ∥·∥ F is the Frobenius norm, I is the identity matrix, β is the weight parameter, ∥·∥2 is the 2-norm, is the i-th slice of the matrix along the first dimension, is the j-th slice of the matrix in the second dimension, ∥·∥ * is the nuclear norm, ∥·∥ 2,1 For the matrix norm, μ is the parameter of the Lagrange multiplier term, σ is the singular value, U and V are unitary matrices, is the matrix used in the nuclear norm update.

[0042] As a preferred solution of the multi-view based low-rank domain adaptation method of the present invention, the affinity matrix and the projection matrix include:

[0043] For high-order graphs The absolute values ​​of the slices in are summed to obtain the affinity matrix represented as:

[0044]

[0045] Using LPP projection to formalize the objective function, the projection matrix is ​​expressed as:

[0046]

[0047] Where diag(·) creates a diagonal matrix from a vector. is the i-th slice of the matrix in the third dimension, is a diagonal matrix, is the sum of the weights of the i-th row of matrix S, is the Laplace matrix, is the view, Π is the projection matrix, Π T is the corresponding transposed matrix, is a diagonal matrix.

[0048] As a preferred solution of the multi-view based low-rank domain adaptation method of the present invention, wherein: predicting the category of the target sample according to the objective function includes:

[0049] The source domain sample image and the target domain sample image in the view are projected into the latent space, the prototype vector is initialized using the K-means algorithm, the conditional probability that the target projection feature belongs to the category of the prototype vector and the category centroid is calculated, the category probability is obtained from the maximum value of the conditional probability, and the category of the merged prediction sample is performed.

[0050] In a second aspect, the present invention provides a system for low-rank domain adaptation based on multi-views, comprising:

[0051] A loss calculation module is used to obtain a source domain sample image and a target domain sample image, perform at least two different enhancement mask operations on the source domain sample image and the target domain sample image, pass the masked source domain sample image and the target domain sample image and the source domain label into the student network for training, and use at least two loss functions to calculate the loss generated by the training to update the network weights, so as to obtain at least two networks;

[0052] a feature extraction module, configured to perform multi-view feature extraction on the source domain sample image and the target domain sample image using the at least two networks, obtain at least two groups of combined features of the source domain sample image features and the target domain sample image features, and capture the global and local structures of the combined features by constructing a similarity graph and a matrix;

[0053] An integration module, used for integrating the similarity graph and the matrix to obtain a high-order graph, converting the high-order graph into a joint optimization of an objective function, optimizing the terms of the objective function to obtain an optimal representation based on the high-order graph, and summing the absolute values ​​of the slices in the high-order graph to obtain an affinity matrix;

[0054] A prediction module is used to obtain a projection matrix based on the affinity matrix using local preservation projection, project the source domain sample image features and the target domain sample image features into a latent space according to the projection matrix, obtain projection features of the source domain sample image and the target sample image, calculate an objective function for the projection features, and perform category prediction on the target sample according to the objective function.

[0055] In a third aspect, the present invention provides a computing device, comprising:

[0056] Memory and processor;

[0057] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the multi-view-based low-rank domain adaptation method are implemented.

[0058] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the multi-view-based low-rank domain adaptation method.

[0059] Beneficial effects of the present invention: The multi-view based low-rank domain adaptation method proposed in the present invention achieves superior domain alignment and enhances the generalization ability of the model. It outperforms the current most advanced methods in various unsupervised domain adaptation tasks and has strong adaptability and effectiveness in dealing with complex domain adaptation challenges. The classification performance is significantly improved by effectively utilizing the complementary advantages of inter-class separability or inter-view consistency. The consistency improvement of different data sets emphasizes the robustness and generalization of the method. At the same time, the experiment found that the average accuracy of the ViT backbone network on various methods including the present method is significantly improved. This trend shows the potential of the ViT backbone in improving the performance of unsupervised domain adaptation. In addition, it shows robustness to hyperparameter changes, and shows that the multi-view method is superior to the single-view method in improving classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0061] Figure 1 An overall flow chart of a multi-view based low-rank domain adaptation method provided by the present invention. DETAILED DESCRIPTION

[0062] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0063] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0064] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0065] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0066] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0067] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0068] Example 1

[0069] Reference Figure 1 , as an embodiment of the present invention, provides a multi-view based low-rank domain adaptation method, comprising:

[0070] S100: obtaining a source domain sample image and a target domain sample image, performing at least two different enhancement mask operations on the source domain sample image and the target domain sample image, passing the masked source domain sample image and the target domain sample image and the source domain label into a student network for training, using at least two loss functions to calculate the loss generated by the training to update the network weights, and obtaining at least two networks;

[0071] In the embodiment of the present application, the student network learns from the source domain through supervision loss, and the loss function is expressed as:

[0072]

[0073] in, To monitor losses, is the jth source domain sample of the student model The probability assigned to class k, m is the number of labeled data in the source domain, C is the total number of samples, is the jth sample The label of the kth category;

[0074] Using adversarial training, we introduce a combination of discriminator, feature extractor and classifier to reduce the performance caused by changes in the distribution data in various fields, and obtain the adversarial loss function expressed as:

[0075]

[0076] Among them, x s is the source domain sample image, x t is the sample image in the target domain, S ψ is the combination of feature extractor and classifier, D δ is the discriminator, The source domain sample The expected value of The target domain sample Expected value;

[0077] Introducing target class alignment and consistency loss, by utilizing contextual relationships to obtain class boundaries in the target feature space, the resulting loss function is expressed as:

[0078]

[0079] in, is the classifier output of target sample k, is the target domain sample image x t The augmentation mask introduced, is the importance matrix for adjusting the influence of each sample based on entropy, k ′ and k is the extracted category k ′ The feature vector associated with k, k″ is all categories, C is the total number of samples, T φ Network for teachers.

[0080] Specifically, the feature space is regarded as a d-dimensional Euclidean space and expressed as Define an output space to represent a set of labels, denoted as Y = *1,2,...,C+; source domain Ds Contains m labeled data Target domain D t Contains n unlabeled data Set samples from different domains to share a common output space.

[0081] Three different feature spaces are generated using different enhancement masking techniques. At least two different masking operations can be color jitter, Gaussian blur, random erasure, etc. By randomly masking image patches, the model is trained to recognize context clues.

[0082] The masked image is transferred to the student model for training, and then the pseudo labels predicted by the teacher model are predicted using the exponential moving average. During the training process, the feature extraction part is divided into three modules, namely the teacher network, the student network and the discriminator. At the same time, a suitable loss function needs to be selected to calculate the loss generated by the training;

[0083] The student network is trained on the source domain, using supervised source loss, and the cross entropy between the predicted label and the true label is calculated. This value can be used to evaluate the prediction confidence of the i-th sample. The larger the value, the greater the confidence of the model in the sample prediction.

[0084] By adopting adversarial training, a combination of a discriminator, a feature extractor and a classifier is introduced to alleviate the performance degradation caused by changes in the distribution of data in various fields. The goal of the student network is to minimize the domain difference, and the goal of the discriminator is to maximize its ability to distinguish domains.

[0085] The preferred target class alignment and consistency loss (TCACL) of the present invention is that W is an importance matrix based on entropy to adjust the influence of each sample. W adjusts the weight of each output during the calculation process to enhance the consistency of class distribution, that is, for different categories k and k ′ The weighted classifier outputs between the two are standardized to promote clearer category distinction. The MCC loss function can also be used.

[0086] It should be noted that the MCC loss function is not as effective as the target class alignment and consistency loss in reducing cross-class confusion. The pairwise class confusion between the source classifier and the target classifier is integrated into the MCC loss function, and W is used to give higher weights to samples that the model is more confident in predicting. Target class alignment and consistency loss also introduce various augmentation masks for the target image. The teacher network generates pseudo labels for the target samples and compares them with the predictions of the masked images. The teacher network uses the EMA method to update its weights, which can better reduce cross-class confusion and achieve more obvious and separable class boundaries.

[0087] In the embodiment of the present application, the student model training is controlled by an optimization strategy characterized by a minimum-maximum game, and the loss function is expressed as:

[0088]

[0089] Among them, ψ, φ are the weight parameters of the teacher and student networks, λ adv and λ tcacl is the weighting factor.

[0090] Specifically, the overall loss is minimized by fine-tuning the weight parameters representing the teacher and student networks respectively. The parameters of the discriminator are denoted as δ, which are adjusted to the maximum adversarial loss. The weighting factor is used to control the impact of the adversarial loss on the target class alignment and consistency loss.

[0091] It should be noted that by minimizing the loss of the student model and maximizing the loss of the discriminator, balanced optimization can be achieved in the model training process, and the generalization ability of the model can be improved. By introducing adversarial loss, the robustness of the model to the input data can be enhanced through the adversarial process with the discriminator. By adjusting the weight parameters of the teacher and student networks, the learning process of the model can be more finely controlled. By adding weighting factors, the proportion of different loss items in the total loss can be flexibly controlled to meet different training needs.

[0092] S200: performing multi-view feature extraction on a source domain sample image and a target domain sample image using at least two networks to obtain at least two groups of combined features of source domain sample image features and target domain sample image features, and capturing the global and local structures of the combined features by constructing a similarity graph and a matrix;

[0093] In the embodiment of the present application, the local structure is captured by constructing a similarity graph;

[0094] The global structure is obtained by constructing the Laplacian matrix and incorporating the normalized relationships between data points;

[0095] The Laplacian matrix is ​​expressed as:

[0096]

[0097] Among them, W i is the similarity matrix, D i is the degree matrix of the similarity matrix, is the degree of the lth node, is the semi-inverse matrix of the degree matrix, j is the node, is the degree of the node in the lth row and jth column in the similarity matrix.

[0098] Specifically, various augmentation mask techniques are used to train the model, extract features from the source domain and the target domain, thereby generating a combined feature space, construct a similarity graph to capture the local structure, and construct a Laplacian matrix to obtain the global structure of the data by merging the normalized relationships between data points.

[0099] It should be noted that through the combined use of similarity graphs and Laplacian matrices, the model can simultaneously consider the local neighborhood structure and global structure of the data, improving the ability to recognize complex data patterns. The acquisition of normalized relationships and global structures helps to optimize the performance of the model on specific tasks, especially when there is a distribution difference between the source domain and the target domain. Adversarial training enhances the robustness of the model to input perturbations. Through the adversarial process with the discriminator, the model can learn more robust feature representations.

[0100] S300: integrating the similarity graph and the matrix to obtain a high-order graph, converting the high-order graph into a joint optimization of the objective function, optimizing the terms of the objective function to obtain an optimal representation based on the high-order graph, summing the absolute values ​​of the slices in the high-order graph to obtain an affinity matrix;

[0101] In the embodiment of the present application, all variables except F in the objective function are fixed to update F, which is expressed as:

[0102]

[0103] Fix all other variables and Update, and use ADMM to calculate and process as follows:

[0104]

[0105] Add to i=1,2,3, as different Derivative each variable of the objective function, and obtain the optimal solution by minimizing the given objective function. The updated expression is:

[0106]

[0107] is the Lagrange multiplier, and the updated Lagrange multiplier is It is expressed as:

[0108] Among them, F i is the feature transformation matrix of the i-th view, is the corresponding transposed matrix, tr(〃) is the trace of the matrix, is the i-th slice of the matrix in the third dimension, L i is the Laplacian matrix of the i-th view, λ is the regularization parameter, ε is a tensor, ∥·∥ F is the Frobenius norm, I is the identity matrix, β is the weight parameter, ∥·∥2 is the 2-norm, is the i-th slice of the matrix along the first dimension, is the j-th slice of the matrix in the second dimension, ∥·∥ * is the nuclear norm, ∥·∥ 2,1 For the matrix norm, μ is the parameter of the Lagrange multiplier term, σ is the singular value, U and V are unitary matrices, is the matrix used in the nuclear norm update.

[0109] Specifically, replace

[0110] Specifically, learn high-level graphs It is used to achieve inter-view consistency and inter-class separability and the graph captures the tensor The structure of , where the tensor ε is the superposition of similarity graphs constructed by the normalized spectral embedding matrix, transforming the tensor high-order learning form into a The subsequent joint optimization problem of all v views of , the objective function is expressed as:

[0111]

[0112] Among them, the first Maximized track for all views L i F i The sum of , effectively enhances the representation of the data points to ensure that they are well separated according to the graph structure. Constraint L i F i Ensure that the feature transformation is orthonormal, preserves distances and maintains a valid orthogonal basis for the transformed features. Minimizing stacked tensors using Frobenius norm and learning graph The difference between The third term is the regularization parameter of the weights. Used to improve the discriminative performance of multi-view clustering, where Represents all elements along the first and second dimensions at the vth index of the third dimension, by The nuclear norm is used to cluster similar instances and separate dissimilar instances by summing the singular values ​​of the contribution matrix. The bidirectional relationship between nodes in the adjacency matrix of an undirected graph is symmetric, thus constraining make sure All horizontal slices of are symmetric. Promoted The low-rank and sparse structure in is also subject to symmetry constraints, where express The weight parameter β is used to adjust the balance between the competition between the third and fourth objective functions.

[0113] It should be noted that by jointly optimizing these terms, an optimal representation based on high-order graphs is obtained, thereby improving the domain alignment performance.

[0114] S400: obtaining a projection matrix by using local preservation projection based on the affinity matrix, projecting source domain sample image features and target domain sample image features into a latent space according to the projection matrix, obtaining projection features of the source domain sample image and the target sample image, calculating an objective function for the projection features, and performing category prediction on the target sample according to the objective function;

[0115] In the embodiment of the present application, the high-order graph The absolute values ​​of the slices in are summed to obtain the affinity matrix represented as:

[0116]

[0117] Using LPP projection to formalize the objective function, the projection matrix is ​​expressed as:

[0118]

[0119] Where diag(·) creates a diagonal matrix from a vector. is the i-th slice of the matrix in the third dimension, is a diagonal matrix, is the sum of the weights of the i-th row of matrix S, is the Laplace matrix, is the view, Π is the projection matrix, Π T is the corresponding transposed matrix, is a diagonal matrix.

[0120] Specifically, after obtaining the high-order graph, the absolute values ​​of the slices are summed to obtain an affinity matrix that integrates information from multiple views and captures comprehensive relationships. Subsequently, the objective function is formalized using LPP projection to determine the projection matrix to ensure that the projected samples within the same class are tightly clustered.

[0121] It should be noted that in the projection matrix calculation, the penalty function Π T Π is incorporated into the objective function to prevent the projection matrix from assuming extreme values, thereby improving the stability and generalization of the projection matrix.

[0122] In an embodiment of the present application, the source domain sample image and the target domain sample image in the view are projected into the latent space, the prototype vector is initialized using the K-means algorithm, the conditional probability that the target projection feature belongs to the category of the prototype vector and the category centroid is calculated, the category probability is obtained from the maximum value of the conditional probability, and the category of the merged prediction sample is performed.

[0123] Specifically, after the source domain sample image and the target domain sample image in the view are projected into the latent space, the distance between each target domain sample and the source domain sample prototype vector and the center point of the target domain sample image are used to make category predictions.

[0124] set up is the projection feature space of view i, expressed as This space contains the combined projection features and projected target features View i category The prototype is represented as:

[0125]

[0126] right K-means clustering is performed to exploit the structural information of the target sample. The objective function of K-means is to reduce the sum of the squared distances between each sample and its assigned center point, with the restriction that each sample can only belong to one cluster; each cluster must contain at least one sample, and the categories The cluster center is represented as

[0127] For each feature extracted from different views i∈*1,…,v+, calculate the target projection feature Conditional probability of belonging to category y given the prototype vector and the category centroid and It is expressed as:

[0128]

[0129] in, is the projected feature space of source domain view i, is the prototype feature vector of category y in source domain view i, are all projected feature vectors of category y in source domain view j, is the value that counts all samples of category y in source domain view j, is the projection feature of the target domain view i, is the prototype projection feature vector of category y in source domain view i, is the prototype projection feature vector of category y in the target domain view i; based on the three views and The class probability obtained from the maximum value of is used to predict the sample class.

[0130] It should be noted that by considering conditional probability, the model can more accurately predict the category of samples. Using the K-means algorithm to initialize the prototype vector helps to discover the intrinsic structure of the data, thereby improving the model's generalization ability for new data. In datasets with imbalanced categories, weighted sampling and other methods can ensure that minority class samples are fully represented, thereby improving the classifier's ability to recognize minority classes.

[0131] The above is a schematic scheme of the multi-view based low-rank domain adaptation method of this embodiment. It should be noted that the technical scheme of the multi-view based low-rank domain adaptation device and the technical scheme of the multi-view based low-rank domain adaptation method mentioned above belong to the same concept. The details of the technical scheme of the multi-view based low-rank domain adaptation device not described in detail in this embodiment can be referred to the description of the technical scheme of the multi-view based low-rank domain adaptation method mentioned above.

[0132] The device for low-rank domain adaptation based on multi-view in this embodiment includes:

[0133] A loss calculation module is used to obtain a source domain sample image and a target domain sample image, perform at least two different enhancement mask operations on the source domain sample image and the target domain sample image, pass the masked source domain sample image and the target domain sample image and the source domain label into the student network for training, and use at least two loss functions to calculate the loss generated by the training to update the network weights, so as to obtain at least two networks;

[0134] a feature extraction module, configured to perform multi-view feature extraction on the source domain sample image and the target domain sample image using the at least two networks, obtain at least two groups of combined features of the source domain sample image features and the target domain sample image features, and capture the global and local structures of the combined features by constructing a similarity graph and a matrix;

[0135] An integration module, used for integrating the similarity graph and the matrix to obtain a high-order graph, converting the high-order graph into a joint optimization of an objective function, optimizing the terms of the objective function to obtain an optimal representation based on the high-order graph, and summing the absolute values ​​of the slices in the high-order graph to obtain an affinity matrix;

[0136] A prediction module is used to obtain a projection matrix based on the affinity matrix using local preservation projection, project the source domain sample image features and the target domain sample image features into a latent space according to the projection matrix, obtain projection features of the source domain sample image and the target sample image, calculate an objective function for the projection features, and perform category prediction on the target sample according to the objective function.

[0137] This embodiment further provides a computing device, which is applicable to low-rank domain adaptation based on multiple views, including:

[0138] Memory and processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the low-rank domain adaptation method based on multiple views as proposed in the above embodiment.

[0139] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the multi-view based low-rank domain adaptation method proposed in the above embodiment is implemented.

[0140] The storage medium proposed in this embodiment and the method for implementing multi-view based low-rank domain adaptation proposed in the above embodiment belong to the same inventive concept. The technical details not described in detail in this embodiment can be found in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0141] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (RandomAccess Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.

[0142] Example 2

[0143] Referring to Table 1, an embodiment of the present invention provides a multi-view based low-rank domain adaptation method to verify and illustrate the technical effects used in the method.

[0144] Taking the dataset Office-Home as an example, it uses 15,588 images and contains 65 categories;

[0145] First, three mask operations are performed on the source sample image, namely: color jitter combined with blur operation, denoted as j b ; Color jitter combined with grayscale mask operation, denoted as j g ; Color jitter combined with random erasure, denoted as j e .

[0146] The source samples, unmasked target samples and masked target samples are passed into the student network training to obtain the supervision loss L src , the unmasked target image is passed into the teacher network training to obtain the TCACL loss L tcacl ,At the same time, the pseudo labels predicted by the teacher network are updated using the exponential moving average,combined with the results of the student model training, so that the teacher network can generate robust pseudo labels using contextual and local information and compare them with the predictions of the masked image.,After the training is completed, features are extracted from three independent feature spaces.

[0147] After obtaining the feature space, a similarity graph is constructed to capture the local structure of the data points. After obtaining the high-order graph, the constructed affinity matrix integrates the information of multiple views and captures the overall relationship. The matrix is ​​then applied to the local preservation projection to obtain the internal structure information of the subspace, and low-rank subspace learning is applied to align the conditional distributions of the source and target samples under different views.

[0148] As shown in Table 1, the accuracy of unsupervised domain adaptive classification using ResNet-50 and ViT features of Office-Home

[0149]

[0150] Among them, A is Art, C is Clipart, P is Product, and R is Real-World, which represent four different fields respectively. For example, A->C means that the model in field A performs unsupervised classification on the data in field C through the field adaptation method.

[0151] As shown in the table, under the ResNet-50 backbone network, MLRGL achieves 63.7% and 88.3% on A→C and P→R, respectively, exceeding the 57.1% and 82.2% of other methods such as SHOT. In the ViT backbone scenario, MLRGL performs well in tasks such as A→P and R→P, reaching 90.9% and 93.0% respectively, which is better than TVT's 85.6% in A→P and MIC's 91.4% in R→P. In addition, the overall average accuracy of this method is 1.2% higher than MIC's 86.2%.

[0152] It shows excellent performance in different domain adaptation scenarios, further consolidating its advantages in unsupervised domain adaptation tasks.

[0153] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-view based low-rank domain adaptation method, characterized in that: include: Obtaining a source domain sample image and a target domain sample image, performing at least two different enhancement mask operations on the source domain sample image and the target domain sample image, passing the masked source domain sample image and the target domain sample image and the source domain label into a student network for training, and using at least two loss functions to calculate the loss generated by the training to update the network weights, thereby obtaining at least two networks; Using the at least two networks to perform multi-view feature extraction on the source domain sample image and the target domain sample image, obtain at least two groups of combined features of the source domain sample image features and the target domain sample image features, and capture the global and local structures of the combined features by constructing a similarity graph and a matrix; Integrating the similarity graph and the matrix to obtain a high-order graph, converting the high-order graph into a joint optimization of an objective function, optimizing the terms of the objective function to obtain an optimal representation based on the high-order graph, and summing the absolute values ​​of the slices in the high-order graph to obtain an affinity matrix; A projection matrix is ​​obtained by using local preservation projection based on the affinity matrix, and the source domain sample image features and the target domain sample image features are projected into the latent space according to the projection matrix to obtain the projection features of the source domain sample image and the target sample image, and an objective function is calculated for the projection features, and the category of the target sample is predicted according to the objective function.

2. The multi-view based low-rank domain adaptation method according to claim 1, characterized in that: The loss generated by training is calculated using at least two loss functions to update the network weights, including: The student network learns from the source domain through supervision loss, and the loss function is expressed as: in, To monitor losses, is the jth source domain sample of the student model The probability assigned to class k, m is the number of labeled data in the source domain, C is the total number of samples, is the jth sample The label of the kth category; Using adversarial training, we introduce a combination of discriminator, feature extractor and classifier to reduce the performance caused by changes in the distribution data in various fields, and obtain the adversarial loss function expressed as: Among them, x s is the source domain sample image, x t is the sample image in the target domain, S ψ is the combination of feature extractor and classifier, D δ is the discriminator, The source domain sample The expected value of The target domain sample Expected value; Introducing target class alignment and consistency loss, by utilizing contextual relationships to obtain class boundaries in the target feature space, the resulting loss function is expressed as: in, is the classifier output of target sample k, is the target domain sample image x t The augmentation mask introduced, is the importance matrix for adjusting the influence of each sample based on entropy, k ′ and k is the extracted category k ′ The feature vector associated with k, k″ is all categories, C is the total number of samples, T φ Network for teachers.

3. The multi-view based low-rank domain adaptation method according to claim 1 or 2, characterized in that: Also includes: The student model training is controlled by an optimization strategy characterized by a min-max game, and the loss function is expressed as: Among them, ψ, φ are the weight parameters of the teacher and student networks, λ adv and λ tcacl is the weighting factor.

4. The multi-view based low-rank domain adaptation method as claimed in claim 3, characterized in that: The global and local structures of combined features are captured by constructing similarity graphs and matrices including: Capturing local structure by constructing similarity graphs; The global structure is obtained by constructing the Laplacian matrix and incorporating the normalized relationships between data points; The Laplacian matrix is ​​expressed as: Among them, W i is the similarity matrix, D i is the degree matrix of the similarity matrix, is the degree of the lth node, is the semi-inverse matrix of the degree matrix, j is the node, is the degree of the node in the lth row and jth column in the similarity matrix.

5. The multi-view based low-rank domain adaptation method according to claim 4, characterized in that: The optimal representation based on the high-order graph by optimizing the terms of the objective function includes: Fix all variables except F in the objective function to update F, expressed as: Fix all other variables and Update, and use ADMM to calculate and process as follows: Add to As different Derivative each variable of the objective function, and obtain the optimal solution by minimizing the given objective function. The updated expression is: is the Lagrange multiplier, and the updated Lagrange multiplier is It is expressed as: Among them, F i is the feature transformation matrix of the i-th view, is the corresponding transposed matrix, tr(·) is the trace of the matrix, is the i-th slice of the matrix in the third dimension, L i is the Laplacian matrix of the i-th view, λ is the regularization parameter, v is a tensor, ||·|| F is the Frobenius norm, I is the identity matrix, β is the weight parameter, ||·||2 is the 2-norm, is the i-th slice of the matrix along the first dimension, is the jth slice of the matrix in the second dimension, ||·|| * is the nuclear norm, ||·|| 2,1 is the l2l2-l1l1 norm of the matrix, μ is the parameter of the Lagrange multiplier term, σ is the singular value, U and V are unitary matrices, is the matrix used in the nuclear norm update.

6. The multi-view based low-rank domain adaptation method according to claim 5, characterized in that: The affinity matrix and projection matrix include: For high-order graphs The absolute values ​​of the slices in are summed to obtain the affinity matrix represented as: Using LPP projection to formalize the objective function, the projection matrix is ​​expressed as: Where diag(·) creates a diagonal matrix from a vector. is the i-th slice of the matrix in the third dimension, is a diagonal matrix, is the sum of the weights of the i-th row of matrix S, is the Laplace matrix, is the view, Π is the projection matrix, Π T is the corresponding transposed matrix, is a diagonal matrix.

7. The multi-view based low-rank domain adaptation method according to claim 6, characterized in that: According to the objective function, the category prediction of the target sample includes: The source domain sample image and the target domain sample image in the view are projected into the latent space, the prototype vector is initialized using the K-means algorithm, the conditional probability that the target projection feature belongs to the category of the prototype vector and the category centroid is calculated, the category probability is obtained from the maximum value of the conditional probability, and the category of the merged prediction sample is performed.

8. A system based on a multi-view low-rank domain adaptation method, characterized in that: include: A loss calculation module is used to obtain a source domain sample image and a target domain sample image, perform at least two different enhancement mask operations on the source domain sample image and the target domain sample image, pass the masked source domain sample image and the target domain sample image and the source domain label into the student network for training, and use at least two loss functions to calculate the loss generated by the training to update the network weights, so as to obtain at least two networks; a feature extraction module, configured to perform multi-view feature extraction on the source domain sample image and the target domain sample image using the at least two networks, obtain at least two groups of combined features of the source domain sample image features and the target domain sample image features, and capture the global and local structures of the combined features by constructing a similarity graph and a matrix; An integration module, used for integrating the similarity graph and the matrix to obtain a high-order graph, converting the high-order graph into a joint optimization of an objective function, optimizing the terms of the objective function to obtain an optimal representation based on the high-order graph, and summing the absolute values ​​of the slices in the high-order graph to obtain an affinity matrix; A prediction module is used to obtain a projection matrix based on the affinity matrix using local preservation projection, project the source domain sample image features and the target domain sample image features into a latent space according to the projection matrix, obtain projection features of the source domain sample image and the target sample image, calculate an objective function for the projection features, and perform category prediction on the target sample according to the objective function.

9. An electronic device, characterized in that: The device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images