Incomplete multi-view learning method based on label displacement and terminal device

Abstract

This invention discloses an incomplete multi-view learning method and terminal device based on label displacement. The method includes obtaining the reconstruction loss and latent common representation of the incomplete multi-view when reconstructing the incomplete multi-view in the source domain; training a classifier to classify the incomplete multi-view by combining the latent common representation, and obtaining the classification loss; determining an objective function based on the reconstruction loss and classification loss; estimating the weights of the label distribution in the incomplete multi-view by combining the objective function, and obtaining the estimated weights; and retraining the classifier by combining the estimated weights to obtain the classifier in the target domain. This invention considers the case of label displacement between the source and target domains, thereby improving the generalization ability of incomplete multi-view learning.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an incomplete multi-view learning method and terminal device based on label displacement. Background Technology

[0002] In image processing, due to the uncertainty of acquisition, images are often composed of partial views. How to effectively process these images is called incomplete multi-view learning. For example, in medical image analysis, there are often multiple health examination tests, such as MRI images, positron emission tomography (PET) scans, pathological portraits, and X-ray images, but patients typically only undergo some of these tests. This type of data is called incomplete multi-view data in machine learning. Specifically, the data usually consists of multiple views, representing different types of features or patterns. However, some uncontrollable factors may cause not all objects to be fully observed. To address this problem, Incomplete Multi-View Learning (IMVL) aims to leverage the consistency and complementarity of information obtained from different incomplete views while reducing the adverse effects of missing parts.

[0003] The incompleteness of multi-view data increases the difficulty of accurate annotation, leading to differences in label distribution between training and test data, known as label shift. IMVL frequently encounters label shift, where the proportion of classes in the source domain (training set) and target domain (test set) differs, even though the feature distribution of each class is the same. For example, based on the incomplete multi-view example in the medical analysis above, the disease distribution varies with location and time. During a pneumonia outbreak, the incidence of pneumonia may be higher than in daily life, i.e., p(y) ≠ q(y). However, the structure and characteristics of the same disease do not change, i.e., p(x|y) = q(x|y). For example, pneumonia symptoms may always include fever and cough. Another example is swans and egrets, which can be represented in different ways, such as images and text. Due to data collection and storage reasons, some samples only have image or text information. Considering the habits of swans, they migrate south in autumn and winter. Therefore, the ratio of swans to egrets will change in different seasons at the same location, i.e., p(y) ≠ q(y). However, the characteristics of swans and egrets do not change with the seasons, i.e., p(x|y) = q(x|y).

[0004] Existing IMVL methods always assume that the label distribution is consistent across all instances, which leads to a decrease in their generalization ability due to label offsets. Summary of the Invention

[0005] This invention provides an incomplete multi-view learning method and terminal device based on label displacement, which solves the technical problem in the prior art that label offset leads to a decrease in IMVL generalization ability.

[0006] The first aspect of this invention discloses an incomplete multi-view learning method based on label displacement, comprising:

[0007] Obtain the reconstruction loss when reconstructing an incomplete multi-view in the source domain and the potential common representation of the incomplete multi-view;

[0008] By combining the latent common representation, a classifier is trained to classify the incomplete multi-view and the classification loss is obtained;

[0009] The objective function is determined based on the reconstruction loss and the classification loss;

[0010] By combining the objective function, the weights of the label distribution in the incomplete multi-view are estimated to obtain the estimated weights;

[0011] By combining the estimated weights, the classifier is retrained to obtain the classifier in the target domain.

[0012] Preferably, obtaining the reconstruction loss when reconstructing incomplete multiple views in the source domain and the potential common representation of the incomplete multiple views specifically includes:

[0013] The reconstruction loss and potential common representation are obtained according to the first formula, which is:

[0014]

[0015] In the formula, For reconstruction loss; S n It is real data. The nth data in an incomplete multi-view; h n Let θ be the potential common representation of the nth data; Θ is θ containing all views. (v) The parameter family; s nv Determine whether the nth data in the vth view is missing; if the data exists, s nv =1, otherwise, s nv =0; f v (h n ;θ (v) ) represents the v-th reconstructed network; θ (v) Let be the parameters of the v-th reconstructed network; This represents the nth data point in the vth view of the actual data.

[0016] Preferably, the classifier is trained to classify the incomplete multi-view to obtain a classification loss, specifically including:

[0017] The classification loss is obtained according to the second formula, which is:

[0018]

[0019] In the formula, For classification loss; It is a class tag in the source domain; h n This represents the potential common representation of the nth data point; It is a 0-1 classification function; For prediction functions; For regularization constraints.

[0020] Preferably, the prediction function and the regularization constraint are determined according to a third formula, which is:

[0021]

[0022] In the formula, It is the distribution of the latent common representation h in class y; h T It is the transpose of h.

[0023] Preferably, the objective function is determined based on the reconstruction loss and the classification loss, specifically including:

[0024] The objective function is determined according to the fourth formula, which is:

[0025]

[0026] In the formula, h is the objective function; n Let Θ be the potential common representation of the nth data point; θ is the parameter θ of the reconstruction network containing all views. (v) Parameter family; For reconstruction loss; For classification loss; λ is the trade-off parameter; S n It is real data. The nth data in an incomplete multi-view; N1 represents the nth data point of the vth view in the actual data; N1 is the total number of incomplete multi-views in the source domain. It is a class tag in the source domain.

[0027] Preferably, by combining the objective function, the weights of the label distribution in the incomplete multi-view are estimated to obtain the estimated weights, specifically including:

[0028] The estimated weights are determined according to a fifth formula, which includes:

[0029]

[0030] In the formula, It is an estimate of the weight w; and is a finite sample estimate of T,u; f(S) is the prediction on the data S; p(y) is the label distribution, which is the prior information of the source domain; β is the parameter that balances the estimation loss and regularization term; diag(.) converts the vector into a diagonal matrix.

[0031] Preferably, the classifier is retrained by combining the estimated weights to obtain a classifier in the target domain, specifically including:

[0032] By combining the estimated weights and minimizing the empirical loss, the classifier is retrained to obtain a classifier for the target domain.

[0033] Preferably, the empirical loss is as described in the sixth formula, which is:

[0034]

[0035] In the formula, Loss of experience; For reconstruction loss; For classification loss; λ is the trade-off parameter; S n It is real data. The nth data in an incomplete multi-view; h n Let Θ be the potential common representation of the nth data point; θ is the parameter θ of the reconstruction network containing all views. (v) The parameter family; N1 is the total number of incomplete multiviews in the source domain. It is a class tag in the source domain. This represents the estimated weight of the nth class label.

[0036] Preferably, after obtaining the classifier in the target domain, the method further includes:

[0037] The reconstruction loss is minimized by using unlabeled data from the target domain.

[0038] A second aspect of this invention discloses a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] This invention provides a label-shift-based incomplete multi-view learning method and terminal device. It integrates information from multiple incomplete views by learning a complete, structured representation, which is more general than each incomplete view individually. To ensure the learned representation satisfies the label shift assumption, multiple multilayer perceptrons are used as reconstruction functions. Furthermore, a clustering pattern is employed as the classification function to enhance the separability of the latent representation. Next, this invention designs a novel weight estimation paradigm and theoretically analyzes the impact of finite samples on weight estimation. Finally, a reweighting framework is applied to obtain a classifier in the target domain and fine-tunes it to reduce the gap between the source and target domain representations. This invention considers the case of label shift between the source and target domains, improving the generalization ability of incomplete multi-view learning. Attached Figure Description

[0041] Figure 1 This is a flowchart of the incomplete multi-view learning method based on label displacement according to an embodiment of the present invention;

[0042] Figure 2 This is a view of sulcus depth, cortical thickness, and curvature in the diagnosis of schizophrenia in an embodiment of the present invention. Detailed Implementation

[0043] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are merely illustrative and explanatory of the present invention and should not be construed as limiting the scope of protection of the present invention. All technologies implemented based on the above content of the present invention are covered within the scope of protection intended by the present invention.

[0044] In recent years, there has been increasing interest in learning from data with multiple incomplete views or data that varies with label distribution, such as incomplete multi-view learning and label shifting. In contrast, this application focuses on a novel setting that simultaneously incorporates multiple incomplete views and label-shifted data, termed Incomplete Multi-View Learning under Label Shifting (see Definition 1). This method combines information from different incomplete views, making different missing samples under different label distributions comparable. Specifically, this invention proposes a new framework called Incomplete Multi-View Learning under Label Shifting (IMLLS). Within this framework, this application first provides a formal definition of IMLLS. Then, a reconstruction loss function is used to learn latent representations; furthermore, this invention uses the learned representations to estimate the weights of the label distribution, designs a novel estimation scheme to balance the errors generated by finite samples, and analyzes the corresponding boundaries. Finally, the classifier, retrained through a reweighting strategy, is fine-tuned to reduce the gap between the two domain representations.

[0045] Definition (1) (Incomplete Multi-View Learning Based on Label Shifting): Given a data distribution taken from the source domain... Tag data and data distribution from the target domain unlabeled data Where S n It is based on real data Incomplete observations The label is a class label, and V represents the number of views. Furthermore, this application also provides the conditions for multi-view label shifting, namely p(y) ≠ q(y) and p(x|y) = q(x|y), where... The ultimate goal of this application is to train a classifier using the aforementioned data that satisfies the target domain distribution. The data is accurately categorized.

[0046] The incomplete multi-view learning based on label shifting in this embodiment of the invention, such as... Figure 1 As shown, it includes:

[0047] Step 1: Obtain the reconstruction loss and potential common representation of the incomplete multiple views when reconstructing the incomplete multiple views in the source domain, specifically including:

[0048] The reconstruction loss and potential common representation are obtained according to the first formula, which is:

[0049]

[0050] In the formula, For reconstruction loss; S n It is real data. The nth data in an incomplete multi-view; h n Let θ be the potential common representation of the nth data; Θ is θ containing all views. (v) The parameter family; s nv Determine whether the nth data in the vth view is missing; if the data exists, then s nv =1, otherwise, s nv =0; f v (h n ;θ (v) ) represents the v-th reconstruction network. The reconstruction network of this invention is a multilayer perceptron, θ (v) Let v be the parameters of the v-th reconstructed network. This represents the nth data point in the vth view of the actual data.

[0051] Potential common representation in the first formula above By integrating information from various incomplete views, encompassing as much information as possible from all incomplete views, and simultaneously satisfying the label shift assumption, h is more general than each incomplete view. Furthermore, even if none of the data exhibits a uniform missing pattern, the complete latent common representation h makes them comparable.

[0052] Step 2: Using latent common representations, train a classifier to classify incomplete multi-views and obtain the classification loss. Specifically, obtain the classification loss according to the second formula, which is:

[0053]

[0054] In the formula, For classifying losses, It is a class tag in the source domain, h n For the potential common representation of the nth data, For 0-1 classification functions, For the prediction function, For regularization constraints.

[0055] This invention aims to learn a representation that not only integrates information from incomplete views but also possesses structural separability to adapt to a classifier. Therefore, a [representation / method] was designed... and As in the third formula:

[0056]

[0057] In the formula, The distribution of the latent common representation h in class y, h T It is the transpose of h.

[0058] Embodiments of the present invention The form allows the 0-1 classification function to not only penalize misclassifications but also restrict the structure of h. Specifically, for the correct classification case, i.e. It will not change h n The structure. For the case of misclassification, i.e. It will force h n The similarity between the distribution center of the true label and the actual label is greater than h. n The similarity between the distribution centers of all mislabeled items and the distribution centers of all mislabeled items. Furthermore, under the influence of representation learning, this invention utilizes constraint terms... This constraint enhances the clustering structure. It makes the predicted distribution centers as consistent as possible with the distribution centers of the true labels, thereby enhancing the discriminative power of the representation.

[0059] Step 3: Determine the objective function based on the reconstruction loss and classification loss, specifically including:

[0060] The objective function is determined based on the fourth formula, which is:

[0061]

[0062] In the formula, h is the objective function; n Let θ be the potential common representation of the nth data point; Θ is the parameter θ of the reconstruction network containing all views. (v) Parameter family; For reconstruction loss; S is the classification loss; λ is the trade-off parameter, balancing the reconstruction loss and the classification loss; n It is real data. The nth data in an incomplete multi-view; N1 represents the nth data point of the vth view in the actual data; N1 is the total number of incomplete multi-views in the source domain. It is a class tag in the source domain.

[0063] Step 4: Combine the objective function to estimate the weights of the label distribution in the incomplete multi-view dataset, obtaining the estimated weights, specifically including:

[0064] The estimated weights are determined based on the fifth formula, which includes:

[0065]

[0066] In the formula, It is an estimate of the weight w; and is a finite sample estimate of T,u; f(S) is the prediction on the data S; p(y) is the label distribution, which is the prior information of the source domain; β is the parameter that balances the estimation loss and regularization term; diag(.) converts the vector into a diagonal matrix.

[0067] Step 5: Combine the estimated weights and retrain the classifier to obtain the classifier in the target domain. This includes:

[0068] By combining the estimated weights and minimizing the empirical loss, the classifier is retrained to obtain the classifier in the target domain.

[0069] The experience loss is illustrated in Formula 6, which states:

[0070]

[0071] In the formula, Loss of experience; For reconstruction loss; For classification loss; λ is the trade-off parameter; S n It is real data. The nth data in an incomplete multi-view; h n Let Θ be the potential common representation of the nth data point; θ is the parameter θ of the reconstruction network containing all views. (v) The parameter family; N1 is the total number of incomplete multiviews in the source domain. It is a class tag in the source domain. This represents the estimated weight of the nth class label.

[0072] Furthermore, since this invention considers scenarios with incomplete multi-view representations, it is necessary to attempt to narrow the gap between the training and testing phases. Therefore, this invention attempts to fine-tune the process of learning the complete representation, making the model more suitable for the target domain. Specifically, after obtaining the classifier in the target domain, this invention uses... To minimize the reconstruction loss, we obtain the new parameter Θ. t This reduces the reconstruction loss of the target domain and ensures the consistency of representations between the training and testing phases. The final representation is obtained. And the final label.

[0073] The second aspect of this invention discloses a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0074] This invention integrates information from multiple incomplete views by learning a complete, structured representation that is more general than each individual incomplete view. To ensure that the learned representation satisfies the label shift assumption, multiple multilayer perceptrons are used as reconstruction functions. Furthermore, clustering patterns are employed as the classification function to enhance the separability of the latent representation. Next, this invention designs a novel weight estimation paradigm and theoretically analyzes the impact of finite samples on weight estimation. Finally, a reweighting framework is applied to obtain a classifier in the target domain, which is then fine-tuned to reduce the gap between the source and target domain representations.

[0075] The effectiveness of the method of the present invention will be verified below with more specific embodiments.

[0076] Schizophrenia is a common severe mental disorder, a clinical syndrome composed of a variety of symptoms. Specifically, it manifests as disturbances in sensation, thought, emotion, and behavior, as well as a lack of coordination between mental activity and the surrounding environment. The complexity and heterogeneity of symptoms pose challenges to the objective diagnosis of schizophrenia based solely on clinical presentation. In particular, studying subtle changes associated with brain disorders through brain imaging scans such as structural magnetic resonance imaging (MRI) is of great significance.

[0077] This invention utilizes two public datasets, UCLA and Cobre, for evaluation. The UCLA data consists of raw 3D-T1 MRI scans from 58 patients with schizophrenia and 132 healthy controls. The Cobre data includes raw MRI scans from 71 patients with schizophrenia and 74 healthy controls. All structural MRI data were preprocessed using FreeSurfer's structural MRI preprocessing pipeline, where a two-hemispheric fs LR32K cortical surface mesh and corresponding vertex morphology measurements, including groove depth, cortical thickness, and curvature, were generated for each subject. Figure 2 As shown. In this invention, 30 patients with schizophrenia and 30 healthy controls were selected as the source domain from the UCLA dataset, with other samples used as the target domain. Similarly, 30 patients with schizophrenia and 40 healthy controls were selected as the source domain from the Cobre dataset, with other samples used as the target domain. Sulcus depth, cortical thickness, and curvature were used as three views of the dataset. Since the original data was complete, this invention randomly omitted some views of the samples at certain percentages: 10%, 20%, 30%, 40%, and 50%.

[0078] This invention compares IMLLS with other algorithms. The results are shown in Table 1 below.

[0079] The results in Table 1 show that:

[0080] (1) The IMLLS method of the present invention achieves better performance in all cases. This fully verifies the effectiveness of IMLLS.

[0081] (2) As the missing rate increases, the performance of IMLLS decreases more slowly, which demonstrates the stability of IMLLS for high missing data.

[0082] (3) Compared with the Cobre dataset, IMLLS shows a significant improvement on the UCLA dataset. This is likely because the label shift in the UCLA dataset is larger than that in the Cobre dataset. Therefore, the weighting strategy proposed in this application can play a greater role.

[0083] Table 1 Comparison of Accuracy and F-score of two schizophrenia datasets under different PERs.

[0084]

[0085] The best performance on each dataset in Table 1 is indicated in bold. · / ⊙ / ο indicates that IMLLS is significantly better / worse than the compared method (paired t-test at 95% significance). The experimental results regarding IMLLS win / draw / loss counts are summarized in the last column.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention, or modify them into equivalent embodiments, without departing from the scope of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention, without departing from the content of the present invention, should fall within the protection scope of the present invention.

Claims

1. A method of incomplete multi-view learning based on label shift, the incomplete multi-view being magnetic resonance images, positron emission tomography and x-ray images, characterized in that, include: Obtain the reconstruction loss when reconstructing an incomplete multi-view in the source domain and the potential common representation of the incomplete multi-view; By combining the latent common representation, a classifier is trained to classify the incomplete multi-view and the classification loss is obtained; The objective function is determined based on the reconstruction loss and the classification loss; By combining the objective function, the weights of the label distribution in the incomplete multi-view are estimated to obtain the estimated weights; By combining the estimated weights, the classifier is retrained to obtain a classifier in the target domain; Obtaining the reconstruction loss when reconstructing incomplete multiple views in the source domain and the potential common representation of the incomplete multiple views specifically includes: The reconstruction loss and potential common representation are obtained according to the first formula, which is: In the formula, For reconstruction loss; It is real data. The first in incomplete multi-view n One data point; h n For the first n The potential shared representation of each data point; Decision No. v The first view n Is any data missing? If the data exists... ,otherwise, ; For the first v A reconstructed network; For the first v The parameters of the reconstructed network; For containing all views Parameter family; The first in the real data v The first view n One data point; Training a classifier to classify the incomplete multi-view and obtaining the classification loss specifically includes: The classification loss is obtained according to the second formula, which is: In the formula, For classification loss; It is a class tag in the source domain; h n For the first n The potential shared representation of each data point; It is a 0-1 classification function; For prediction functions; For regularization constraints; The prediction function and the regularization constraint are determined according to a third formula, which is: wherein is y potential consensus representation in the class of h distribution of h T is h transpose of Based on the reconstruction loss and the classification loss, the objective function is determined, specifically including: The objective function is determined according to the fourth formula, which is: In the formula, The objective function is... h n For the first n The potential shared representation of each data point; Parameters for a reconstructed network that includes all views Parameter family; For reconstruction loss; For classification loss; It is a trade-off parameter; It is real data. The first in incomplete multi-view n One data point; The first in the real data v The first view n One data point; N 1 represents the total number of incomplete multiviews in the source domain; It is a class tag in the source domain; Based on the objective function, the weights of the label distribution in the incomplete multi-view are estimated to obtain the estimated weights, specifically including: The estimated weights are determined according to a fifth formula, which includes: In the formula, Weight The estimated value; ; and for Finite sample estimation; f ( S ) for in data S The above prediction; It is the label distribution, which is the prior information of the source domain; It is a parameter that balances the estimated loss and the regularization term; Convert a vector into a diagonal matrix.

2. The method of claim 1 wherein, By combining the estimated weights, the classifier is retrained to obtain a classifier in the target domain, specifically including: By combining the estimated weights and minimizing the empirical loss, the classifier is retrained to obtain a classifier for the target domain.

3. The method of claim 2, wherein, The empirical loss is described in Formula 6, which is: In the formula, Loss of experience; For reconstruction loss; For classification loss; It is a trade-off parameter; It is real data. The first in incomplete multi-view n One data point; h n For the first n The potential shared representation of each data point; Parameters for a reconstructed network that includes all views Parameter family; N 1 represents the total number of incomplete multiviews in the source domain. It is a class tag in the source domain. For the first n Estimated weights of each class label.

4. The method according to any one of claims 1 to 3, characterized in that, After obtaining the classifier in the target domain, the following steps are also included: The reconstruction loss is minimized by using unlabeled data from the target domain.

5. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 4.

Images