Incomplete multi-view multi-label classification method based on task enhancement and missing view completion

Abstract

The invention relates to an incomplete multi-view multi-label classification method based on task enhancement and missing view completion. The method comprises the steps of obtaining to-be-classified multi-view data; inputting the to-be-classified multi-view data into the missing view completion and multi-label classification model, and outputting a multi-label classification result; wherein the missing view completion and multi-label classification model comprises a task-oriented feature selection module, a missing view recovery module, an adaptive learning fusion module and a multi-label classification module, and the missing view completion and multi-label classification model is obtained based on training of a training set, information bottleneck loss, reconstruction loss and target classification loss. The training set is incomplete multi-view multi-label data. According to the method, a new two-stage framework is provided, acquisition of task related information and complementation of missing views are combined, task related features are extracted in the first stage, and complete multi-view data are obtained in the second stage, so that the accuracy of the classification process is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition, and in particular to an incomplete multi-view multi-label classification method based on task enhancement and missing view completion. Background Art

[0002] With the development of data collection technology, many researchers have found that multi-view data integrated from different sources breaks through the inherent limitations of a single data source. It can provide more detailed and diverse object representations, which helps to alleviate the problem of insufficient information. For example, in medical diagnosis, three unique visual information obtained through X-rays, CT scans, and magnetic resonance imaging constitute a multi-view dataset. In addition, single-label data is often not enough to fully reflect the complexity of the real world. For example, a photo of a flower may not only require the label "flower", but also additional labels such as "winter" or "greenhouse". This shows that multi-label can provide a richer category feature space and can more accurately preserve the correlation between labels. In view of the diversity of objects in multi-label classification tasks, researchers began to combine multi-view learning with multi-label classification, which promoted multi-view multi-label classification to become an important and active research field.

[0003] For multi-view multi-label classification, many methods have been proposed in recent years, such as latent semantics-aware multi-view multi-label classification methods, multi-view probabilistic models, and methods based on maximizing label-feature dependencies. However, these methods assume that all views and their corresponding labels are complete, which is not always in line with the actual situation. For example, in video materials, audio tracks are often missing, and text descriptions may also be incomplete. These practical problems lead to incomplete multi-view features and category information in real datasets, making the incomplete multi-view multi-label learning problem a challenging research problem. To solve such problems, some methods construct shared low-dimensional subspaces based on matrix decomposition, some methods propose to maintain the global consistency and local properties of the label structure, and some methods use deep neural networks to extract deeper semantic representations from multi-view data. However, the above methods introduce missing view indicator matrices to avoid unknown views and rely only on available data for classification, failing to fully utilize the potential cross-view complementary information and the intrinsic relationship of missing data, so the effect is poor.

[0004] In order to solve the above problems, the present invention provides an incomplete multi-view multi-label classification method based on task enhancement and missing view completion. Summary of the invention

[0005] The purpose of the present invention is to provide an incomplete multi-view multi-label classification method based on task enhancement and missing view completion, so as to improve the reliability of missing view recovery.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] Incomplete multi-view multi-label classification method based on task enhancement and missing view completion, including:

[0008] Obtain multi-view data to be classified;

[0009] Inputting the multi-view data to be classified into a missing view completion and multi-label classification model, and outputting a multi-label classification result;

[0010] Among them, the missing view completion and multi-label classification model includes a task-oriented feature selection module, a missing view recovery module, an adaptive learning fusion module and a multi-label classification module. The missing view completion and multi-label classification model is obtained based on a training set and information bottleneck loss, reconstruction loss, and target classification loss training. The training set is incomplete multi-view multi-label data.

[0011] Optionally, the training set further includes preprocessing the training set before model training: filling in missing view features and missing labels in the incomplete multi-view multi-label data.

[0012] Optionally, inputting the to-be-classified multi-view data into a missing view completion and multi-label classification model includes:

[0013] Inputting the multi-view data to be classified into the task-oriented feature selection module to extract highly task-related features, wherein the task-oriented feature selection module is composed of a variational encoder;

[0014] Inputting the highly task-related features into the missing view recovery module for feature fusion and reconstruction, outputting a complete view set if the multi-view data to be classified is incomplete, and outputting the input highly task-related features if the multi-view data to be classified is complete, wherein the missing view recovery module is an autoencoder architecture;

[0015] Extracting features from the complete view set output by the missing view recovery module or the highly task-related features to obtain a view identification representation;

[0016] Inputting the identification representation into the adaptive learning fusion module to calculate the complete view fusion feature;

[0017] The complete view fusion feature is input into the multi-label classification module to obtain the multi-label classification result.

[0018] Optionally, inputting the highly task-related features into the missing view recovery module for feature fusion and reconstruction includes:

[0019] If the multi-view data to be classified is incomplete, projecting the highly task-related features into an embedding space through a view feature extraction network, aligning the dimensions of different views in the embedding space, and calculating view fusion features, wherein the view feature extraction network is composed of a multi-layer perceptron;

[0020] Inputting the view fusion features into a view feature decoding network to obtain a reconstructed view set;

[0021] The reconstructed view set is inserted into the missing data at the corresponding position in the highly task-related features to obtain the complete view set.

[0022] Optionally, the method for calculating view fusion features is:

[0023]

[0024] in, is the discriminative representation of the i-th sample specific to the l-th view, h i is the fusion representation of the i-th sample, m represents the number of views, and U i，l Indicates whether the lth view of the i-th sample is missing; U i，r Indicates whether the r-th view of the i-th sample is missing.

[0025] Optionally, the method for calculating the complete view fusion feature is:

[0026]

[0027] in, is a more discriminative representation of the i-th sample specific to the l-th view, is a more discriminative fusion representation of the i-th sample, m represents the number of views, σ l Represents the learnable weights of the softmax process and is adaptively updated according to the network training process.

[0028] Optionally, the multi-label classification module consists of a fully connected layer with a sigmoid activation function.

[0029] Optionally, the task-oriented feature selection module is the first stage of the missing view completion and multi-label classification model, and the information bottleneck loss is the loss function of the first stage:

[0030] The loss function of the first stage is:

[0031]

[0032]

[0033] Among them, L 阶段1is the first stage loss function, is the information bottleneck loss of the lth view, n is the number of samples, c is the number of label categories, and Y i，j represents the jth true label value of the i-th sample, represents the j-th predicted label probability distribution value of the i-th sample in the l-th view, U i，l Indicates whether the lth view of the i-th sample is missing, G i，j Indicates whether the jth label of the i-th sample is missing, α is the adjustment factor, Yes V (l) The characteristic dimension, V (l) Identify features of data that are highly relevant to the task, Indicates the corresponding V (l) The mean of the w-th dimension feature of the ith sample, Indicates the corresponding V (l) The standard deviation of the w-th dimension feature of the i-th sample.

[0034] Optionally, the missing view restoration module, the adaptive learning fusion module and the multi-label classification module are the second stage of the missing view completion and multi-label classification model, and the reconstruction loss and the target classification loss are the loss function of the second stage;

[0035] The loss function of the second stage is:

[0036] L 阶段2 =L c +βL re

[0037]

[0038]

[0039] Among them, L 阶段2 is the second stage loss function, β is the adjustment factor of the reconstruction loss, L re is the reconstruction loss, L c is the target classification loss, Y i，j Represents the jth true label value of the i-th sample; represents the final probability distribution value of the jth predicted label of the i-th sample, V (l) Identify features of data that are highly relevant to the task, Yes V (l) The characteristic dimension of represents the highly task-related feature representation of the i-th sample in the l-th view, is The reconstructed feature representation of .

[0040] The beneficial effects of the present invention are:

[0041] 1) This paper proposes a new two-stage framework that combines the acquisition of task-related information with the completion of missing views. The first stage extracts task-related features, and the second stage aims to obtain complete multi-view data to improve the accuracy of the classification process.

[0042] 2) Different from previous missing view restoration methods, this paper adopts the information bottleneck theory to eliminate redundant information irrelevant to the task before restoration, effectively preventing the accumulation of noise caused by this information. This ensures that the model can focus on mining and utilizing useful information in the data, thereby improving the reliability of missing view restoration.

[0043] 3) The present invention applies a padding strategy in the autoencoder-based architecture, which can increase the amount of available information, thereby enhancing the model's ability to better understand and capture consistent and complementary information across multiple views. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 This is a result diagram of an incomplete multi-view multi-label classification network based on task orientation and missing view completion according to an embodiment of the present invention;

[0046] Figure 2 This is a flowchart of an incomplete multi-view multi-label classification network training based on task orientation and missing view completion and a flowchart of a test sample label prediction according to an embodiment of the present invention, wherein (a) is a flowchart of classification network training and (b) is a flowchart of test sample label prediction. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] This embodiment provides an incomplete multi-view multi-label classification method based on task enhancement and missing view completion. For the incomplete multi-view multi-label classification task, the symbol definition is as follows: given multi-view data is represented as The corresponding incomplete multi-label set is represented as Y∈{0,1} n×c , missing view indicator matrix U∈{0,1} n×m and missing label indicator matrix G∈{0,1} n×c . Where m is the number of views, n is the number of samples, For view X (l) The feature dimension of is, c is the number of label categories. The matrix element U i,l = 1 means that the lth view of the ith sample is available, U i,l = 0 means the corresponding view is missing. Similarly, G i,j =1 indicates that the jth label of the ith sample is available, otherwise it is a missing category. The deep neural network model proposed in this embodiment trains model parameters based on incomplete multi-view multi-label data, and infers its predicted labels for complete multi-view or incomplete multi-view data in testing / actual applications.

[0050] An incomplete multi-view multi-label classification method based on task enhancement and missing view completion, specifically including:

[0051] Obtain multi-view data to be classified;

[0052] Input the multi-view data to be classified into the missing view completion and multi-label classification model, and output the multi-label classification result;

[0053] Among them, the missing view completion and multi-label classification model includes a task-oriented feature selection module, a missing view recovery module, an adaptive learning fusion module and a multi-label classification module. The missing view completion and multi-label classification model is obtained based on the training set and information bottleneck loss, reconstruction loss, and target classification loss. The training set is incomplete multi-view multi-label data.

[0054] Specifically, this embodiment designs a more robust, more flexible and higher performance two-stage task-enhanced missing view completion and incomplete multi-view multi-label classification model. The structure diagram of the model is as follows: Figure 1 As shown in Figure 2, it consists of four key components: a task-oriented feature selection module, a missing view restoration module, an adaptive learning fusion module, and a multi-label classification module.

[0055] The task-oriented feature selection module consists of a variational encoder, which aims to extract high-quality features from each view that are highly relevant to the task; the missing view recovery module based on the autoencoder architecture is designed to reconstruct the missing views and integrate the reconstructed data with the existing data to build a complete dataset to obtain more sufficient information. The adaptive learning fusion module combines the learned fusion weights to make full use of information from different views to obtain a more discriminative representation. The multi-label classification module consists of a fully connected layer with a sigmoid activation function, which realizes end-to-end multi-label category prediction. In order to eliminate the negative impact of missing labels, a priori missing label indicator matrix weighting is introduced in the loss.

[0056] Furthermore, the training set also includes preprocessing the training set before model training: filling in missing view features and missing labels in incomplete multi-view multi-label data.

[0057] Furthermore, the multi-view data to be classified is input into the missing view completion and multi-label classification model, and the output of the complete multi-view data and multi-label classification results include:

[0058] Inputting the incomplete multi-view data to be classified into a task-oriented feature selection module to extract highly task-related features, wherein the task-oriented feature selection module is composed of a variational encoder;

[0059] Input the highly relevant features of the task into the missing view recovery module for feature fusion and reconstruction to obtain a complete view set. If the input view data is incomplete, the missing view recovery module will output its complete view set. If the highly relevant features of the input data view are complete, that is, there is no missing view, then the output of the missing view recovery module is the input feature itself, where the missing view recovery module is an autoencoder architecture;

[0060] Extract features from the complete view set or highly task-related features output by the missing view recovery module to obtain view identification representation;

[0061] The discriminant representation is input into the adaptive learning fusion module to calculate the complete view fusion feature, where the complete view fusion feature is calculated as shown in formula (10).

[0062] The complete view fusion features are input into the multi-label classification module to obtain the multi-label classification results.

[0063] Specifically, the task-oriented feature selection module: When solving the complex challenge of multi-view data loss recovery, it is noted that the original data often contains a large amount of redundant information that is irrelevant to the task, which may affect the effectiveness of the subsequent recovery process and further affect the accuracy of the final classification result. Therefore, the primary goal of the present invention is to obtain data identification feature representation V that is highly relevant to the task before recovery.(l) , replacing the original view data X with redundant features (l) To achieve this goal, this embodiment proposes to minimize the following loss function based on the information bottleneck theory:

[0064]

[0065] In this formula, represents the information bottleneck loss of the l-th view, I(V (l) ; Y) represents the view representation V (l) The mutual information between the actual label Y and the actual label Y is maximized to extract more useful information related to the task; I(V (l) ;X (l) ) indicates V (l) With X (l) The mutual information between them is minimized to remove redundant information irrelevant to the task. In this process, this embodiment introduces a unified adjustment factor α to balance the retention of task-related information in all views.

[0066] However, directly calculating mutual information is often accompanied by high computational complexity. To solve this problem, first, we convert the first term -I(V (l) ; Y) is approximately a generalized common cross entropy loss:

[0067]

[0068] This approximation method not only maintains a high sensitivity to task-related information, but also significantly reduces the computational burden and improves the practical feasibility of the algorithm. Where n represents the number of samples and c represents the number of labels. i，j Represents the jth true label value of the i-th sample; represents the jth predicted label probability distribution value of the i-th sample in the l-th view. Secondly, for the second term I(V (l) ;X (l) )Using variational techniques, specifically, the upper bound is:

[0069]

[0070] Among them, E() represents expectation, p(V (l) ) can be arbitrarily set to a multivariate Gaussian distribution N(V (l) |0, I) as the approximate probability distribution, I represents the identity matrix; and p(V (l) |X (l) ) can be set as another multivariate Gaussian distribution N(V (l) |μ (l) ,∑ (l)). In order to determine μ (l) and∑ (l) , this embodiment designs a special variational encoder, which accepts the data matrix X (l) As input, and output two matrices P (l) and Q (l) , corresponding to V (l) The mean and standard deviation of V can be obtained by the following formula (l) :

[0071] V (l) =P (l) +ε⊙Q (l) (4)

[0072] Here ε is a matrix drawn from a standard Gaussian distribution, and ⊙ represents the Hadamard product. (l) ) and p(V (l) |X (l) ) into equation (3), and perform the decomposition and simplification steps, that is, the second term I(V (l) ;X (l) ) can be restated as the following loss:

[0073]

[0074] in, Yes V (l) The characteristic dimension of Indicates the corresponding V (l) The mean of the w-th dimension feature of the i-th sample; Indicates the corresponding V (l) The standard deviation of the w-th dimension feature of the ith sample. Therefore, combining equation (2) and equation (5), the loss function of the l-th view in the first stage of this embodiment is:

[0075]

[0076] U i，l Indicates whether the lth view of the i-th sample is missing; G i，j Indicates whether the jth label of the i-th sample is missing. By introducing the missing view indication matrix U and the missing label indication matrix G, the harmful interference of missing views and missing labels on the model training process can be systematically eliminated.

[0077] Missing view recovery module: This embodiment obtains task-related information through the first stage while filtering out redundant information. On this basis, this embodiment further proposes a new missing view recovery module to restore missing views and improve performance. Different from traditional methods that mainly focus on shallow features of data, this embodiment chooses to use deep autoencoders to extract high-level embedded features of multi-view data and reconstruct the data.

[0078] Specifically, this embodiment projects the task-related information obtained in the first stage into the embedding space through a specific view extraction network composed of a multi-layer perceptron, ensuring that the extracted high-level embedding features retain the view-specific semantic information, while projecting different views in the embedding space into the same dimension to obtain aligned views, and then forms a unified and comprehensive representation through average fusion as follows:

[0079]

[0080] in, is the discriminative representation of the i-th sample specific to the l-th view, h i is the fusion representation of the i-th sample, and m represents the number of views. i，l Indicates whether the lth view of the i-th sample is missing; U i，r Indicates whether the rth view of the i-th sample is missing. By combining the fusion representations of all samples, a unified fusion representation matrix H can be obtained, that is, in h i Then, we reconstruct the data of all views by training a view feature decoding network connected to H, namely:

[0081]

[0082] Yes V (l) The reconstructed data, D (l) It is the decoder corresponding to the lth view. The decoder consists of multiple linear layers and ReLu activation layers. Yes D (l) The corresponding network parameters.

[0083] This embodiment uses the representation of the non-missing initial view data to apply the following reconstruction loss to train the reconstruction model:

[0084]

[0085] Yes V (l) The i-th row vector of the matrix represents the feature representation of the i-th sample in the l-th view that is highly relevant to the task; is The i-th row vector of the matrix is Similarly, it is necessary to introduce the missing view matrix U to shield the interference of the missing view.

[0086] Finally, if Figure 1 As shown in Figure 2, the features reconstructed by decoding are inserted into the missing view positions in the original sample to obtain a complete view set. This viewset has more information and can better utilize the cross-view Figure 1 When the input data view is complete, the complete view set

[0087] Adaptive learning fusion module: After obtaining complete view data, this embodiment first uses pre-trained view-specific encoders to capture the deep semantic information in each view. The next goal is to create a unified and consistent representation to fully characterize multi-view data. Since different views may have different qualities, that is, the amount of discriminative information they carry is different, this embodiment uses a weighted fusion method based on adaptive learnable weights when the view data is complete, rather than the fixed weight fusion method used when the data is incomplete. Specifically, the fusion feature of sample i The definition is as follows:

[0088]

[0089] in, is a more discriminative representation of the i-th sample specific to the l-th view, is a more discriminative fusion representation of the i-th sample, m represents the number of views, σ l Represents the learnable weights of the softmax process and is adaptively updated according to the network training process. This method allows the model to dynamically adjust the weights according to the importance of each view, thereby more effectively integrating the information of each view. Similarly, the fusion representations of all samples are merged to obtain a unified fusion representation matrix Right now in for The transpose of .

[0090] Multi-label classification module: This embodiment constructs a direct mapping from the joint feature space to the multi-label classification space. This mapping is achieved by applying two linear layers with Sigmoid activation functions to independently generate a probability score for each potential class in each sample. These probability scores are constrained between 0 and 1 through the Sigmoid function transformation, thereby providing a confidence score between 0 and 1 for each class. Mathematically, this classification process can be abstracted as:

[0091]

[0092] in, represents the probability distribution of c categories predicted for n samples, ψ represents the classifier, d e express In order to deal with the problem of missing labels, the present invention adopts a missing label indicator matrix G to adjust the binary cross entropy loss function, that is, a weighted binary cross entropy loss function is used to evaluate the predicted label probability The difference between the true label Y is:

[0093]

[0094] Among them, Y i，j Represents the jth true label value of the i-th sample; G represents the final probability distribution value of the jth predicted label of the i-th sample. i，j Indicates whether the jth label of the i-th sample is missing.

[0095] Furthermore, the highly task-related features are input into the missing view recovery module for feature fusion and reconstruction, including:

[0096] If the multi-view data to be classified is incomplete, the highly task-related features are projected into the embedding space through the view feature extraction network, the dimensions of different views in the embedding space are aligned, and the view fusion features are calculated, where the view feature extraction network is composed of a multi-layer perceptron; the method for calculating the view fusion features is shown in formula (7).

[0097] Input the view fusion features into the view feature decoding network to obtain the reconstructed view set;

[0098] The reconstructed view set is inserted into the missing data at the corresponding position in the highly relevant feature of the task to obtain the complete view set. When there is no missing view in the input data, there is no need to complete the missing view feature, so the complete view set is the highly relevant feature of the input task.

[0099] Furthermore, the multi-label classification module consists of fully connected layers with sigmoid activation function.

[0100] Furthermore, the task-oriented feature selection module is the first stage of the missing view completion and multi-label classification model, and the information bottleneck loss is the loss function of the first stage, as shown in formulas (6) and (12).

[0101] The missing view restoration module, the adaptive learning fusion module and the multi-label classification module are the second stage of the missing view completion and multi-label classification model. The reconstruction loss and the target classification loss are the loss functions of the second stage, as shown in formulas (9), (11) and (13).

[0102] Specifically, the complete target loss function of the missing view completion and incomplete multi-view multi-label classification model: As mentioned above, the model used in this method mainly consists of two stages and four modules. The first stage includes information bottleneck loss, and the second stage includes reconstruction loss and target classification loss. Therefore, the total model training loss function is expressed as:

[0103]

[0104] L 阶段2 =L c +βL re (13)

[0105] Among them, L 阶段1 is the first stage loss function, L 阶段2 is the second stage loss function, β is the adjustment factor of the reconstruction loss, is the information bottleneck loss of the l-th view.

[0106] The method of this embodiment includes: Figure 2 (a)-(b) show two stages of model training and sample label prediction:

[0107] Detailed steps of the training phase:

[0108] Step 1: Data preprocessing: In this embodiment, missing view features and missing labels are filled with 0 values.

[0109] Step 2: Data division. The preprocessed feature matrix and label matrix are randomly divided into non-repeating batches, of which Indicates rounding up.

[0110] Step 3: Enter the first stage and extract features that are highly relevant to the task. Y, U and G) are sequentially input into the task-oriented feature selection network to extract the features of each view that are highly relevant to the task, denoted as

[0111] Step 4: Calculate the information bottleneck loss. Calculate the information bottleneck loss of each view according to formula (6): And update the network parameters.

[0112] Step 5: Determine whether the maximum number of training rounds has been reached. If so, proceed to step 6, otherwise return to step 3.

[0113] Step 6: The first-stage network outputs view data for each view that is highly relevant to the task.

[0114] Step 7: Enter the second stage and restore the missing view. Input it to the view feature extraction network, calculate the view fusion feature by formula (7), and then input it to the view feature reconstruction network, calculate the reconstruction loss according to formula (9), and get the reconstructed view set

[0115] Step 8: Get a complete data set. When the input data has missing views, use filling The missing data at the corresponding position in the , and the new complete data, that is, the complete view set, is obtained. If the input data does not have a missing view, then

[0116] Step 9: Multi-view adaptive learning weighted fusion. The input is fed into the view feature extraction network to obtain the discriminative representation of each view, and then the view fusion feature is calculated using formula (10).

[0117] Step 10: Missing multi-label classification. Input the fusion representation obtained in the previous step into the multi-label classification module to obtain the label prediction probability distribution of the input sample. And calculate the classification loss L of the fusion feature according to formula (11): c .

[0118] Step 11: Network loss calculation. 阶段2 =L c +βL re Calculate the total loss of the second stage of the model and update the network parameters.

[0119] Step 12: Determine whether the maximum number of training rounds has been reached. If so, proceed to step 13; otherwise, return to step 7.

[0120] Step 13: Save model parameters and end training.

[0121] Testing phase steps:

[0122] For the convenience of later explanation, the multi-view test data of the i-th sample is defined as An index vector indicating whether its view is missing. Taking the i-th sample as an example, the specific test process is described as follows:

[0123] Step 1: Data preprocessing. Will Missing view features in are filled with 0 values.

[0124] Step 2: Enter the first stage and extract features that are highly relevant to the task. Input into the task-oriented feature selection network to extract the features of each view that are highly relevant to the task, denoted as

[0125] Step 3: Enter the second stage and restore the missing view. Input it into the view feature extraction network, calculate the view fusion feature through formula (7), and then input it into the reconstruction network to obtain the reconstructed view set

[0126] Step 4: Get a complete data set. When the input data has missing views, that is, When there are 0 value elements in filling The missing data at the corresponding position in the , and the new complete data, that is, the complete view set, is obtained. when When there is no 0 value element in ,

[0127] Step 5: Multi-view adaptive learning weighted fusion. The input is fed into the view feature extraction network to obtain the discriminative representation of each view, and then the complete view fusion feature is calculated using formula (10).

[0128] Step 6: Output classification results. Input the fusion representation obtained in the previous step into the multi-label classification module to obtain the label prediction probability distribution of the input sample. Define rules based on multi-label classification labels Get the final 0-1 binary label vector y′ of the input data i .

[0129] Table 1 shows the average precision (AP) obtained on five datasets when the view missing rate is 50%, the label missing rate is 50%, and the training sample rate is 70%. The experimental results on multiple datasets prove the superiority of this method. Among them, LMVCAT is currently an excellent method for handling incomplete multi-view incomplete multi-label classification tasks.

[0130] Table 1

[0131]

[0132]

[0133] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An incomplete multi-view multi-label classification method based on task enhancement and missing view completion, characterized in that: include: Obtain multi-view data to be classified; Inputting the multi-view data to be classified into a missing view completion and multi-label classification model, and outputting a multi-label classification result; Among them, the missing view completion and multi-label classification model includes a task-oriented feature selection module, a missing view recovery module, an adaptive learning fusion module and a multi-label classification module. The missing view completion and multi-label classification model is obtained based on a training set and information bottleneck loss, reconstruction loss, and target classification loss training. The training set is incomplete multi-view multi-label data.

2. The incomplete multi-view multi-label classification method based on task enhancement and missing view completion according to claim 1, characterized in that: The training set also includes preprocessing the training set before model training: filling in missing view features and missing labels in the incomplete multi-view multi-label data.

3. The incomplete multi-view multi-label classification method based on task enhancement and missing view completion according to claim 1, characterized in that: Inputting the multi-view data to be classified into the missing view completion and multi-label classification model includes: Inputting the multi-view data to be classified into the task-oriented feature selection module to extract highly task-related features, wherein the task-oriented feature selection module is composed of a variational encoder; Inputting the highly task-related features into the missing view recovery module for feature fusion and reconstruction, outputting a complete view set if the multi-view data to be classified is incomplete, and outputting the input highly task-related features if the multi-view data to be classified is complete, wherein the missing view recovery module is an autoencoder architecture; Extracting features from the complete view set output by the missing view recovery module or the highly task-related features to obtain a view identification representation; Inputting the identification representation into the adaptive learning fusion module to calculate the complete view fusion feature; The complete view fusion feature is input into the multi-label classification module to obtain the multi-label classification result.

4. The incomplete multi-view multi-label classification method based on task enhancement and missing view completion according to claim 3 is characterized in that: Inputting the highly relevant features of the task into the missing view recovery module for feature fusion and reconstruction includes: If the multi-view data to be classified is incomplete, projecting the highly task-related features into an embedding space through a view feature extraction network, aligning the dimensions of different views in the embedding space, and calculating view fusion features, wherein the view feature extraction network is composed of a multi-layer perceptron; Inputting the view fusion features into a view feature decoding network to obtain a reconstructed view set; The reconstructed view set is inserted into the missing data at the corresponding position in the highly task-related features to obtain the complete view set.

5. The incomplete multi-view multi-label classification method based on task enhancement and missing view completion according to claim 4, characterized in that: The method for calculating view fusion features is: in, is the discriminative representation of the i-th sample specific to the l-th view, h i is the fusion representation of the i-th sample, m represents the number of views, and U i,l Indicates whether the lth view of the i-th sample is missing, U i,r Indicates whether the r-th view of the i-th sample is missing.

6. The incomplete multi-view multi-label classification method based on task enhancement and missing view completion according to claim 3, characterized in that: The method for calculating the complete view fusion feature is: in, is a more discriminative representation of the i-th sample specific to the l-th view, is a more discriminative fusion representation of the i-th sample, m represents the number of views, σ l Represents the learnable weights of the softmax process and is adaptively updated according to the network training process.

7. The incomplete multi-view multi-label classification method based on task enhancement and missing view completion according to claim 3, characterized in that: The multi-label classification module consists of fully connected layers with sigmoid activation function.

8. The incomplete multi-view multi-label classification method based on task enhancement and missing view completion according to claim 1, characterized in that: The task-oriented feature selection module is the first stage of the missing view completion and multi-label classification model, and the information bottleneck loss is the loss function of the first stage: The loss function of the first stage is: Among them, L 阶段1 is the first stage loss function, is the information bottleneck loss of the lth view, n is the number of samples, c is the number of label categories, and Y ,j represents the jth true label value of the i-th sample, represents the j-th predicted label probability distribution value of the i-th sample in the l-th view, U i,l Indicates whether the lth view of the i-th sample is missing, G i,j Indicates whether the hth label of the uth sample is missing, α is the adjustment factor, Yes V (l) The characteristic dimension, V (l) Identify features of data that are highly relevant to the task, Indicates the corresponding V (l) The mean of the w-th dimension feature of the ith sample, Indicates the corresponding V (l) The standard deviation of the w-th dimension feature of the i-th sample.

9. The incomplete multi-view multi-label classification method based on task enhancement and missing view completion according to claim 8, characterized in that: The missing view restoration module, the adaptive learning fusion module and the multi-label classification module are the second stage of the missing view completion and multi-label classification model, and the reconstruction loss and the target classification loss are the loss function of the second stage; The loss function of the second stage is: L 阶段2 =L c +βL re Among them, L 阶段2 is the second stage loss function, β is the adjustment factor of the reconstruction loss, L re is the reconstruction loss, L c is the target classification loss, Y i,j Represents the jth true label value of the i-th sample; represents the final probability distribution value of the jth predicted label of the i-th sample, V (l) Identify features of data that are highly relevant to the task, Yes V (l) The characteristic dimension of represents the highly task-related feature representation of the i-th sample in the l-th view, is The reconstructed feature representation of .

Images