A multi-view multi-label classification method based on consistency and specificity subspace

Abstract

The present invention discloses a multi-view multi-label classification method based on consistency and specificity subspace, which relates to the technical field of multi-view multi-label subspace, S1. Acquire a multi-view multi-label image data set and perform preprocessing; S2. Map the feature vectors of each instance of the data set to a high-dimensional space through a kernel function; S3. Construct a subspace based on the consistency and specificity of the multi-view data and use the feature representation of each information source and its category information to respectively construct a feature manifold and a label manifold structure. The multi-view multi-label classification method based on consistency and specificity subspace described in the present invention first expands the multi-view data to a nonlinear space through a kernel function, and then proposes a convex relaxation alternating optimization algorithm to restore the consistency subspace and the specificity subspace; on this basis, the feature manifold and the label manifold are used to enhance the generalization of the model, thereby providing a more accurate and effective solution to the multi-view multi-label classification problem.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-view multi-label subspace, and in particular to a multi-view multi-label classification method based on consistency and specificity subspace. Background Art

[0002] Modern applications increasingly require multi-label classification methods, such as protein functional classification, music classification, and semantic scene classification. In semantic scene classification, a photo can belong to multiple concept categories at the same time, such as sunset and beach. Similarly, in music classification, a song may belong to multiple genres. Multi-label learning technology is used to deal with objects with multiple meanings in the real world and has become a classic learning paradigm.

[0003] Real-life data is usually collected using different measurement methods, and single-view data cannot fully describe the complete information of the sample. For example, the color information and texture information of an image are two different features and can be regarded as two-view data. It is of great significance to make full use of information from multiple views. As multi-view data becomes more and more widespread in real-world applications, multi-view learning has attracted great attention.

[0004] The multi-view and multi-label characteristics of data pose severe challenges to the effectiveness, accuracy and computability of traditional data analysis theories, methods and technologies. Therefore, studying multi-view multi-label classification methods based on consistency and specificity subspace learning is conducive to building more accurate and reliable learning models, and is conducive to leveraging the advantages of artificial intelligence technology to promote the common development and prosperity of interdisciplinary disciplines. Summary of the invention

[0005] The present invention provides a multi-view multi-label classification method based on consistency and specificity subspaces to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A multi-view multi-label classification method based on consistency and specificity subspaces, the specific steps are as follows:

[0008] S1. Obtain a multi-view multi-label image dataset and perform preprocessing;

[0009] S2. Map the feature vectors of each instance of the data set to a high-dimensional space through a kernel function;

[0010] S3. Construct subspaces based on the consistency and specificity of multi-view data and use the feature representation of each information source and its category information to construct feature manifold and label manifold structures respectively;

[0011] S4. Iteratively optimize the objective function to obtain the optimal multi-view subspace representation;

[0012] S5. Input the multi-view and multi-label data to be processed into the model for classification and compare it with the corresponding prediction results to evaluate the performance of the proposed algorithm.

[0013] Furthermore, step S1 specifically includes:

[0014] S11. GIST, RGB, LAB, HSV, HarrisHue, and HarrisSift description operators in the public datasets Corel5k, Pascal07, and ESPgame are used as multi-view representations; each view in is the d of the i-th sample in the v-th view v dimensional feature vector;

[0015] S12. For the labels of multi-label data, we write them as {l1,l2,...,l q}(1≤j≤q); where l j is the jth category, q is the total number of labeled categories;

[0016] S13. Use the five-fold cross-validation method to verify the algorithm: divide the processed standardized data into training data and test data in a ratio of 8:2.

[0017] Furthermore, step S2 specifically includes:

[0018] S21. Use a nonlinear function φ(·) (here set to RBF kernel function, other kernel functions can also be used) to map the feature space of each view to the high-dimensional Hilbert space Convert the view data X(v) to φ(X(v)); use It represents the similarity between data sample i and data sample j in view v, and its calculation formula is:

[0019]

[0020] S22. Repeat step S21 to obtain a kernel matrix K(v) based on each view.

[0021] Furthermore, step S3 specifically includes:

[0022] S31. Using the consistency and specificity of multiple views, the following optimization objective function is constructed for solution:

[0023]

[0024] Where C is the consistent subspace; ∥·∥* represents the nuclear norm, which is defined as the sum of non-zero singular values; the nuclear norm can ensure the low-rank property and well preserve the spatial structure and high-order information to enhance the robustness of the learned consistent subspace; is the L2 norm, which imposes an L2 norm constraint on the specificity subspace S(v), aiming to capture the unique properties in each view;

[0025] S32. Since each feature can be represented by its adjacent features in theory, sparse representation is used to restore such a feature manifold. The calculation formula is: Among them, H -i =[h1,…,h i-1 ,h i+1 ,…,h d ] is divided by h i Feature sets other than is the reconstruction coefficient matrix for modeling feature correlation, and λ is used to balance the reconstruction error and the sparse regularization term L1 norm; with the help of the reconstruction coefficient matrix A, the underlying geometric structure of the feature manifold can be further explored:

[0026]

[0027] Where Tr(·) is the trace, which is the sum of all elements on the main diagonal of the matrix; L (v) =D (v) -A (v) is the Laplacian matrix of the vth view, D (v) yes The diagonal matrix of ;

[0028] S33. Further capture the tag correlation of multiple tags; similarly, the sparse representation is applied to the reconstruction of tag relations, that is, the high-order correlation between each tag and other tags, and the solution formula is: Among them, P -i =[p1,…,p i-1 ,p i+1 ,…,p l ] is divided by p i outside the tag set; is the reconstruction coefficient matrix for modeling the label correlation, and τ is used to balance the reconstruction error and the sparse regularization term L1 norm; with the help of the reconstruction coefficient matrix B, the underlying geometric structure of the label manifold can be further explored:

[0029]

[0030] Furthermore, step S4 specifically includes:

[0031] S41. Based on step S3, the overall optimization function is obtained:

[0032]

[0033] S42. Since the optimization of the nuclear norm is very difficult, convex relaxation is required before optimizing C; specifically, assuming a variable J as a substitute for C, the solution formula becomes:

[0034]

[0035] stC=J

[0036] This problem can be solved by the augmented Lagrangian algorithm; the equivalent augmented Lagrangian function can be written as:

[0037]

[0038] where <·,·> is the Frobenius inner product, μ>0 is the regularization term parameter and M is the Lagrange multiplier; solving the derivative of C and setting it to zero gives the following equation:

[0039] P C C+CQ C =Z C

[0040] Where P C =μI+((K (v) ) T K (v) +λ4(L (v) +(L (v) ) T )), This equation is a typical Sylvester equation and can be solved by the Bartels-Start algorithm;

[0041] S43. In order to update C, an intermediate variable J is introduced. The solution formula of J is:

[0042]

[0043] This problem can be solved by the Singular Value Threshold method; specifically, given

[0044]

[0045] U and V are unitary matrices, ∑ = diag{σ i} is a singular value matrix; then J can be updated by the following formula:

[0046]

[0047] Where D ε (x) = diag{max(|x|-ε,0)};

[0048] S44. Fixing other parameters, solving for the derivative of S and setting it to zero yields the following equation:

[0049] U1S (v) +S (v) U2=U3

[0050] Where U1=2λ2I+2(K (v) ) T K (v) +λ3(L (v) +(L (v) ) T ),U2=2λ4(IB T -B+BB T ), U3=2(K (v) ) T (K (v) CY)+λ3(L (v) +(L (v) ) T )C+2λ4C(IB T -B+BB T )); This equation is a typical Sylvester equation and can be solved by the Bartels-Start algorithm;

[0051] S45. Fix other parameters and solve the Lagrange multiplier M. The solution formula is:

[0052] M (t+1) =M (t) +μ(CJ)

[0053] S46. Repeat steps S42-S45 until the optimization target converges, and return the consistency subspace representation C and the specificity subspace representation

[0054] Furthermore, step S5 specifically includes:

[0055] The proposed method is used to predict the category label of the test data set, and the solution formula is:

[0056]

[0057] where Y pre is the prediction result of classification, is the kernel matrix based on the test set; the following four indicators are used to evaluate the performance of the proposed algorithm:

[0058] A. Hamming loss: used to examine the misclassification of samples on a single label. The smaller the evaluation index, the better.

[0059] B. Coverage: It is used to examine the search depth required to cover all relevant tags in the sorted sequence of class tags of the sample. The smaller the evaluation index, the better;

[0060] C. Sorting loss: used to examine the situation where sorting errors occur in the sample's category label sorting sequence. The smaller the evaluation index, the better;

[0061] D. Average precision: It is used to examine whether the labels before the relevant labels in the sorting sequence of the category labels of the samples are still relevant labels. The larger the evaluation index, the better.

[0062] Due to the adoption of the above technical solution, the present invention has the following technical advances compared with the prior art:

[0063] 1. The present invention provides a multi-view multi-label classification method based on consistency and specificity subspaces, which can construct and fuse the consistency and specificity subspace information while utilizing feature manifolds and label manifolds to provide a reference for the classification of multi-view multi-label data.

[0064] 2. The present invention provides a multi-view multi-label classification method based on consistency and specificity subspaces. The proposed method performs well on several authoritative benchmark datasets and obtains more accurate and reliable classification results.

[0065] 3. The present invention provides a multi-view multi-label classification method based on consistency and specificity subspaces, which can provide high-precision classification results and provide a basis for the multi-view multi-label classification problem. DETAILED DESCRIPTION

[0066] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0067] A multi-view multi-label classification method based on consistency and specificity subspaces comprises the following steps: S1. obtaining a multi-view multi-label image dataset and performing preprocessing;

[0068] S11. GIST, RGB, LAB, HSV, HarrisHue, and HarrisSift description operators in the public datasets Corel5k, Pascal07, and ESPgame are used as multi-view representations. in is the d of the i-th sample in the v-th view v dimensional feature vector;

[0069] S12. For the labels of multi-label data, we write them as {l1,l2,...,l q}(1≤j≤q). j is the jth category, q is the total number of labeled categories;

[0070] S13. Use the five-fold cross-validation method to verify the algorithm: divide the processed standardized data into training data and test data in a ratio of 8:2.

[0071] S2. Map the feature vectors of each instance of the data set to a high-dimensional space through a kernel function;

[0072] S21. Use a nonlinear function φ(·) (here set to RBF kernel function, other kernel functions can also be used) to map the feature space of each view to the high-dimensional Hilbert space Convert the view data X(v) to φ(X(v)). It represents the similarity between data sample i and data sample j in view v, and its calculation formula is:

[0073]

[0074] S22. Repeat step S21 to obtain a kernel matrix K(v) based on each view.

[0075] S3. Construct subspaces based on the consistency and specificity of multi-view data and use the feature representation of each information source and its category information to construct feature manifold and label manifold structures respectively;

[0076] S31. Using the consistency and specificity of multiple views, the following optimization objective function is constructed for solution:

[0077]

[0078] Where C is the consistent subspace. ∥·∥* represents the nuclear norm, which is defined as the sum of non-zero singular values. The nuclear norm can ensure the low-rank property and well preserve the spatial structure and high-order information to enhance the robustness of the learned consistent subspace. is the L2 norm, which imposes an L2 norm constraint on the specificity subspace S(v), aiming to capture the unique properties in each view.

[0079] S32. Since each feature can be represented by its adjacent features in theory, sparse representation is used to restore such a feature manifold. The calculation formula is: Among them, H -i =[h1,…,h i-1 ,h i+1 ,…,h d ] is divided by h iOther feature sets. is the reconstruction coefficient matrix for modeling feature correlation, and λ is used to balance the reconstruction error and the sparse regularization term L1 norm. With the help of the reconstruction coefficient matrix A, the underlying geometric structure of the feature manifold can be further explored:

[0080]

[0081] Where Tr(·) is the trace, which is the sum of all elements on the main diagonal of the matrix. (v) =D (v) -A (v) is the Laplacian matrix of the vth view, D (v) yes The diagonal matrix of .

[0082] S33. Further capture the tag correlation of multiple tags. Similarly, sparse representation is applied to the reconstruction of tag relations, that is, the high-order correlation between each tag and other tags, and the solution formula is: Among them, P -i =[p1,…,p i-1 ,p i+1 ,…,p l ] is divided by p i outside the tag set. is the reconstruction coefficient matrix for modeling the label correlation, and τ is used to balance the reconstruction error and the sparse regularization term L1 norm. With the help of the reconstruction coefficient matrix B, the underlying geometric structure of the label manifold can be further explored:

[0083]

[0084] S4. Iteratively optimize the objective function to obtain the optimal multi-view subspace representation;

[0085] S41. Based on step S3, the overall optimization function is obtained:

[0086]

[0087] S42. Since the optimization of the nuclear norm is very difficult, convex relaxation is required before optimizing C. Specifically, assuming a variable J as a substitute for C, the solution formula becomes:

[0088]

[0089] stC=J

[0090] This problem can be solved by the enhanced Lagrangian algorithm. The equivalent augmented Lagrangian function can be written as:

[0091]

[0092] where <·,·> is the Frobenius inner product, μ>0 is the regularization term parameter and M is the Lagrange multiplier. Solving for the derivative of C and setting it to zero gives the following equation:

[0093] P C C+CQ C =Z C

[0094] Where P C =μI+((K (v) ) T K (v) +λ4(L (v) +(L (v) ) T )), This equation is a typical Sylvester equation and can be solved by the Bartels-Start algorithm.

[0095] S43. In order to update C, an intermediate variable J is introduced. The solution formula of J is:

[0096]

[0097] This problem can be solved by the Singular Value Threshold method. Specifically, given

[0098]

[0099] U and V are unitary matrices, ∑ = diag{σ i} is a singular value matrix. Then J can be updated by the following formula:

[0100]

[0101] Where D ε (x) = diag{max(|x|-ε,0)}.

[0102] S44. Fixing other parameters, solving for the derivative of S and setting it to zero yields the following equation:

[0103] U1S (v) +S (v) U2=U3

[0104] Where U1=2λ2I+2(K (v) ) T K (v) +λ3(L (v) +(L (v) )T ),U2=2λ4(IB T -B+BB T ), U3=2(K (v) ) T (K (v) CY)+λ3(L (v) +(L (v) ) T )C+2λ4C(IB T -B+BB T )). This equation is a typical Sylvester equation and can be solved by the Bartels-Start algorithm.

[0105] S45. Fix other parameters and solve the Lagrange multiplier M. The solution formula is:

[0106] M (t+1) =M (t) +μ(CJ)

[0107] Repeat steps S42-S45 until the optimization target converges, and return the consistency subspace representation C and the specificity subspace representation

[0108] S5. Input the multi-view multi-label data to be processed into the model for classification and compare with the corresponding prediction results to evaluate the performance of the proposed algorithm.

[0109] The proposed method is used to predict the category label of the test data set, and the solution formula is:

[0110]

[0111] where Y pre is the prediction result of classification, is the kernel matrix based on the test set. The prediction results of the model are compared with iMVWL (Q.Tan, G.Yu, C.Domeniconi, J.Wang, Z.Zhang, Incomplete multi-view weak-label learning, IJCAI, 2018: 2703-2709) and ICM2L (Q.Tan, G.Yu, J.Wang, C.Domeniconi, X.Zhang, Individuality-and Commonality-based Multiview Multilabel Learning, IEEE Transactions on Cybernetics, 2019, 51(3): 1716-1727) and NAIM 3L (X. Li and S. Chen, A concise yet effective model for non-aligned incomplete multi-view and missing multi-label learning, IEEE Transactions on Pattern Analysis and Machine Intelligence 44.10 (2021): 5918-5932) method for comparison, the experimental results are shown in Table 1. Algorithm 1 corresponds to the verification result of the algorithm proposed in this invention; Algorithm 2 corresponds to the verification result of iMVWL; Algorithm 3 corresponds to the verification result of ICM2L; Algorithm 4 corresponds to the verification result of NAIM 3 The verification result of L.

[0112] It can be seen from Table 1 that on the Corel5k dataset, the present invention is better than other algorithms in most of the evaluation indicators.

[0113] Table 1

[0114] Hamming loss Coverage Sorting loss Average Precision Algorithm 1 0.0122±0.0001 0.1605±0.0104 0.0640±0.0031 0.5113±0.0060 Algorithm 2 0.0129±0.0001 0.2622±0.0046 0.1167±0.0033 0.3019±0.0076 Algorithm 3 0.0130±0.0001 0.3208±0.0044 0.1456±0.0023 0.2611±0.0071 Algorithm 4 0.0130±0.0000 0.2512±0.0104 0.1103±0.0051 0.3231±0.0115

[0115] It can be seen from Table 2 that on the Pascal07 dataset, the present invention is better than other algorithms in most evaluation indicators.

[0116] Table 2

[0117] Hamming loss Coverage Sorting loss Average Precision Algorithm 1 0.0644±0.0006 0.1615±0.0034 0.1173±0.0021 0.6067±0.0049 Algorithm 2 0.0722±0.0020 0.3020±0.0249 0.2411±0.0214 0.4657±0.0126 Algorithm 3 0.0732±0.0008 0.3191±0.0091 0.2490±0.0070 0.4540±0.0047 Algorithm 4 0.0702±0.0021 0.2503±0.0372 0.1975±0.0339 0.5055±0.0383

[0118] It can be seen from Table 3 that on the ESPgame dataset, the present invention is better than other algorithms in most evaluation indicators.

[0119] Table 3

[0120] Hamming loss Coverage Sorting loss Average Precision Algorithm 1 0.0171±0.0000 0.3108±0.0020 0.1184±0.0013 0.3850±0.0041 Algorithm 2 0.0174±0.0001 0.4335±0.0060 0.1825±0.0019 0.2636±0.0023 Algorithm 3 0.0174±0.0001 0.4842±0.0057 0.2092±0.0024 0.2119±0.0057 Algorithm 4 0.0174±0.0001 0.4110±0.0217 0.1734±0.0098 0.2545±0.0107

[0121] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-view multi-label classification method based on consistency and specificity subspaces, characterized by: The specific steps are: S1. Obtain a multi-view multi-label image dataset and perform preprocessing; S2. Map the feature vectors of each instance of the data set to a high-dimensional space through a kernel function; Step S2 specifically includes: S21. Use a nonlinear function φ(·), which is set as the RBF kernel function here. Other kernel functions can also be used to map the feature space of each view to the high-dimensional Hilbert space. Convert the view data X(v) to φ(X(v)); use It represents the similarity between data sample i and data sample j in view v, and its calculation formula is: S22. Repeat step S21 to obtain a kernel matrix K(v) based on each view respectively; S3. Construct subspaces based on the consistency and specificity of multi-view data and use the feature representation of each information source and its category information to construct feature manifold and label manifold structures respectively; Step S3 specifically includes: S31. Using the consistency and specificity of multiple views, the following optimization objective function is constructed for solution: Where C is the consistent subspace; ∥·∥* represents the nuclear norm, which is defined as the sum of non-zero singular values; the nuclear norm can ensure the low-rank property and well preserve the spatial structure and high-order information to enhance the robustness of the learned consistent subspace; is the L2 norm, which imposes an L2 norm constraint on the specificity subspace S(v), aiming to capture the unique properties in each view; S32. Since each feature can be represented by its adjacent features in theory, sparse representation is used to restore such a feature manifold. The calculation formula is: Among them, H -i =[h1, ..., h i-1 ,h i+1 , ..., h d ] is divided by h i Other feature sets; is the reconstruction coefficient matrix for modeling feature correlation, and λ is used to balance the reconstruction error and the sparse regularization term L1 norm; with the help of the reconstruction coefficient matrix A, the underlying geometric structure of the feature manifold can be further explored: Where Tr(·) is the trace, which is the sum of all elements on the main diagonal of the matrix; L (v) =D (v) -A (v) is the Laplacian matrix of the vth view, D (v) yes The diagonal matrix of ; S33. Further capture the tag correlation of multiple tags; similarly, the sparse representation is applied to the reconstruction of tag relations, that is, the high-order correlation between each tag and other tags, and the solution formula is: Among them, P -i =[p1, ..., p i-1 , p i+1 , ..., p l ] is divided by p i outside the tag set; is the reconstruction coefficient matrix for modeling the label correlation, and τ is used to balance the reconstruction error and the sparse regularization term L1 norm; with the help of the reconstruction coefficient matrix B, the underlying geometric structure of the label manifold can be further explored: S4. Iteratively optimize the objective function to obtain the optimal multi-view subspace representation; Step S4 specifically includes: S41. Based on step S3, the overall optimization function is obtained: S42. Since the optimization of the nuclear norm is very difficult, convex relaxation is required before optimizing C; specifically, assuming a variable J as a substitute for C, the solution formula becomes: stC=J This problem can be solved by the augmented Lagrangian algorithm; the equivalent augmented Lagrangian function can be written as: where <·,·> is the Frobenius inner product, μ>0 is the regularization term parameter and M is the Lagrange multiplier; solving the derivative of c and setting it to zero gives the following equation: P C C+CQ C =Z C Where P C =μI+((K (v)T K (v) +λ4(L (v) +(L (v) ) T )), This equation is a typical Sylvester equation and can be solved by the Bartels-Start algorithm; S43. In order to update C, an intermediate variable J is introduced. The solution formula of J is: This problem can be solved by the Singular Value Threshold method; specifically, given U and V are unitary matrices, Σ=diag{σ i } is a singular value matrix; then J can be updated by the following formula: Where D ε (x)=diag{max(|x|-ε, 0)}; S44. Fixing other parameters, solving for the derivative of S and setting it to zero yields the following equation: U1S (v) +S (v) U2=U3 Where U1=2λ2I+2(K (v) ) T K (v) +λ3(L (v) +(L (v) ) T ),U2=2λ4(IB T -B+BB T ), U3=2(K (v) ) T (K (v) CY)+λ3(L (v) +(L (v) ) T )C+2λ4C(IB T -B+BB T )); This equation is a typical Sylvester equation and can be solved by the Bartels-Start algorithm; S45. Fix other parameters and solve the Lagrange multiplier M. The solution formula is: M (t+1) =M (t) +μ(C-J) S46. Repeat steps S42-S45 until the optimization target converges, and return the consistency subspace representation C and the specificity subspace representation s; S5. Input the multi-view and multi-label data to be processed into the model for classification and compare it with the corresponding prediction results to evaluate the performance of the proposed algorithm.

2. The multi-view multi-label classification method based on consistency and specificity subspace according to claim 1, characterized in that: Step S1 specifically includes: S11. GIST, RGB, LAB, HSV, HarrisHue, and HarrisSift description operators in the public datasets Corel5k, Pascal07, and ESPgame are used as multi-view representations; each view in is the d of the i-th sample in the v-th view v dimensional feature vector; S12. For the labels of multi-label data, they are denoted as {l1, l2, ..., l q }(1≤j≤q); where l j is the jth category, q is the total number of labeled categories; S13. Use the five-fold cross-validation method to verify the algorithm: divide the processed standardized data into training data and test data in a ratio of 8:

2.

3. The multi-view multi-label classification method based on consistency and specificity subspace according to claim 1, characterized in that: Step S5 specifically includes: The proposed method is used to predict the category label of the test data set, and the solution formula is: where Y pre is the prediction result of classification, is the kernel matrix based on the test set; the following four indicators are used to evaluate the performance of the proposed algorithm: A. Hamming loss: used to examine the misclassification of samples on a single label. The smaller the evaluation index, the better. B. Coverage: It is used to examine the search depth required to cover all relevant tags in the sorted sequence of class tags of the sample. The smaller the evaluation index, the better; C. Sorting loss: used to examine the situation where sorting errors occur in the sample's category label sorting sequence. The smaller the evaluation index, the better; D. Average precision: It is used to examine whether the labels before the relevant labels in the sorting sequence of the category labels of the samples are still relevant labels. The larger the evaluation index, the better.

Images