A multi-graph constrained canonical correlation analysis method and system for multi-modal data

Abstract

The application discloses a kind of multi-modal data-oriented multi-graph constraint canonical correlation analysis method and system, including the following steps: constructing multi-view dataset;Based on the self-expression property of multi-view dataset, a graph learning term is constructed, and the graph structure corresponding to different views is obtained;Consistency graph is obtained by fusing the graph structure corresponding to different views through an adaptive weight mechanism;A multi-graph constraint canonical correlation analysis function is constructed, and the consistency graph is used as the constraint term of the function to obtain an improved multi-view feature learning objective function.The function obtained by the application can not only utilize the consistency information between different view data when performing multi-view dimension reduction, but also utilize the similarity relationship between data, so that the low-dimensional features can better reflect the relationship between the original data, and improve the clustering and classification performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of machine learning, and relates to a multi-graph constraint canonical correlation analysis method and system for multi-modal data. BACKGROUND

[0002] Multi-view learning is a machine learning strategy that fuses two or more view data. Canonical correlation analysis (CCA) is an important method of multi-view learning, which finds mapping vectors (canonical vectors) of each view to a common space by maximizing the correlation coefficient between two views, and in the common space, each view has maximum correlation. Multi-view canonical correlation analysis (MCCA) generalizes CCA to be applicable to two or more views, thereby widening the use range of CCA.

[0003] MCCA only considers the consistency features between multi-view data and ignores the intrinsic characteristics of the data, lacking mining and utilization of the structural relationship information of the data. Therefore, graph multi-view canonical correlation analysis (GMCCA) is proposed, which constructs structural constraints between data, namely graph constraints, according to prior knowledge, and utilizes the similarity between data samples, and the classification and clustering performance is obviously improved compared with MCCA.

[0004] The existing GMCCA algorithm has some problems. First, a predefined graph structure is used to construct a graph for each view, and the dependence on prior knowledge is large. This graph construction method not only needs to try the best number of nearest neighbor samples, but also needs to determine the distance calculation method of the nearest neighbor matrix. In actual application, multiple attempts and comparisons are often needed, which not only reduces the ease of use of the algorithm, but also is difficult to guarantee that the most reasonable similarity matrix is constructed. Secondly, in constructing the graph constraint term, the multiple graphs are added or a view data is selected to construct a graph, and the graph structure selection and fusion method is suboptimal. This graph construction method depends on multiple attempts, which may lose the structural information of the unused views, and also may cause the final result to be suboptimal because the distribution difference between different view data and the irrelevance of part of the structural information are not considered. SUMMARY

[0005] The application aims to solve the problem that the prior art algorithm cannot achieve the optimal result due to the large dependence on prior knowledge, the suboptimal graph structure selection and fusion method, and the pre-defined graph structure for each view during analysis, and provides a multi-graph constraint canonical correlation analysis method and system for multi-modal data Figure One The method simultaneously performs multi-view correlation and multi-graph constraint feature map learning, adaptively learns and fuses graph structure information of different views, and uses the fused graph structure information as a constraint term to improve the performance of multi-view canonical correlation analysis.

[0006] To achieve the above object, the application adopts the following technical solutions:

[0007] A multi-graph constraint canonical correlation analysis method for multi-modal data comprises the following steps:

[0008] S1: constructing a multi-view data set;

[0009] S2: constructing a graph learning term based on the self-expression property of the multi-view data set, and obtaining a graph structure corresponding to different views;

[0010] S3: fusing the graph structures corresponding to different views through an adaptive weight mechanism to obtain a consistent graph;

[0011] S4: constructing a multi-graph constraint canonical correlation analysis function, taking the consistent graph as a constraint term of the function, and obtaining an improved multi-view feature learning objective function.

[0012] The application is further improved in that:

[0013] The graph learning term of step S2 is constructed by formula (1):

[0014]

[0015] Wherein, A m represents the graph structure corresponding to each view data X m .

[0016] The graph structures corresponding to different views are fused by formula (2):

[0017]

[0018] Wherein, G represents the fused graph; d m represents the importance of each view; A m represents the graph structure corresponding to each view data X m .

[0019] The step of constructing the multi-graph constraint canonical correlation analysis function is:

[0020]

[0021] where X m represents the multi-view data from which features are to be extracted; U m represents the corresponding projection matrix; A m represents the consistency feature of each view data X m represents the corresponding graph structure; S represents the consistency feature of the multi-view data; L G represents the Laplacian matrix corresponding to G; G represents the fused graph; d m represents the importance of each view.

[0022] The step S3 further comprises, based on the obtained new multi-view feature learning objective function, iteratively calculating the optimization objective function by alternation.

[0023] The step of iteratively calculating the optimization objective function by alternation comprises:

[0024] Fixing A m , d, G, solving U m and S, the original objective function is transformed into:

[0025]

[0026] Taking the partial derivative of U m and setting it to 0, we can get:

[0027]

[0028] Introducing an intermediate variable:

[0029]

[0030] Performing eigenvalue decomposition on C, S is constructed from the eigenvectors corresponding to the largest ρ eigenvalues of C, and the obtained S is substituted back into the expression of U m , then the corresponding U m can be obtained.

[0031] Fixing S, U m , d, G, solving A m :

[0032] Taking the partial derivative of the objective function with respect to A m and setting it to 0, we can get:

[0033]

[0034] The step of iteratively calculating the optimization objective function by alternation further comprises the following steps:

[0035] Fixing S, U m , d, A m , solving G:

[0036] The original objective function is transformed into:

[0037]

[0038] Optimize each i respectively, given a certain i, the expansion is transformed into:

[0039]

[0040] Define the vector c i , let its Jth element Further transform the objective function into the following form:

[0041]

[0042] Wherein, Indicates the i-th row vector of A m ;

[0043] Fix S, U m , G, A m , solve d, and the objective function is transformed into:

[0044]

[0045] Let The above formula is equivalent to the following function:

[0046]

[0047]

[0048] Repeat the process of alternating iteration calculation until the objective function converges, that is, the projection matrix U m and the consistency feature S can be obtained.

[0049] A multi-graph constraint canonical correlation analysis system for multi-modal data, comprising a data set construction module, a graph structure acquisition module, a graph structure fusion module and an objective function construction module;

[0050] The data set construction module is used for constructing a multi-view data set.

[0051] The graph structure acquisition module is used for constructing a graph learning term based on the self-expression property of the multi-view data set, and acquiring graph structures corresponding to different views.

[0052] The graph structure fusion module is used for fusing the graph structures corresponding to different views through an adaptive weight mechanism to obtain a consistent graph.

[0053] The target function construction module is configured to construct a multi-graph constraint canonical correlation analysis function, take the consistency graph as a constraint term of the function, and obtain an improved multi-view feature learning target function.

[0054] A terminal device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method according to any one of the present application when executing the computer program.

[0055] A computer readable storage medium stores a computer program, and the computer program implements the steps of the method according to any one of the present application when executed by a processor.

[0056] Compared with the prior art, the present application has the following beneficial effects:

[0057] The application discloses a multi-graph constraint canonical correlation analysis method for multi-modal data. For the constructed multi-view data set, the automatic learning of each view data is realized based on the self-expression property of the multi-view data set, the graph structure reflecting the similarity relationship is obtained, and the graph structures corresponding to different views are fused through an adaptive weight mechanism, the structure information of the unused view is avoided to be lost due to multiple attempts during graph construction, the distribution difference between different view data is considered, and the fused graph structure with commonality is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0059] Figure 1 The flow framework diagram of the present application is shown in the figure. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme of the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application, and obviously, the described embodiments are part of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0061] Therefore, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the application claimed, but merely represents selected embodiments of the application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the application, without making creative efforts, fall within the scope of the application claimed.

[0062] It should be noted that similar reference numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0063] In the description of the embodiments of the application, it should be noted that if the terms "upper", "lower", "horizontal", "inner", and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the application is usually placed, and are merely for the convenience of describing the application and simplifying the description, and therefore, cannot be understood as indicating or implying that the device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore, cannot be understood as limiting the application. In addition, the terms "first", "second", and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0064] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0065] In the description of the embodiments of the application, it should also be noted that unless otherwise explicitly specified and limited, if the terms "arrangement", "installation", "connection", "connection" appear, they should be understood in a broad sense, for example, they can be fixedly connected, or detachably connected, or integrally connected; can be mechanically connected, or electrically connected; can be directly connected, or indirectly connected through an intermediate medium; can be the communication between two elements inside. For those of ordinary skill in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0066] The application will be further described in detail below with reference to the accompanying drawings:

[0067] Reference should be made to Figure 1The embodiment of the application discloses a multi-graph constraint canonical correlation analysis method (MCCA with Graph Learning and Fusion, GFMCCA) for multi-modal data, and comprises the following steps: constructing a multi-view data set, realizing automatic learning of each view data based on a self-expression property of a matrix, and obtaining a graph structure reflecting a similar relationship; for the graph structures corresponding to different views, different weights are given according to different importance of the views to a task, adaptive fusion is performed, and an optimal consistency graph is obtained; the fused graph structure information is used as a constraint term of a multi-view canonical correlation analysis algorithm, a new multi-view feature learning objective function is obtained, and the new model is solved through an alternating optimization mode.

[0068] Specifically, the method comprises the following steps:

[0069] For an undirected graph G with N nodes, suppose that the consistency feature is smooth on the graph G, and the MCCA objective function after adding the graph constraint term is as follows:

[0070]

[0071] wherein, is multi-view data from which features need to be extracted, is a corresponding projection matrix, S is in R d×N is a consistency feature of the multi-view data, L G is a Laplacian matrix corresponding to G.

[0072] By using a self-expression property of the data set, a graph learning term is constructed, and a corresponding objective function is established as follows:

[0073]

[0074] wherein, A m represents each view data X m corresponding graph structure.

[0075] An adaptive weight mechanism is used to fuse the graph structures of different views, and a corresponding objective function is established as follows:

[0076]

[0077] wherein, G is a fused graph, d m represents the importance of each view.

[0078] In summary, the objective function of the complete improved algorithm is as follows:

[0079]

[0080] Further, as a preferred scheme of the embodiment of the present application, the problem is solved by using alternating optimization:

[0081] (1) Fix A m , d, G, solve U m and S, the original objective function is transformed into:

[0082]

[0083] Take the partial derivative of U m and set it to 0, and we get:

[0084]

[0085] Introduce an intermediate variable:

[0086]

[0087] Characteristic decomposition is performed on C, and S is constructed from the eigenvectors corresponding to the largest ρ eigenvalues of C. Substitute the obtained S back into the expression of U m , and the corresponding U m can be obtained.

[0088] (2) Fix S, U m , d, G, solve A m :

[0089] Take the partial derivative of the objective function with respect to A m and set it to 0, and after rearrangement, we get:

[0090]

[0091] The alternating iterative calculation optimization objective function further includes the following steps:

[0092] (3) Fix S, U m , d, A m , solve G:

[0093] At this time, the original objective function is transformed into:

[0094]

[0095] Optimize each i respectively, and given a certain i, the expansion is transformed into:

[0096]

[0097] Define a vector c i , and let its jth element be Further transform the objective function into the following form:

[0098]

[0099] where, represents A m the i-th row vector of A

[0100] (4) fixing S, U m , G, A m , solving d, at this time the objective function is transformed into:

[0101]

[0102] Let The above formula is equivalent to the following function:

[0103]

[0104] Optimization problem formula (10) and formula (12) can be solved by the algorithm proposed by J.Duchi et al., and the specific solving process is as follows:

[0105] Consider the following optimization problem which is the same as the above two problem forms:

[0106]

[0107] The solution of the optimal w is as follows:

[0108] w i =max{v i -θ,0} (14)

[0109] Repeat the process of alternating iterative calculation until the objective function converges, that is, the projection matrix U m and the consistency feature S can be obtained.

[0110] The present application discloses a specific embodiment:

[0111] Given the multi-view data to be extracted features where M≥2, D m is the dimension of the mth view data sample, N is the number of data set samples; Given the dimension d of the reduced data, the regular term coefficients α, β, γ, η.

[0112] Based on the original GMCCA algorithm, the optimal consistency graph adjacency matrix is calculated by self-expression learning and adaptive fusion method, and the fused graph structure information is used as a constraint term to perform related analysis on the multi-view data. The objective function of the complete improved algorithm is:

[0113]

[0114] The specific operation includes the following steps:

[0115] Step 1: input multi-view data The dimensionality-reduced data dimension d, the regularization term coefficients a, b, g, h.

[0116] Step two: initialize an M-dimensional vector d, each component being d m = 1 / M.

[0117] Step three: initialize the graph structure of each view using a Gaussian kernel function

[0118] Step four: in the objective function, take A m , d, G as known, and solve U m and S.

[0119] Step five: in the objective function, take S, U m , d, G as known, and solve A m .

[0120] Step six: in the objective function, take S, U m , d, A m as known, and solve G.

[0121] Step seven: in the objective function, take S, U m , G, A m as known, and solve d.

[0122] Step eight: repeat steps four to seven until the objective function converges.

[0123] The effects of the present application can be further illustrated by the following experiments on real databases.

[0124] The classification and clustering effects of the multi-graph-constrained typical correlation analysis method (GFMCCA) for multi-modal data are tested on the UCI handwritten digit dataset, the Caltech7 dataset, the Caltech20 dataset and the NUS dataset, and are compared with the multi-view typical correlation analysis algorithm (MCCA), the graph-constrained multi-view typical correlation analysis algorithm (GMCCA), the principal component analysis algorithm (PCA) and the graph-constrained principal component analysis algorithm (GPCA), to verify the superiority of the disclosed algorithm.

[0125] The classification accuracy results are shown in Table 1, and it can be seen that the present algorithm has good performance on multiple classification tasks, and the classification accuracy is greatly improved compared with the GMCCA algorithm based on a pre-defined graph structure.

[0126] Table 1 Classification accuracy results of the multi-graph-constrained typical correlation analysis method (GFMCCA) for multi-modal data and comparative algorithms

[0127]

[0128] The clustering accuracy, normalized mutual information and purity results of the clustering are shown in Table 2, Table 3 and Table 4, and it can be seen that the algorithm can achieve good clustering effect on multiple data sets. It is shown that the multi-graph constraint canonical correlation analysis method for multi-modal data can more effectively obtain the consistency features of multi-view data, which benefits from the algorithm that can constrain the multi-view dimension reduction process through adaptive graph learning, so that the features after dimension reduction can not only retain the consistent information between different view data, but also better utilize the similarity relationship between the data.

[0129] Table 2 Clustering accuracy results of the multi-graph constraint canonical correlation analysis method (GFMCCA) for multi-modal data and its comparative algorithm

[0130]

[0131] Table 3 Clustering NMI results of the multi-graph constraint canonical correlation analysis method (GFMCCA) for multi-modal data and its comparative algorithm

[0132]

[0133]

[0134] Table 4 Clustering purity results of the multi-graph constraint canonical correlation analysis method (GFMCCA) for multi-modal data and its comparative algorithm

[0135]

[0136] When performing multi-view canonical correlation analysis of graph constraint, the present application learns and constructs a graph structure reflecting the similarity relationship of the multi-view data set based on the self-expression property of the multi-view data set, and the dependence on prior knowledge is smaller, and a more reasonable similarity matrix can be constructed without multiple attempts, thereby enhancing the ease of use of the algorithm.

[0137] When constructing the graph constraint term, the present application fuses the graph structures corresponding to different views through an adaptive weight mechanism to obtain a consistency graph, rather than simply adding multiple graphs or selecting a view data for graph construction, so that the structural information implied by each view is not lost, and the distribution difference between different view data is fully considered, thereby obtaining a most common fused graph structure.

[0138] When using the present application for multi-view dimension reduction, not only the consistency information between different view data can be utilized, but also the similarity relationship between the data can be utilized, so that the low-dimensional features can better reflect the relationship between the original data, thereby improving the clustering and classification performance.

[0139] The embodiment of the present application further discloses a multi-graph constraint canonical correlation analysis system for multi-modal data, comprising a data set construction module, a graph structure acquisition module, a graph structure fusion module and a target function construction module.

[0140] The data set construction module is used for constructing a multi-view data set.

[0141] The graph structure acquisition module is used for constructing a graph learning term based on the self-expression property of the multi-view data set, and acquiring graph structures corresponding to different views.

[0142] The graph structure fusion module is used for fusing the graph structures corresponding to different views through an adaptive weight mechanism to obtain a consistent graph.

[0143] The target function construction module is used for constructing a multi-graph constraint canonical correlation analysis function, taking the consistent graph as a constraint term of the function, and obtaining an improved multi-view feature learning target function.

[0144] An embodiment of the present application provides a schematic diagram of a terminal device. The terminal device of the embodiment comprises a processor, a memory and a computer program stored in the memory and executable on the processor. The processor implements the steps in each of the method embodiments when executing the computer program. Alternatively, the processor implements the functions of each module / unit in each of the device embodiments when executing the computer program.

[0145] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present application.

[0146] The terminal device can be a desktop computer, a notebook computer, a palm computer and a cloud server, etc. The terminal device can include, but is not limited to, a processor and a memory.

[0147] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0148] The memory can be configured to store the computer programs and / or modules, and the processor realizes various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory.

[0149] The modules / units integrated in the terminal device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can realize the steps of the above-mentioned various method embodiments when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0150] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-graph constrained canonical correlation analysis method for multimodal data, characterized in that, Includes the following steps: S1: Construct a multi-view dataset; S2: Construct graph learning terms based on the self-expressive properties of multi-view datasets to obtain graph structures corresponding to different views; S3: A consistent graph is obtained by fusing the graph structures corresponding to different views through an adaptive weighting mechanism; S4: Construct a multi-graph constrained canonical correlation analysis function, and use the consistency graph as the constraint term of the function to obtain the improved multi-view feature learning objective function; Construct the graph learning terms for step S2 using equation (1): in, Represents the data for each view The corresponding graph structure, The graph structures corresponding to different views are merged using equation (2): in, This represents the merged graph; Indicate the importance of each view; Represents the data for each view The corresponding graph structure; The steps for constructing the multi-graph constrained canonical correlation analysis function are as follows: in, This indicates multi-view data from which features need to be extracted; This represents the corresponding projection matrix; Represents the data for each view The corresponding graph structure; This indicates the consistency characteristics of multi-view data; express The corresponding Laplace matrix; This represents the merged graph; This indicates the importance of each view.

2. The method for multi-graph constrained canonical correlation analysis of multimodal data according to claim 1, characterized in that, Step S3 further includes learning an objective function based on the obtained new multi-view features and optimizing the objective function through alternating iterative calculations.

3. The method for multi-graph constrained canonical correlation analysis of multimodal data according to claim 2, characterized in that, The steps for calculating and optimizing the objective function using alternating iterations are as follows: fixed , , Solve and At this point, the original objective function becomes: right Taking the partial derivative and setting it to 0, we get: Introducing intermediate variables: right If we perform eigenvalue decomposition, then Depend on The largest The eigenvectors corresponding to the eigenvalues ​​are constructed, and the obtained eigenvalues ​​are... Substitute return The expression can then be used to obtain the corresponding result. ; fixed , , Solve : objective function pair Taking the partial derivatives and setting them to zero, we can simplify to get:

4. The method for multi-graph constrained canonical correlation analysis of multimodal data according to claim 3, characterized in that, The alternating iterative calculation of the optimization objective function also includes the following steps: fixed , , Solve : At this point, the original objective function becomes: For each To optimize, given a specific... The expanded form is: Define vector , let his first element The objective function is further transformed into the following form: in, express The Row vectors; fixed , , Solve At this point, the objective function becomes: make The above expression is equivalent to the following function: Repeat the alternating iterative calculation process until the objective function converges, and the projection matrix can be obtained. and consistency features .

5. A multi-graph constrained canonical correlation analysis system for multimodal data that implements the method of claim 1, characterized in that, It includes a dataset construction module, a graph structure acquisition module, a graph structure fusion module, and an objective function construction module; The dataset building module is used to build multi-view datasets; The graph structure acquisition module is used to construct graph learning terms based on the self-expressive properties of multi-view datasets and obtain the graph structure corresponding to different views. The graph structure fusion module is used to fuse the graph structures corresponding to different views through an adaptive weighting mechanism to obtain a consistent graph. The objective function construction module is used to construct a multi-graph constrained canonical correlation analysis function, using the consistency graph as a constraint term of the function to obtain an improved multi-view feature learning objective function.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-4.

Images