Method and device for determining consistency feature representation of multi-modal data

Abstract

The invention provides a method and device for determining consistency feature representation of multi-modal data, electronic equipment and a storage medium, and relates to the field of artificial intelligence, and the method comprises the steps that terminal equipment determines an atlas structure of the multi-modal data according to the multi-modal data and an atlas learning network; the terminal device determines the consistency feature representation of the multi-modal data according to the multi-modal data, the atlas structure and a multi-modal data consistency feature representation learning network, and the multi-modal data consistency feature representation learning network is combined with an information bottleneck thought. According to the method, the consistency feature representation of the multi-modal data is effectively and accurately determined.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular, to a method, device, electronic device and storage medium for determining consistent feature representations of multimodal data. Background Art

[0002] Multimodal representation learning refers to the fusion of information of the modal data given the same sample to obtain a consistent feature representation. This feature representation should have sufficient information contained in the original multimodal data and can be used as a high-dimensional semantic feature required by a unimodal algorithm, that is, it can be used in a variety of downstream tasks. Existing multimodal representation learning methods include two major categories of methods, namely shallow model multimodal representation learning methods and deep model multimodal representation learning methods.

[0003] Existing shallow multimodal representation learning often performs poorly when faced with multimodal data of higher dimensions or with more complex data structures. The reason is that the shallow model has limited expressive power and cannot extract effective feature representations from multimodal data.

[0004] Existing deep model multimodal representation learning methods consider the correlation between the original samples, start from the graph perspective to obtain multimodal consistent representation, and perform well. However, these algorithms all assume that the original dataset should have a given graph structure between samples, which is different from the situation in actual application scenarios where there are mostly only original multimodal data. Almost all algorithms applicable to multimodal datasets that do not contain graph structures do not consider introducing graph structures to assist in the construction of consistent feature representations. Therefore, when faced with data with more complex relationships between samples, existing deep model multimodal representation learning methods perform poorly. Summary of the invention

[0005] The embodiments of the present application provide a method for determining a consistent feature representation of multimodal data, thereby achieving a more efficient and accurate determination of a consistent feature representation of multimodal data.

[0006] In a first aspect, an embodiment of the present application provides a method for determining a consistent feature representation of multimodal data, comprising:

[0007] Determining a graph structure of the multimodal data based on the multimodal data and the graph learning network;

[0008] Determining a consistent feature representation of the multimodal data according to the multimodal data, the graph structure and a multimodal data consistent feature representation learning network,

[0009] Among them, the multimodal data consistency feature representation learning network combines the information bottleneck idea.

[0010] In a second aspect, an embodiment of the present application provides a device for determining a consistent feature representation of multimodal data, including:

[0011] A processing unit, configured to determine a graph structure of the multimodal data based on the multimodal data and the graph learning network;

[0012] The processing unit is further configured to determine a consistent feature representation of the multimodal data according to the multimodal data, the graph structure and a multimodal data consistent feature representation learning network,

[0013] Among them, the multimodal data consistency feature representation learning network combines the information bottleneck idea.

[0014] In a third aspect, the present application provides an electronic device, including:

[0015] a processor adapted to implement computer instructions; and,

[0016] The memory stores computer instructions, where the computer instructions are suitable for being loaded by the processor and executing the method of the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are read and executed by a processor of a computer device, the computer device executes the method of the first aspect above.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the first aspect above.

[0019] The above technical scheme determines the graph structure of multimodal data through multimodal data and graph learning network. The determination of the graph structure can integrate the graph information into the subsequent modal feature representation process, making the modal feature representation more reliable; the terminal device determines the consistent feature representation of the multimodal data based on the multimodal data, the determined graph structure and the multimodal data consistent feature representation learning network, wherein the multimodal data consistent feature representation learning network combines the information bottleneck idea, which reduces the redundant information and invalid information in the consistent feature representation of the multimodal data, making the consistent feature representation of the multimodal data more effective and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a graph structure provided in an embodiment of the present application;

[0021] Figure 2 An optional schematic diagram of a system architecture involved in an embodiment of the present application;

[0022] Figure 3 A schematic flowchart of a method for determining a feature representation of multimodal data provided in an embodiment of the present application;

[0023] Figure 4 A schematic block diagram of a graph learning network structure provided in an embodiment of the present application;

[0024] Figure 5 A schematic block diagram of an automatic encoder provided in an embodiment of the present application;

[0025] Figure 6 A schematic block diagram of a multimodal data consistency feature representation learning network structure provided in an embodiment of the present application;

[0026] Figure 7 A schematic flowchart of a method for determining a feature representation of multimodal data provided in an embodiment of the present application;

[0027] Figure 8 is a schematic block diagram of a device according to an embodiment of the present application;

[0028] Fig. 9 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0030] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0031] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0032] Multimodal Machine Learning (MML) is a direction of artificial intelligence software technology.

[0033] Modality refers to some ways of expressing or perceiving things. Each source or form of information can be called a modality. For example, human touch, hearing, vision and smell; for example, voice, video, text, etc. can all be called a modality.

[0034] Multimodality is the expression or perception of things from multiple modes.

[0035] For example, multimedia data describing the same object can describe a specific object through video, picture, voice, text and other information.

[0036] For example, data collected by different sensors on the same object, such as image data generated by different examination equipment on the same object in medical imaging, including data generated by B-scan ultrasonography, computer tomography, magnetic resonance imaging and other examination equipment on the same object.

[0037] Multimodal representation learning in multimodal machine learning refers to the fusion of information of the modal data based on the given multimodal data of the same sample to obtain a consistent feature representation. This feature representation should have sufficient information contained in the original multimodal data and can be used as the high-dimensional semantic features required by the unimodal algorithm, that is, it can be used in a variety of downstream tasks.

[0038] Existing multimodal representation learning methods include two major categories, namely, shallow model multimodal representation learning methods and deep model multimodal representation learning methods.

[0039] Most of the shallow model multimodal representation learning methods are based on two methods: non-negative matrix factorization and canonical correlation analysis. Non-negative matrix factorization aims to decompose the original data matrix through different basis matrices, project the multi-view data into a common subspace, and obtain shared feature representations. Canonical correlation analysis explores consistent representations by maximizing the correlation between paired modes.

[0040] Deep model multimodal representation learning methods are mainly based on deep neural networks. In addition to some algorithms that migrate shallow methods to deep models, such as deep canonical correlation analysis, there are many deep learning methods that use new network architectures and ideas. Nested autoencoders (Autoencoder in Autoencoder Networks, AE2-Nets) can learn consistent multi-view representations from modal datasets through nested autoencoders; Multi-view Information Bottleneck (MIB) is based on the information bottleneck idea and the assumption that there is mutual redundancy between multi-view data. It calculates the mutual information (MI) between the feature representations of each modality through a deep neural network, thereby discarding redundant information to obtain a simple multimodal consistent representation; Collaborative Multi-View Information Bottleneck Networks (CMIB-Nets) further combines the autoencoder and information bottleneck ideas, while considering the complementarity and effectiveness of each modality, and uses neural network learning to obtain a simple and effective multimodal consistent feature representation.

[0041] Existing shallow multimodal representation learning often performs poorly when faced with multimodal data of higher dimensions or with more complex data structures. The reason is that the expressive power of shallow models is limited and they are unable to extract effective feature representations from multimodal data.

[0042] Existing deep model multimodal representation learning methods consider the correlation between the original samples, start from the graph perspective to obtain multimodal consistent representation, and perform well. However, these algorithms all assume that the original dataset should have a given graph structure between samples, which is different from the situation in actual application scenarios where there are mostly only original multimodal data. Almost all algorithms suitable for multimodal datasets that do not contain graph structures do not consider introducing graph structures to assist in the construction of consistent feature representations, such as nested autoencoders (Autoencoder in Autoencoder Networks, AE2-Nets) and collaborative multi-view information bottleneck networks (Collaborative Multi-View Information Bottleneck Networks, CMIB-Nets). Although they have a considerable improvement in performance compared to traditional shallow algorithms, they do not consider the graph structure between samples, so they perform poorly when facing data with more complex relationships between samples.

[0043] Therefore, this application starts from the scenario where the actual multimodal data does not have a given graph structure, and proposes a method for consistent feature representation of multimodal data to address the problem that the existing multimodal representation learning cannot reasonably utilize the graph structure. This method determines the consistent feature representation of multimodal data through the constructed graph learning network and the multimodal data consistent feature representation learning network. First, the graph structure of the multimodal data is determined through the multimodal data and the graph learning network. The determination of the graph structure can enable the graph information to be integrated into the subsequent modal feature representation process, making the modal feature representation more reliable; secondly, according to the multimodal data, the determined graph structure and the multimodal data consistent feature representation learning network, the consistent feature representation of the multimodal data is determined, wherein the multimodal data consistent feature representation learning network combines the information bottleneck idea, which reduces the redundant information and invalid information in the consistent feature representation of the multimodal data, making the consistent feature representation of the multimodal data more effective and accurate.

[0044] It should be understood that graph structure can also be called graph information. In the field of machine learning, graph structure often refers to the relationship between samples: if different samples are abstracted into points, and the relationship (similarity) between them is represented by lines connecting different points, then a graph connecting all points can be obtained. This graph is the relationship between samples, which is defined as the structure between samples. According to the knowledge related to data structure, it can be represented by the form of adjacency matrix. Since the structure is often expressed as an undirected graph, the adjacency matrix is ​​generally a symmetric matrix. Figure 1 As shown in the figure, given samples of different categories, kitten A, kitten B, kitten C, bird D, and bird E, the similarity relationship is shown on the left: the kittens have a high similarity, the birds also have a high similarity, and kitten C and bird D also have a certain similarity due to their similar colors; the corresponding adjacency matrix form is shown in the figure on the right, the blue part is the similarity value between kittens, the yellow part is the similarity value between birds, and the green part indicates that there are relatively similar samples between kittens and birds (i.e., kitten C and bird D); it is worth noting that in the graph structure, the influence of itself on itself is often ignored, such as the similarity between kitten A and kitten A itself, which also leads to the constraint that the diagonal of the adjacency matrix is ​​0 in many algorithms.

[0045] Figure 2This is an optional schematic diagram of the system architecture 100 involved in the embodiment of the present application. The system architecture 100 includes a graph structure learning module 110, a multimodal data consistency feature representation learning module 120, and a downstream task processing module 130. Among them, the graph structure learning module 110 is used to determine the graph structure of the original multimodal data; the multimodal data consistency feature representation learning module 130 is used to determine the consistency feature representation of the multimodal data based on the original multimodal data and the graph structure; the downstream task processing module 130 refers to the use of the consistency feature representation of the multimodal data to perform tasks such as classification and clustering on the multimodal data.

[0046] Different modules can be implemented through different algorithms. For example, the graph structure learning module 110 can use an autoencoder to determine the graph structure of multimodal data; the consistency feature representation learning module 120 of multimodal data can be implemented using a graph convolutional network, an autoencoder network, a fusion network, and an ablation network.

[0047] The solution provided by the embodiments of the present application is described below in conjunction with the accompanying drawings.

[0048] Figure 3 A schematic flow chart of a method 200 for determining a feature representation of multimodal data provided in an embodiment of the present application. The method 200 can be executed by any electronic device having data processing capabilities. For example, the electronic device can be implemented as a server or a terminal device. The following description is made by taking the electronic device as a terminal device as an example. Figure 3 As shown, method 200 may include steps S210 to S220.

[0049] S210, the terminal device determines a graph structure of the multimodal data according to the multimodal data and the graph learning network;

[0050] S220, the terminal device determines a consistent feature representation of the multimodal data according to the multimodal data, the graph structure and a multimodal data consistent feature representation learning network,

[0051] Among them, the multimodal data consistency feature representation learning network combines the information bottleneck idea.

[0052] In method 200, the terminal device determines the graph structure of the multimodal data through the multimodal data and the graph learning network. The determination of the graph structure can enable the graph information to be incorporated into the subsequent modal feature representation process, making the modal feature representation more reliable; the terminal device determines the consistent feature representation of the multimodal data based on the multimodal data, the determined graph structure and the multimodal data consistent feature representation learning network, wherein the multimodal data consistent feature representation learning network combines the information bottleneck idea, which reduces the redundant information and invalid information in the consistent feature representation of the multimodal data, making the consistent feature representation of the multimodal data more effective and accurate.

[0053] In order to more clearly understand the embodiment of the present application, the method 200 is described in steps below.

[0054] Optionally, the method 200 further includes:

[0055] Performing normalization preprocessing on the multimodal data to obtain processed multimodal data;

[0056] Determining the graph structure of the multimodal data according to the multimodal data and the graph learning network includes:

[0057] Determining a graph structure of the multimodal data according to the processed multimodal data and the graph learning network;

[0058] The step of determining the consistent feature representation of the multimodal data according to the multimodal data, the graph structure and the multimodal data consistent feature representation learning network includes:

[0059] The consistent feature representation of the multimodal data is determined based on the processed multimodal data, the graph structure and the multimodal data consistent feature representation learning network.

[0060] Optional, for a given multimodal dataset Where N is the total number of instances, m is the total number of modes, Indicates that it has d v The original matrix of the v-th mode of the dimension attribute, Indicates that the data belongs to the real number domain. Each mode is normalized and the corresponding formula is shown in formula (1):

[0061]

[0062] Where i = 1, 2, ..., N, Represents the matrix X (v)The i-th row of is the vector corresponding to the i-th data instance in the modality v; max(·) and min(·) represent the functions that take the maximum and minimum element values ​​in the matrix or vector, respectively.

[0063] By performing normalization preprocessing on the multimodal data, the multimodal data can be compressed into the same data interval, which is beneficial to subsequent calculations.

[0064] Optionally, the graph learning network includes multiple encoders, fully connected layers and multiple decoders, the multiple encoders correspond to the multiple decoders one by one, and the fully connected layer is constructed according to the self-expression matrix G.

[0065] The first encoder obtains the hidden code of the first modal data, the hidden code is input into the fully connected layer for self-expression learning, the fully connected layer outputs the hidden code after self-expression learning to the first decoder, and the first decoder outputs the reconstructed first modal data.

[0066] The first encoder is any encoder among the multiple encoders, the first decoder is any decoder among the multiple decoders, and the first modal data is data of any modality among the multi-modal data.

[0067] Specifically, Figure 4 A schematic block diagram of a graph learning network structure provided in an embodiment of the present application. Figure 4 As shown in , in this graph learning network, in order to avoid the high complexity of the original data and the difficulty in directly obtaining the graph structure, for each modal data v, the autoencoder structure is used to obtain the specific hidden code of the corresponding modal data, and it is combined with the self-expression learning idea to obtain the multi-modal data consistency graph structure. Figure 4 In the modal data X (v) Using the encoder Get the modal data X (v) Specific hidden code, the modal data X (v) The specific hidden code is input to the fully connected layer constructed according to the self-expression matrix G to obtain the processed modal data X (v) Specific hidden code, the processed modal data X (v) Specific hidden code input to the decoder Get reconstructed modal data

[0068] The above multiple one-to-one corresponding encoders and decoders are also called autoencoders. Autoencoders are a common neural network architecture in the field of unsupervised learning and can be used in application scenarios such as data feature extraction, restoration, and noise reduction. Its structure is as follows Figure 5 As shown, Figure 5A schematic block diagram of an autoencoder provided in an embodiment of the present application, wherein the autoencoder as a whole is composed of two parts: an encoder network and a decoder network: the encoder part is dedicated to learning from the original data X to obtain a sufficiently representative high-level hidden feature representation S; the decoder hopes to start from the representation S and recover the reconstructed data that is sufficiently similar to the original data X. If the original data X and the reconstructed data If the data is similar enough, it means that the learned hidden feature representation S is good enough to represent the original data X at a high-dimensional semantic level. For example, if a photo of a kitten is regarded as the original data, the color, pattern, size, hair length, etc. of the kitten are all high-level hidden semantic features. If all valid features can be obtained, a picture similar enough to the original photo can be reconstructed based on these features; and conversely, if the reconstructed picture is similar enough to the original photo, it also shows that the acquired features are complete and accurate enough. Therefore, the objective function of the autoencoder network can generally be set as the original data X and the reconstructed data The commonly used relationship measures include mean square error, Frobenius norm, etc. For example, the objective function can be set as shown in formula (2):

[0069]

[0070] Self-expressive learning is a graph learning method used to learn graph structures on raw data. This method is based on the assumption that the feature representation of any sample in the data can be approximately expressed by a weighted combination of the feature representations of multiple other samples. For example, the features of kitten C should be approximately represented by the posture, body shape, and appearance of kitten A and kitten B, and the color of bird D. Since other samples that can represent the current sample should have a high degree of similarity with the current sample, when the assumption is true, it can actually be regarded as establishing a graph structure of similarity relationships between all samples. Based on this assumption, if the original data is represents the i-th row of the X matrix, that is, the feature representation of the i-th sample, then the following formula (3) can be obtained:

[0071] X i =G i X,stG i ≥0, i=1…N (3)

[0072] in, It is intended to represent the weighted fusion of all other sample features to obtain the current sample X i Considering that the influence of the sample itself in the self-expression process should be excluded, the corresponding weight value should be set to 0. If formula (3) is written in matrix form, formula (4) is obtained:

[0073] X=GX,stG i ≥0, diag(G)=0 (4)

[0074] According to the method name, generally It is called the self-expressive matrix. Because the weight values ​​represented by its matrix elements also imply the similarity relationship between samples, the self-expressive matrix also represents the accurate graph structure of the original data, which can be regarded as the adjacency matrix representation of the graph information. Its objective function is usually written as follows:

[0075]

[0076] In the above formula (5), the first term is the basis of the self-expression concept, that is, any sample can be represented by a combination of other samples; the second term is the sparsity constraint on the self-expression matrix, which prevents too many relationships between samples from being established during the algorithm learning process, that is, only the association relationship between the most similar samples is retained as much as possible.

[0077] Optionally, the method further includes:

[0078] The terminal device determines the loss function of the graph learning network based on the multimodal data, the functions of the multiple encoders, the self-expression matrix G and the functions of the multiple decoders.

[0079] Optionally, the loss function of the graph learning network is as shown in formula (6):

[0080]

[0081] Where G≥0, diag(G)=0, X (v) represents the first modal data, represents the decoder function of the vth modal data, represents the encoder function of the vth modal data, γ is the penalty parameter, represents the square of the Frobenius norm. The first term in the loss function (6) is the reconstruction loss of the autoencoder, the second term is the reconstruction loss of the self-expression idea, and the third term is the sparsity constraint on the self-expression matrix.

[0082] Specifically, if the encoder and decoder functions of the autoencoder on the modality v are and Then the modality v specific hidden code here can be expressed as Based on the assumption that data between multiple modalities should share the same graph structure, the self-expression matrix should also be the same, so the loss function of the graph learning network can be obtained as shown in formula (6).

[0083] Optionally, the terminal device determines the graph structure of the multimodal data according to the multimodal data and the graph learning network, including:

[0084] Initializing the graph learning network and the self-expression matrix;

[0085] Calculating the loss function value of the graph learning network according to the multimodal data;

[0086] Determining whether the loss function of the graph learning network converges according to the loss function value;

[0087] When the loss function converges, the graph structure of the multimodal data is determined according to the self-expression matrix G.

[0088] Optionally, the method further includes:

[0089] When the loss function does not converge, the gradient is calculated according to the back-propagation process and updated iteratively.

[0090] Specifically, the embodiment of the present application adopts an alternating direction optimization algorithm to iteratively update the model parameters of the graph learning network. For a given multimodal data set, after initializing the graph learning network and the self-expression matrix G, forward propagation is performed according to the corresponding network structure; the loss function is calculated according to formula (6) to determine whether it converges: if the loss function converges, the multimodal consistency graph structure A is calculated; if the loss function does not converge, the gradient is calculated according to the back propagation process of the graph learning network and iteratively updated.

[0091] Optionally, determining whether the loss function of the graph learning network converges according to the loss function value includes:

[0092] Calculate the difference between the loss function value and the loss function value calculated last time;

[0093] When the absolute value of the difference is less than a given threshold, it is determined that the loss function of the graph learning network converges.

[0094] Specifically, if the absolute value of the difference between the currently calculated loss function value and the last calculated objective function value is less than a given threshold, the convergence condition is met; otherwise, it is considered not converged.

[0095] Optionally, when the loss function converges, determining the graph structure of the original multimodal data according to the self-expression matrix G includes:

[0096] The graph structure of the original multimodal data is calculated according to formula (7):

[0097]

[0098]

[0099] Among them, |·| is the absolute value calculation function, G ij , G i and are the corresponding element value of row i and column j of G, the row vector of G, and The i-th row and j-th column of the corresponding element value, is the conjugate matrix of matrix A, G is greater than or equal to 0.

[0100] In actual experiments, since the subsequent modules of this application further integrate and complete the modal feature information after graph embedding, the actual performance is not sensitive to the value of k. For simplicity, k is taken as 10 in this application. It is worth noting that here, according to the constraint on G in formula (6), diag(A)=0 exists.

[0101] Optionally, the terminal device performs sparseness on the graph structure to obtain a sparse graph structure.

[0102] Specifically, the terminal device sparsifies the graph structure to obtain a sparse graph structure. This is because, in the graph embedding process, there is generally a sparse requirement for the graph structure. The purpose is to ignore the relationship between noise sample points as much as possible and to reduce the amount of calculation. Therefore, after obtaining the graph structure, the graph structure is sparsified to obtain a sparse graph structure.

[0103] It should be understood that graph learning methods with similar ideas can also be used, such as directly using self-expression ideas to learn consistent graph structures on multimodal data.

[0104] It should also be understood that the self-expression idea is used to learn the modality-specific graph structure on each modality for subsequent graph embedding calculations on the corresponding modality.

[0105] The above part describes, through specific embodiments, given multimodal data that does not contain a graph structure, this application first obtains a simple multimodal consistency graph structure by combining an autoencoder structure with the idea of ​​self-expression learning. The determination of the graph structure can enable the graph information to be incorporated into the subsequent modal feature representation process, making the modal feature representation more reliable.

[0106] The following is a detailed description of how a terminal device determines the consistent feature representation of the multimodal data based on the multimodal data, the graph structure and the multimodal data consistent feature representation learning network through an embodiment.

[0107] Optionally, the multimodal data consistency feature representation learning network includes multiple graph convolutional neural networks, multiple encoders, multiple decoders, multiple fusion networks and multiple ablation networks.

[0108] The first graph convolutional neural network determines a feature representation of embedded graph information of the first modal data according to the first modal data and the graph structure, the first graph convolutional neural network is any encoder of the multiple graph convolutional neural networks, and the first modal data is data of any modality in the multimodal data.

[0109] A first encoder obtains a feature representation of the first modal data including original sample information, wherein the first encoder is any encoder among the multiple encoders.

[0110] A first fusion network fuses the feature representation of the embedded graph information of the first modality data and the feature representation of the first modality data containing original sample information to obtain a fused feature representation of the first modality data, wherein the first fusion network is any fusion network among the multiple fusion networks.

[0111] A first decoder outputs the reconstructed first modality data according to the fused feature representation of the first modality data, wherein the first decoder is any one of the multiple decoders.

[0112] The first ablation network obtains the mapped feature representation of the first modality data according to the consistent feature representation of the multimodal data.

[0113] Specifically, Figure 6 A schematic block diagram of a multimodal data consistency feature representation learning network structure provided in an embodiment of the present application.

[0114] like Figure 6 As shown in the figure, the multimodal data consistency feature represents that in the learning network, the terminal device first converts the modal data X (v) And the graph structure is input into the graph convolutional network to obtain the modal data X (v) The feature representation of the embedded graph information is obtained when the modal data X (v) In the process of embedding the feature representation of the graph information, in the modal data X (v) Using graph convolutional neural networks Combine the sparse graph structure A with the modal data X (v) As input, we can obtain the modal feature representation embedded in the graph information. As shown in formula (8):

[0115]

[0116] If the degree matrix D can be expressed by Calculated, the parameter of the lth layer of the above graph convolutional network is W l (v) , then the output of its lth layer It can be calculated by the following formula (9):

[0117]

[0118] Among them, σ(·) represents the activation function in the graph convolutional network, It represents the initial feature input of the graph convolutional network, namely X (v) Considering the constraint diag(A) = 0, in the graph convolution process, the corresponding representation of any sample is obtained only from the joint representation of other samples, and the original feature information is missing. Therefore, the embodiment of the present application uses an encoder structure to obtain its original hidden feature representation from the original feature.

[0119] The terminal device inputs the modal data and the graph structure into the graph convolutional network to obtain the feature representation of the modal embedded graph information. The feature representation of the embedded graph information of each modality lacks the feature information originally possessed by the modal data. Therefore, an automatic encoder is used to obtain the feature representation of each modality containing the original sample information from each original modal data.

[0120] The terminal device transmits the modal data X (v) Input encoder Get the modal data X (v) The feature representation containing the original sample information As shown in formula (10):

[0121]

[0122] After obtaining the modal data X (v) Modal feature representation of embedded graph information and the modal data X (v) The feature representation containing the original sample information After that, the terminal device fuses the modal data X (v) Modal feature representation of embedded graph information and the modal data X (v) The feature representation containing the original sample information A fused feature representation of each modality is obtained.

[0123] The terminal device fuses the modality data X according to the fusion network (v) Modal feature representation of embedded graph information and the modal data X (v) The feature representation containing the original sample information A fused feature representation of each modality is obtained.

[0124] In order to fuse the feature representation embedded in the graph information and feature representation containing original sample information The embodiment of the present application introduces a self-attention mechanism.

[0125] If used represents the self-attention layer, then the self-attention weight can be calculated by the following formula (11):

[0126]

[0127] Accordingly, the fused modality-specific feature representation S v It can be calculated according to the following formula (12):

[0128]

[0129] It should be understood that the purpose of the self-attention mechanism in the embodiment of the present application is to adaptively fuse the features in the original data and the data features of similar samples brought by graph embedding. Other methods such as self-weighting, fixed parameters or improved attention modules can also be used to replace the self-attention mechanism.

[0130] In order to ensure S (v) With sufficient effective information, the embodiment of the present application uses an automatic encoder structure. The automatic encoder structure also includes a decoder, and the decoder function is Then the objective function is shown in the following formula (13):

[0131]

[0132] The terminal device determines the feature representation of the embedded graph information of each modality according to the determined graph structure and each original modality data, as well as the feature representation containing the original sample information of each modality, and fuses the feature representation of the embedded graph information of each modality with the feature representation containing the original sample information to obtain a fused feature representation of each modality. The fused feature representation includes both the graph information and the original sample information, so the information of the fused feature representation is more comprehensive.

[0133] If you directly concatenate multiple modal S (v) To obtain the consistent feature representation H of multimodal data, that is, H = [S (1) , S (2) , …, S (m) ], it is obvious that the H obtained in this way has a lot of redundant or invalid information. The reason why S is directly concatenated with multiple modes is (v) Obtaining consistent features of multimodal data indicates that H has a large amount of redundant or invalid information because:

[0134] First, since the data between multiple modalities should be consistent, such as the description of the category and serial number of the same instance, the different representations of the same sample in multiple modalities should contain a lot of the same information, that is, S (1) , S (2) , …, S (m) There is implicit common feature information between them;

[0135] Secondly, this direct splicing method inevitably contains the order information of the modalities, which often has no practical use.

[0136] The redundant or invalid information contained in H will greatly affect its performance in downstream tasks. Therefore, it is necessary to remove the redundant information of the fused feature representation of each modality to obtain a consistent feature representation of the multimodal data.

[0137] Therefore, the present invention uses the information bottleneck idea to eliminate the modality-specific feature representation S (v) The redundant information in is used to obtain a simple and effective multimodal consistent feature representation H.

[0138] Optionally, the method further includes:

[0139] According to the constraint conditions of the information bottleneck idea, the loss function of the multimodal data consistency feature representation learning network is determined.

[0140] The constraints of the information bottleneck concept include:

[0141] Maximize the relationship between the fused feature representation of the first modality and the consistent feature representation of the multimodal data, minimize the relationship between the consistent feature representation of the multimodal data and the first modality data, and minimize the same amount of information between the fused feature representations of the multimodal data.

[0142] Specifically, mutual information I(·,·) is used to measure the amount of information between two independent variables. The idea of ​​the information bottleneck in the embodiment of the present application can be simply expressed as shown in the following formula (14):

[0143]

[0144] The meanings of the three terms in the formula can be understood as: maximizing the modal specific feature representation S (v) Relationship between multimodal consistent features H and minimization of H and original data X (v) relationship between them, minimizing the S (v) The same amount of information between.

[0145] However, it is not reasonable to directly optimize formula (14) as the objective function. The optimization result of the third term will violate the consistency of multimodal data, resulting in S (v) There is very little effective information, which makes the multimodal consistency feature representation H invalid in downstream tasks. Therefore, the present invention introduces the idea of ​​ablation network auxiliary information bottleneck, integrating S between multiple modalities (v) To obtain H. The difference from the traditional neural network is that the ablation network regards the modality-specific feature representation H as a variable as the input of the ablation network, calculates the loss function according to the corresponding network output and reversely constrains H for optimization update.

[0146] Specifically, according to the modality-specific feature S (v) The hypothesis that H is mapped from the current subspace can be represented by multimodal consistency features. At the same time, the variational idea is introduced to design a variational ablation network on each modality. Its output is recorded as To establish S (v) The effective connection between θ and H uses the loss function shown in the following formula (15):

[0147]

[0148] According to the nonlinear mapping of the neural network, the loss itself can be regarded as maximizing S (v) The information measure between H and H. Therefore, for ease of calculation, formula (15) will replace the first term in formula (14).

[0149] Then, the second and third items in formula (14) are optimized from the sample level. For simplicity, lowercase letters are used to replace their corresponding single independent samples, such as h and H. Based on this, the second item in formula (14) can be obtained according to the mutual information calculation formula as shown in formula (16):

[0150]

[0151] Where p(·) represents the corresponding prior probability. However, it is not easy to directly calculate formula (16), for example, it is difficult to measure the marginal distribution of p(h). Therefore, the present invention uses That is, the normal distribution replaces the calculation of p(h). In addition, considering the KL divergence D between the two KL [p(h),r(h)] is always positive, so we can get the following derivation:

[0152]

[0153] Based on this derivation, the upper bound of formula (16) can be further obtained as follows:

[0154]

[0155] According to the previous analysis, directly applied to S (v) The third term in formula (14) is obviously unreasonable. Therefore, to ensure S (v) Contains sufficient effective information. The present invention applies the mutual information constraint between modes to Because According to H mapped by neural network, on the one hand, H removes a large number of S contained in different modes. (v) On the one hand, it ensures that H has enough valid information to reconstruct different S (v) Therefore, the third term in formula (14) can be expressed as However, it is different from I(X,H (v) ) are calculated differently. and It only represents different mapping representations of H on different modes, and both should be treated equally. Therefore, the present invention uses a deep neural network combined with a mutual information calculation method to introduce a mutual information calculation network f mi (·,·) is used to measure the same amount of information between the two. Therefore, the third term of formula (14) can be written as shown in the following formula (19):

[0156]

[0157] At this point, the loss function of the multimodal data consistency feature representation learning network based on the information bottleneck is shown in the following formula (20):

[0158]

[0159] Optionally, determining the consistent feature representation of the multimodal data according to the original multimodal data, the graph structure and the multimodal data consistent feature representation learning network includes:

[0160] Initializing the multimodal data consistent feature representation learning network and the consistent feature representation of the multimodal data;

[0161] Calculating a loss function value of a learning network representing the consistency features of the multimodal data according to the original multimodal data;

[0162] Determining whether the loss function of the multimodal data consistency feature representation learning network converges according to the loss function value;

[0163] When the loss function converges, a consistent feature representation of the multimodal data is determined.

[0164] Specifically, after initializing the multimodal data consistent feature representation learning network and the multimodal data consistent feature representation H, forward propagation is performed according to the multimodal data consistent feature representation learning network structure to obtain the modality-specific feature representation S (v) The overall loss function value is calculated according to formula (20) to determine whether the loss function of the multimodal data consistency feature representation learning network converges: when the loss function converges, the modality-specific feature representation S is obtained. (v) and multimodal consistency feature representation H;

[0165] Optionally, determining whether the loss function of the multimodal data consistency feature representation learning network converges includes:

[0166] Calculate the difference between the loss function value and the loss function value calculated last time;

[0167] When the absolute value of the difference is less than a given threshold, it is determined that the multimodal data consistency feature represents that the loss function of the learning network converges.

[0168] Specifically, according to the loss function value currently calculated, the difference between the loss function value calculated last time and the absolute value of the difference is less than a given threshold, then the convergence condition is met; otherwise, it is considered that the multimodal data consistency feature indicates that the learning network has not yet converged.

[0169] If the multimodal data consistency feature representation learning network has not converged, back propagation is performed on the parameters and variables of the multimodal data consistency feature representation learning network to calculate the gradient, and the parameters and variables are updated step by step according to the alternating direction optimization algorithm:

[0170] 1. Update

[0171] 2. Update f mi (·,·);

[0172] 3. Update H.

[0173] Given the original multimodal data that does not contain a graph structure, this application first obtains a simple multimodal consistent graph structure and performs sparse processing by combining the autoencoder structure with the self-expression learning idea; then uses the graph convolutional network and self-attention mechanism to embed the graph structure and the original multimodal features to obtain specific feature representations under each modality; finally, combined with the information bottleneck idea, the redundant information between feature representations under different modalities is removed to obtain a simple and effective multimodal consistent feature representation. This representation can be used in a variety of practical scenarios, including clustering and classification tasks. For example, given a series of front and side view photos of kittens and birds, the front and side views are regarded as different modalities, and kittens and birds are regarded as two categories; first, the photos in different modalities are input into the graph learning network respectively to obtain the association between different samples of kittens and birds, that is, the graph structure; then, based on the obtained graph structure and the original sample data, with the help of a quasi-autoencoder structure, the modality-specific feature representation embedded with the graph information is obtained by fusion. In the high-dimensional space, the distance between the corresponding samples of the same type should be close, while the distance between samples of different types is far, and it contains a variety of accurate feature representations such as color, pattern, body shape, posture, presence or absence of wings, feathers or hair; then, through the information bottleneck technology, the overly redundant information between the feature representations on different modalities, such as posture, is eliminated, and only sufficiently representative information, such as color, presence or absence of wings, feathers or hair, is retained to obtain a simple and effective multi-modal consistency feature representation; due to its simple, accurate and effective characteristics, this representation has a great positive improvement effect on the performance of subsequent tasks such as clustering and classification.

[0174] The above describes a method for determining a multimodal data feature representation proposed in the present application through a graph learning network and a multimodal data consistency feature representation learning network. In order to more clearly understand the embodiment of the present application, the following describes a method 300 for determining a multimodal data feature representation proposed in the present application through a specific flow chart, such as Figure 7 As shown, Figure 7 A schematic flow chart of a method 200 for determining a feature representation of multimodal data provided in an embodiment of the present application. The method 200 can be executed by any electronic device having data processing capabilities. For example, the electronic device can be implemented as a server or a terminal device. The following description is made by taking the electronic device as a terminal device as an example. Figure 7 As shown, the method 300 may include steps S310 to S370.

[0175] S310, the terminal device performs normalization preprocessing on the multimodal data set to obtain the processed multimodal data.

[0176] S320, the terminal device establishes a graph learning network.

[0177] S330, the terminal device establishes a multimodal data consistency feature representation learning network.

[0178] S340, the terminal device determines the loss function of the graph learning network and the loss function of the multimodal data consistency feature representation learning network.

[0179] S350, the terminal device determines the loss function value of the graph learning network and the loss function value of the multimodal data consistency feature representation learning network.

[0180] S360, the terminal device determines whether the loss function of the graph learning network converges, and the terminal device determines whether the loss function of the multimodal data consistency feature representation learning network converges.

[0181] S370, the terminal device obtains a consistent feature representation of the multimodal data.

[0182] It should be understood that in S350 and S360, the terminal device will sequentially execute judgments on different networks, that is, the terminal device determines the loss function value of the graph learning network, and then determines whether the loss function of the graph learning network converges based on the loss function value of the graph learning network. When the terminal device determines that the loss function of the graph learning network converges, the terminal device determines the loss function value of the multimodal data consistency feature representation learning network, and then determines whether the loss function of the multimodal data consistency feature representation learning network converges based on the loss function value of the multimodal data consistency feature representation learning network.

[0183] The above method 300 can be understood by referring to the specific step details in method 200. In order to avoid repetition, it will not be repeated here.

[0184] Through the above method 300, the terminal device determines the graph structure of the multimodal data based on the multimodal data and the graph learning network, and the terminal device determines the consistent feature representation of the multimodal data based on the multimodal data, the graph structure and the multimodal data consistent feature representation learning network. The multimodal data consistent feature representation learning network combines the information bottleneck idea to make the consistent feature representation of the multimodal data more effective and accurate. The multimodal consistent feature representation, as a single-modal high-level semantic feature corresponding to the sample, can be applied to a variety of downstream tasks, such as clustering, classification, etc.

[0185] In order to verify the performance of a method for determining the consistent feature representation of multimodal data proposed in an embodiment of the present application, it is compared with 11 currently more representative multimodal representation learning models on 5 data sets in clustering and classification tasks. In the clustering task, the multimodal consistent feature representations obtained by each algorithm are clustered by K-means, and the model performance is verified using standardized mutual information, F-score and Rand index; in the classification task, a 70% / 30% training set / test set ratio is set, and the K nearest neighbor classifier is used for classification, and then the accuracy, precision and F-score are used to characterize the effectiveness of the experimental results. Each group of experiments was repeated 10 times and the mean and standard deviation were taken. It is worth noting that the standard deviation of the results of each algorithm in the classification task is less than 0.01, so it is not given. The larger the indicators, the better the experimental results.

[0186] The experimental comparison methods are:

[0187] 1. Multi-view Clustering via Joint nonnegativematrix factorization (MultiNMF);

[0188] 2. Graph Regularized Nonnegative Matrix Factorization (GNMF);

[0189] 3. Partition level Multiview Subspace Clustering (PMSC);

[0190] 4. Generalized Canonical Correlation Analysis (GCCA);

[0191] 5. Deep Generalized Canonical Correlation Analysis (DGCCA);

[0192] 6. Deep Canonically Correlated Autoencoders (DCCAE);

[0193] 7. Nested Autoencoders (Autoencoder in Autoencoder Networks, AE2-Nets);

[0194] 8. Multi-view Information Bottleneck (MIB);

[0195] 9. Collaborative Multi-View Information Bottleneck Networks (CMIB-Nets);

[0196] 10. VAE-based multi-view clustering framework (Multi-VAE) based on variational autoencoder;

[0197] 11. Collaborative Unsupervised Multi-View Representation Learning (GUMRL).

[0198] The data sets used in the experiment are:

[0199] 1. Handwritten digital data set (Handwritten)

[0200] 2. Berkeley Drosophila Genome Project Dataset

[0201] 3. Cambridge University Face Dataset (Orl)

[0202] 4. Yale University Face Dataset (Yale)

[0203] 5. Notting Hill Movie Face Dataset (Notting-Hill)

[0204] The information is shown in Table 1:

[0205] Table 1 Dataset description

[0206] Dataset Number of instances Number of categories Mode number Handwritten 2000 10 2 BDGP 2500 5 2 Orl 400 40 3 Yale 165 15 3 Notting-Hill 110 5 3

[0207] Tables 2 to 6 and Tables 7 to 11 respectively give the clustering and classification experimental comparison results of the model of the embodiment of the present application and other comparative models on 5 data sets. The experiments show that the clustering performance of the model of the present invention on all data sets is significantly better than that of other models. This proves that the model makes full use of the learned graph structure knowledge and borrows the information bottleneck method to effectively obtain a simple and accurate multimodal consistency feature representation, and this representation can also play an important role in downstream tasks. Table 12 gives the modality-specific features S obtained by the present invention on 4 data sets. (v)The clustering results with the multimodal consistency feature H prove that the multimodal feature fusion method based on the information bottleneck idea is effective.

[0208] Table 2 Comparison of average clustering performance of each model on the Handwritten dataset (mean ± standard deviation)

[0209]

[0210] Table 3 Comparison results of average clustering performance of each model on BDGP dataset (mean ± standard deviation)

[0211]

[0212]

[0213] Table 4 Comparison results of average clustering performance of each model on the Orl dataset (mean ± standard deviation)

[0214]

[0215] Table 5 Comparison results of average clustering performance of each model on the Yale dataset (mean ± standard deviation)

[0216]

[0217] Table 6 Comparison results of average clustering performance of each model on Notting-Hill dataset (mean ± standard deviation)

[0218]

[0219] Table 7 Comparison of the average classification performance of each model on the Handwritten dataset (mean ± standard deviation)

[0220]

[0221] Table 8 Comparison results of average classification performance of each model on BDGP dataset (mean ± standard deviation)

[0222]

[0223] Table 9 Comparison results of average classification performance of each model on the Orl dataset (mean ± standard deviation)

[0224]

[0225] Table 10 Comparison of the average classification performance of each model on the Yale dataset (mean ± standard deviation)

[0226]

[0227]

[0228] Table 11 Comparison of the average classification performance of each model on the Notting-Hill dataset (mean ± standard deviation)

[0229]

[0230] Table 12 Modality-specific feature representation S on four datasets (v) Comparison of clustering performance with multimodal consistent feature representation H (mean)

[0231]

[0232]

[0233] The specific implementation modes of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above implementation modes. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific implementation modes can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different implementation modes of the present application can also be arbitrarily combined, and as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0234] It should also be understood that in the various method embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. It should be understood that these sequence numbers can be interchanged where appropriate, so that the embodiments of the present application described can be implemented in an order other than those shown or described.

[0235] Combination of the above Figures 1 to 7 , describes in detail the method embodiment of the present application, and the following is combined with Figure 8 and Fig. 9 , describe in detail the device embodiments of the present application.

[0236] Figure 8 4 is a schematic block diagram of an apparatus 400 according to an embodiment of the present application, and the apparatus 400 can implement the functions of the terminal device in the above method. Figure 8 As shown, the apparatus 400 may include a processing unit 410 and an acquisition unit 420 .

[0237] The processing unit 410 is used to determine the graph structure of the multimodal data according to the multimodal data and the graph learning network;

[0238] The processing unit 410 is further configured to determine a consistent feature representation of the multimodal data according to the multimodal data, the graph structure and a multimodal data consistent feature representation learning network,

[0239] Among them, the multimodal data consistency feature representation learning network combines the information bottleneck idea.

[0240] In one embodiment, the processing unit 410 is further used to perform normalization processing on the multimodal data to obtain the processed multimodal data;

[0241] The processing unit 410 is further configured to:

[0242] Determining a graph structure of the multimodal data according to the processed multimodal data and the graph learning network;

[0243] The processing unit 410 is further configured to:

[0244] The consistent feature representation of the multimodal data is determined based on the processed multimodal data, the graph structure and the multimodal data consistent feature representation learning network.

[0245] In one embodiment, the graph learning network includes multiple encoders, fully connected layers and multiple decoders, the multiple encoders correspond to the multiple decoders one by one, and the fully connected layer is constructed according to the self-expression matrix G.

[0246] The first encoder obtains the hidden code of the first modal data, the hidden code is input into the fully connected layer for self-expression learning, the fully connected layer outputs the hidden code after self-expression learning to the first decoder, and the first decoder outputs the reconstructed first modal data.

[0247] The first encoder is any encoder among the multiple encoders, the first decoder is any decoder among the multiple decoders, and the first modal data is data of any modality among the multi-modal data.

[0248] In one embodiment, the processing unit 410 is further configured to:

[0249] Initializing the graph learning network and the self-expression matrix;

[0250] Calculating the loss function value of the graph learning network according to the multimodal data;

[0251] Determining whether the loss function of the graph learning network converges according to the loss function value;

[0252] When the loss function converges, the graph structure of the multimodal data is determined according to the self-expression matrix G.

[0253] In one embodiment, the processing unit 410 is further configured to:

[0254] The graph structure of the original multimodal data is calculated according to formula (21):

[0255]

[0256]

[0257] Among them, |·| is the absolute value calculation function, G ij , G i and are the corresponding element value of row i and column j of G, the row vector of G, and The i-th row and j-th column of the corresponding element value, is the conjugate matrix of matrix A, G is greater than or equal to 0.

[0258] In one embodiment, the processing unit 410 is further configured to:

[0259] The loss function of the graph learning network is determined based on the multimodal data, the functions of the multiple encoders, the self-expression matrix G, and the functions of the multiple decoders.

[0260] In one embodiment, the loss function of the graph learning network is as shown in formula (22):

[0261]

[0262] Where G≥0, diag(G)=0, X (v) represents the first modal data, represents the decoder function for the first modality data, represents the encoder function of the first modal data, γ is the penalty parameter, Represents the square of the Frobenius norm.

[0263] In one embodiment, the multimodal data consistent feature representation learning network includes multiple graph convolutional neural networks, multiple encoders, multiple decoders, multiple fusion networks and multiple ablation networks.

[0264] in,

[0265] The first graph convolutional neural network determines a feature representation of the embedded graph information of the first modal data according to the first modal data and the graph structure, the first graph convolutional neural network is any encoder of the multiple graph convolutional neural networks, and the first modal data is data of any modality in the multimodal data.

[0266] A first encoder obtains a feature representation of the first modal data including original sample information, wherein the first encoder is any encoder among the multiple encoders.

[0267] A first fusion network fuses the feature representation of the embedded graph information of the first modality data and the feature representation of the first modality data containing original sample information to obtain a fused feature representation of the first modality data, wherein the first fusion network is any fusion network among the multiple fusion networks.

[0268] A first decoder outputs the reconstructed first modality data according to the fused feature representation of the first modality data, wherein the first decoder is any one of the multiple decoders.

[0269] The first ablation network obtains the mapped feature representation of the first modality data according to the consistent feature representation of the multimodal data.

[0270] In one embodiment, the processing unit 410 is further configured to: determine the consistent feature representation of the multimodal data according to the original multimodal data, the graph structure and the multimodal data consistent feature representation learning network, including:

[0271] Initializing the multimodal data consistent feature representation learning network and the consistent feature representation of the multimodal data;

[0272] Calculating a loss function value of a learning network representing the consistency features of the multimodal data according to the original multimodal data;

[0273] Determining whether the loss function of the multimodal data consistency feature representation learning network converges according to the loss function value;

[0274] When the loss function converges, a consistent feature representation of the multimodal data is determined.

[0275] In one embodiment, the processing unit 410 is further configured to:

[0276] According to the constraint conditions of the information bottleneck idea, the loss function of the multimodal data consistency feature representation learning network is determined.

[0277] The constraints of the information bottleneck concept include:

[0278] Maximize the relationship between the fused feature representation of the first modality and the consistent feature representation of the multimodal data, minimize the relationship between the consistent feature representation of the multimodal data and the first modality data, and minimize the same amount of information between the fused feature representations of the multimodal data.

[0279] In one embodiment, the processing unit 410 is further configured to:

[0280] Calculate the difference between the loss function value and the loss function value calculated last time;

[0281] When the absolute value of the difference is less than a given threshold, it is determined that the multimodal data consistency feature represents that the loss function of the learning network converges.

[0282] In one embodiment, the acquisition unit 420 is further used to: acquire the multimodal data.

[0283] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, when the data processing device 400 in this embodiment may correspond to the execution subject of the method 200 of the embodiment of the present application, the aforementioned and other operations and / or functions of each module in the device 400 are respectively to implement Figure 4 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0284] The above describes the device and system of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0285] like Fig. 9 It is a schematic block diagram of an electronic device 500 provided in an embodiment of the present application.

[0286] like Fig. 9 As shown, the electronic device 500 may include:

[0287] The memory 510 and the processor 520, the memory 510 is used to store the computer program and transmit the program code to the processor 520. In other words, the processor 520 can call and run the computer program from the memory 510 to implement the method in the embodiment of the present application.

[0288] For example, the processor 520 may be used to execute the steps of each execution subject in the above method 300 according to the instructions in the computer program.

[0289] In some embodiments of the present application, the processor 520 may include but is not limited to:

[0290] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0291] In some embodiments of the present application, the memory 510 includes but is not limited to:

[0292] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0293] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 510 and executed by the processor 520 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 500.

[0294] Optionally, the electronic device 500 may further include:

[0295] The communication interface 530 may be connected to the processor 520 or the memory 510 .

[0296] The processor 520 may control the communication interface 530 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. Exemplarily, the communication interface 530 may include a transmitter and a receiver. The communication interface 530 may further include an antenna, and the number of antennas may be one or more.

[0297] It should be understood that the various components in the electronic device 500 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0298] According to one aspect of the present application, a communication device is provided, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory, so that the encoder executes the method of the above method embodiment.

[0299] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0300] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the above method embodiment.

[0301] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0302] It should be understood that in the embodiment of the present application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based only on A, and B can also be determined based on A and / or other information.

[0303] In the description of the present application, unless otherwise specified, "at least one" means one or more, and "plurality" means two or more than two. In addition, "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0304] It should also be understood that the first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects, without any distinction of order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0305] It should also be understood that the specific features, structures or characteristics related to the embodiments in the specification are included in at least one embodiment of the present application. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0306] In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or device.

[0307] It is understandable that in the specific implementation of this application, user information and other related data may be involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0308] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0309] In the several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0310] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0311] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining consistent feature representation of multimodal data, characterized in that: include: Determining a graph structure of the multimodal data based on the multimodal data and the graph learning network; Determining a consistent feature representation of the multimodal data according to the multimodal data, the graph structure and a multimodal data consistent feature representation learning network, Among them, the multimodal data consistency feature representation learning network combines the information bottleneck idea.

2. The method according to claim 1, characterized in that The graph learning network includes multiple encoders, fully connected layers and multiple decoders, the multiple encoders correspond to the multiple decoders one by one, and the fully connected layer is constructed according to the self-expression matrix G. The first encoder obtains the hidden code of the first modal data, the hidden code is input into the fully connected layer for self-expression learning, the fully connected layer outputs the hidden code after self-expression learning to the first decoder, and the first decoder outputs the reconstructed first modal data. The first encoder is any encoder among the multiple encoders, the first decoder is any decoder among the multiple decoders, and the first modal data is data of any modality among the multi-modal data.

3. The method according to claim 2, characterized in that Determining the graph structure of the multimodal data according to the multimodal data and the graph learning network includes: Initializing the graph learning network and the self-expression matrix; Calculating the loss function value of the graph learning network according to the multimodal data; Determining whether the loss function of the graph learning network converges according to the loss function value; When the loss function converges, the graph structure of the multimodal data is determined according to the self-expression matrix G.

4. The method according to claim 3, characterized in that When the loss function converges, determining the graph structure of the multimodal data according to the self-expression matrix G includes: The graph structure of the multimodal data is calculated according to formula (1): Among them, |·| is the absolute value calculation function, G ij , G i and are the corresponding element value of row i and column j of G, the row i vector of G, and The i-th row and j-th column of the corresponding element value, is the conjugate matrix of matrix A, G is greater than or equal to 0.

5. The method according to claim 3, characterized in that: The method further comprises: The loss function of the graph learning network is determined based on the multimodal data, the functions of the multiple encoders, the self-expression matrix G, and the functions of the multiple decoders.

6. The method according to claim 5, characterized in that The loss function of the graph learning network is shown in formula (2): Where g≥0, diag(G)=0, X (v) represents the first modal data, represents the decoder function for the first modality data, represents the encoder function of the first modal data, γ is the penalty parameter, Represents the square of the Frobenius norm.

7. The method according to claim 1, characterized in that The multimodal data consistency feature representation learning network includes multiple graph convolutional neural networks, multiple encoders, multiple decoders, multiple fusion networks and multiple ablation networks. in, The first graph convolutional neural network determines a feature representation of the embedded graph information of the first modal data according to the first modal data and the graph structure, the first graph convolutional neural network is any encoder of the multiple graph convolutional neural networks, and the first modal data is data of any modality in the multimodal data. A first encoder obtains a feature representation of the first modal data including original sample information, wherein the first encoder is any encoder among the multiple encoders. A first fusion network fuses the feature representation of the embedded graph information of the first modality data and the feature representation of the first modality data containing original sample information to obtain a fused feature representation of the first modality data, wherein the first fusion network is any fusion network among the multiple fusion networks. A first decoder outputs the reconstructed first modality data according to the fused feature representation of the first modality data, wherein the first decoder is any one of the multiple decoders. The first ablation network obtains the mapped feature representation of the first modality data according to the consistent feature representation of the multimodal data.

8. The method according to claim 7, characterized in that Determining a consistent feature representation of the multimodal data according to the original multimodal data, the graph structure, and a multimodal data consistent feature representation learning network includes: Initializing the multimodal data consistent feature representation learning network and the consistent feature representation of the multimodal data; Calculating a loss function value of a learning network representing the consistency features of the multimodal data according to the original multimodal data; Determining whether the loss function of the multimodal data consistency feature representation learning network converges according to the loss function value; When the loss function converges, a consistent feature representation of the multimodal data is determined.

9. The method according to claim 8, characterized in that The method further comprises: According to the constraint conditions of the information bottleneck idea, the loss function of the multimodal data consistency feature representation learning network is determined. The constraints of the information bottleneck concept include: Maximize the relationship between the fused feature representation of the first modality and the consistent feature representation of the multimodal data, minimize the relationship between the consistent feature representation of the multimodal data and the first modality data, and minimize the same amount of information between the fused feature representations of the multimodal data.

10. The method according to claim 8, characterized in that The determining whether the loss function of the multimodal data consistency feature representation learning network converges includes: Calculate the difference between the loss function value and the loss function value calculated last time; When the absolute value of the difference is less than a given threshold, it is determined that the multimodal data consistency feature represents that the loss function of the learning network converges.

11. The method according to claim 1, characterized in that: The method further comprises: Normalizing the multimodal data to obtain processed multimodal data; Determining the graph structure of the multimodal data according to the multimodal data and the graph learning network includes: Determining a graph structure of the multimodal data based on the processed multimodal data and the graph learning network; The step of determining the consistent feature representation of the multimodal data according to the multimodal data, the graph structure and the multimodal data consistent feature representation learning network includes: The consistent feature representation of the multimodal data is determined based on the processed multimodal data, the graph structure and the multimodal data consistent feature representation learning network.

12. A device for determining consistent feature representation of multimodal data, characterized in that: include: A processing unit, configured to determine a graph structure of the multimodal data based on the multimodal data and the graph learning network; The processing unit is also used to determine the consistent feature representation of the multimodal data based on the multimodal data, the graph structure and the multimodal data consistent feature representation learning network.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores instructions, and when the processor runs the instructions, the processor executes the method according to any one of claims 1 to 11.

14. A computer storage medium, characterized in that: The method comprises instructions which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 11.

Images