A Multimodal Semi-Supervised Representation Learning Method for Hypergraph Variational Autoencoders

Abstract

The present application relates to a multimodal semi-supervised representation learning method for a hypergraph variational autoencoder. The method comprises: constructing a multimodal hypergraph using multimodal data to calculate an original Laplace relationship matrix; constructing a single-layer hypergraph convolution using the original Laplace relationship matrix and extracting a potential fusion representation; introducing standard Gaussian distribution prior information into the fusion representation using a variational inference model and inferring a joint representation result; performing an inner product calculation on the joint representation result to generate a reconstructed Laplace relationship matrix; generating difference information between the reconstructed Laplace relationship matrix and the original Laplace relationship matrix using a KL divergence constraint; using cross entropy to measure the loss between the label of the multimodal data and the fusion representation based on the difference information, initializing the training parameters of the variational autoencoder, and performing gradient descent processing on the optimized objective function using the Adam algorithm to update the joint representation result. This method significantly improves the multimodal data processing performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of data fusion processing and relates to a multimodal semi-supervised representation learning method of a hypergraph variational autoencoder. Background Art

[0002] With the development of modern intelligent technology, the sources and types of data are increasing, and the requirements for the fusion and application of multi-source data are becoming increasingly demanding. In real-world environments, intelligent devices perceive the external environment through multimodal data. For example, multi-source information fusion is performed based on multimodal data generated by image vision, radar detection, and laser detection, and the fused multimodal data is used to perceive the external environment. For example, in scenarios where radar and laser jointly identify targets, the core issue is how to fuse the target features of radar and laser to improve recognition accuracy. How to fuse feature data from different sensors or different modalities is a core issue in multimodal learning. The current mainstream research work focuses on using deep networks and labels to learn potential fusion representations.

[0003] The amount of data from radar and laser sensors is larger than that from a single sensor, and there are practical problems such as higher difficulty in data registration and labeling. Although multimodal or multi-sensor data can provide multi-source information and reveal the complementary relationship between modalities in the real world, the data labeling process is more time-consuming and expensive than normal single-modal data. Therefore, it is necessary to study how to mine the interaction of multimodal data under missing labels under semi-supervised conditions. At present, in the process of fusing the features of multimodal data using multimodal learning methods, existing variational autoencoders mostly focus on how to use the low-order correlation of multimodal data to mine and utilize the interaction of multimodal data. However, in the process of realizing the present invention, the inventors found that in the traditional multimodal data processing technology, there is a technical problem of low multimodal data processing performance. Summary of the Invention

[0004] In response to the problems existing in the above-mentioned traditional methods, the present invention proposes a multimodal semi-supervised representation learning method of a hypergraph variational autoencoder that can significantly improve the multimodal data processing performance, a multimodal semi-supervised representation learning device of a hypergraph variational autoencoder, a computer device and a computer-readable storage medium.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] On the one hand, a multimodal semi-supervised representation learning method for a hypergraph variational autoencoder is provided, comprising the steps of:

[0007] Acquire multimodal data; multimodal data includes radar feature data, laser feature data, and image feature data;

[0008] Constructing multimodal hypergraphs using multimodal data;

[0009] The original Laplace relationship matrix is ​​calculated based on the multimodal hypergraph;

[0010] The original Laplace relationship matrix is ​​used to construct a single-layer hypergraph convolution, and the potential fusion representation is extracted based on the single-layer hypergraph convolution;

[0011] The variational inference model is used to introduce standard Gaussian distribution prior information into the fusion representation, and the joint representation of multimodal data is obtained by inference.

[0012] Perform inner product calculation on the joint representation results to generate a reconstructed Laplace relation matrix;

[0013] The KL divergence constraint is used to generate the difference information between the reconstructed Laplace relationship matrix and the original Laplace relationship matrix;

[0014] According to the difference information, the cross entropy is used to measure the loss between the semi-supervised label and the fusion representation of the multimodal data to obtain the optimized objective function;

[0015] Initialize the training parameters of the variational autoencoder and use the Adam algorithm to perform gradient descent on the optimized objective function to update the joint characterization result; the joint characterization result is used for multi-sensor joint target recognition.

[0016] On the other hand, a multimodal semi-supervised representation learning device for a hypergraph variational autoencoder is also provided, comprising:

[0017] A data acquisition module is used to acquire multimodal data; the multimodal data includes radar feature data, laser feature data and image feature data;

[0018] Hypergraph construction module, used to construct multimodal hypergraph using multimodal data;

[0019] A matrix calculation module is used to calculate the original Laplace relation matrix based on the multimodal hypergraph;

[0020] The convolution fusion module is used to construct a single-layer hypergraph convolution using the original Laplace relationship matrix and extract the potential fusion representation based on the single-layer hypergraph convolution;

[0021] The joint representation module is used to introduce standard Gaussian distribution prior information into the fusion representation using the variational inference model, and infer the joint representation results of multimodal data;

[0022] The matrix reconstruction module is used to perform inner product calculation on the joint representation results and generate a reconstructed Laplace relation matrix;

[0023] A difference calculation module is used to generate difference information between the reconstructed Laplace relationship matrix and the original Laplace relationship matrix using the KL divergence constraint;

[0024] The objective function module is used to obtain the optimized objective function based on the difference information and the cross entropy measurement of the loss between the semi-supervised label and the fusion representation of the multimodal data;

[0025] The representation update module is used to initialize the training parameters of the variational autoencoder and use the Adam algorithm to perform gradient descent on the optimized objective function to update the joint representation result; the joint representation result is used for multi-sensor joint target recognition.

[0026] On the other hand, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned multimodal semi-supervised representation learning method of a hypergraph variational autoencoder are implemented.

[0027] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal semi-supervised representation learning method of the above-mentioned hypergraph variational autoencoder are implemented.

[0028] One of the above technical solutions has the following advantages and beneficial effects:

[0029] The aforementioned multimodal semi-supervised representation learning method using a hypergraph variational autoencoder (VVA) employs a general framework based on VVAs. First, multimodal data is acquired and a multimodal hypergraph is constructed. The original Laplacian matrix is ​​then calculated, and a single-layer hypergraph convolution is constructed to extract the underlying fused representation. A variational inference model is then used to introduce standard Gaussian distribution prior information into the fused representation, inferring the joint representation of the multimodal data. The joint representation is then used to reconstruct the Laplacian relationship matrix. After generating the difference between the two Laplacian matrices using the KL divergence constraint, the cross-entropy measure is used to measure the loss between the semi-supervised labels of the multimodal data and the fused representation, resulting in an optimized objective function. Finally, the VVA training parameters are initialized, and the Adam algorithm is used to perform gradient descent on the optimized objective function. The joint representation is then updated, completing the multimodal data fusion process.

[0030] Compared to existing technologies, this method proposes a hypergraph variational autoencoder for multimodal semi-supervised representation. This method automatically encodes high-order correlations in multimodal data and effectively learns the underlying fusion representation by introducing standard Gaussian distribution prior information through variational inference. The hypergraph variational autoencoder is more versatile, capable of processing complex multimodal data and fusing feature data from different modalities using only a small number of semi-supervised labels. This significantly improves multimodal data processing performance, effectively enhancing the performance of data fusion, such as radar and laser features, and enhancing the accuracy of joint target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 1 is a flow chart of a multimodal semi-supervised representation learning method for a hypergraph variational autoencoder in one embodiment;

[0033] Figure 2 Schematic diagram of the process of constructing a multimodal hypergraph in one embodiment;

[0034] Figure 3 1 is a flowchart of a multimodal semi-supervised representation learning method for a hypergraph variational autoencoder in one embodiment;

[0035] Figure 4 2. t-SNE visualization diagram of an embodiment verified on the ModelNet40 dataset; wherein (a) is the original feature of modality one, (b) is the original feature of modality two, and (c) is the obtained potential fusion representation;

[0036] Figure 5 The accuracy curves of the training set and the test set on the ModelNet40 dataset in one embodiment are shown;

[0037] Figure 6 Schematic diagram of the module structure of a multimodal semi-supervised representation learning device of a hypergraph variational autoencoder in one embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0040] It should be noted that reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of such a phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments.

[0041] Those skilled in the art will appreciate that the embodiments described herein may be combined with other embodiments. The term "and / or" used in the present specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0042] A challenging problem in multimodal semi-supervised learning is how to match the complex correlations under modalities. Existing methods mostly focus on using the low-order correlations of multimodal data to learn interactions within the modalities. Faced with the problem of feature fusion in the process of joint target recognition by radar and laser, the mainstream research work is to use deep networks and labels to learn potential fusion representations. At present, hypergraph learning has been used to simulate high-order correlations between data in different tasks, and a hypergraph neural network has been proposed to learn hidden layer representations by processing the interactions between data and deployed on visual target recognition tasks. In the research work, the inventors found that the hypergraph structure can mine the high-order correlations between multimodal data. Therefore, the high-order correlations between multimodal data and the potential fusion representations that can be learned by the neural network are the important basis and support for the research of this application, which is of great significance for the subsequent introduction of joint representation of radar and laser target characteristics.

[0043] Traditional hypergraph neural network methods ignore prior knowledge about the correlations of multimodal data, which can be introduced to improve data fitting capabilities. Therefore, this application introduces a variational inference framework into the hypergraph structure, capturing high-order correlations while also incorporating prior knowledge to achieve efficient joint representation of multimodal data. This approach can then be applied to radar and laser joint target recognition, effectively improving recognition accuracy.

[0044] Alternatively, multimodal generative models can be used to learn joint representations from the latent space, deriving meaningful latent representations from each modality through the structure of an autoencoder. On the one hand, generative models can not only obtain low-dimensional representations from high-dimensional modalities but can also be used as unsupervised methods to transfer to semi-supervised tasks. On the other hand, by fitting the distribution, variational inference techniques can be used to introduce additional priors. However, these generative model approaches have difficulty encoding the high-order correlations in multimodal data and also ignore these relationships between multimodal data.

[0045] This application performs variational reasoning on the hypergraph structure, which can not only mine high-order relationships between multimodal data, but also introduces the standard Gaussian distribution as prior knowledge (information) for the potential joint representation of multimodal fusion. It also uses the variational lower bound to narrow the gap between the training set and the validation set of the semi-supervised task, ensuring that the features of multimodal data can be effectively jointly represented under semi-supervised conditions. Therefore, for the application scenario of radar and laser joint target recognition, the use of the hypergraph variational autoencoder proposed in this application can effectively improve the performance of feature fusion of radar and laser, thereby improving the accuracy of joint target recognition.

[0046] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings.

[0047] See also Figure 1 In one embodiment, the present application provides a multimodal semi-supervised representation learning method for a hypergraph variational autoencoder, comprising the following processing steps S12 to S28:

[0048] S12, acquiring multimodal data; the multimodal data includes radar feature data, laser feature data, and image feature data.

[0049] It is understood that multimodal data can include radar feature data output by radar detection, laser features output by laser sensor detection, and image feature data of visual images. It can also include feature data of other sensors not described in this specification. The specific data modality can be determined based on the data modalities included in the multi-source data generated in the actual application scenario. Multimodal data can be pre-collected and stored in a local or cloud database for retrieval and use, or it can be collected online through connected sensors. The specific data supply can be determined based on the actual data supply of the application scenario.

[0050] S14, construct a multimodal hypergraph using multimodal data.

[0051] It can be understood that for the sake of convenience in describing the acquired multimodal data, for example, the m modal feature data included in the multimodal data can be represented as X=(X1,...,X mAfter obtaining the modal feature data of multiple modalities, these modal feature data can be used to construct a hypergraph, thereby obtaining a multimodal hypergraph structure representation.

[0052] S16, calculate the original Laplace relationship matrix according to the multimodal hypergraph.

[0053] It can be understood that after constructing the multimodal hypergraph, the hypergraph index matrix of the multimodal hypergraph can be used to calculate the Laplace relationship matrix. The Laplace relationship matrix obtained here is called the original Laplace relationship matrix, which can be used to represent the association relationship between each vertex in the multimodal hypergraph.

[0054] S18, uses the original Laplace relationship matrix to construct a single-layer hypergraph convolution, and extracts the potential fusion representation based on the single-layer hypergraph convolution.

[0055] It can be understood that after obtaining the original Laplace relationship matrix of the hypergraph, the matrix can be used to define a single-layer hypergraph convolution, that is, to construct the aforementioned single-layer hypergraph convolution, and then use the single-layer hypergraph convolution to extract the latent variable representation under each modality to obtain the potential fusion representation.

[0056] S20, using the variational inference model to introduce standard Gaussian distribution prior information into the fusion representation, and inferring the joint representation result of multimodal data.

[0057] It can be understood that the variational inference model, that is, the existing variational inference algorithm, can introduce prior knowledge into the fused latent variable representation (that is, the aforementioned potential fusion representation), such as the aforementioned standard Gaussian distribution prior information (that is, the prior information with the standard Gaussian distribution as prior knowledge), as the prior knowledge of the potential joint representation of multimodal fusion, thereby utilizing the variational lower bound of the variational inference model to narrow the gap between the training set and the validation set of the semi-supervised task, and ensuring that under semi-supervised conditions, the characteristics of the multimodal data are effectively jointly represented, and the joint representation results of the multimodal data are obtained, thereby realizing the joint representation of the characteristics of the multimodal data.

[0058] S22, performing inner product calculation on the joint representation result to generate a reconstructed Laplace relation matrix.

[0059] It can be understood that the inner product calculation of the joint representation result and the reconstruction of the Laplace relationship matrix can be expressed as:

[0060]

[0061] Where Z represents the potential fusion representation, L represents the original Laplace relationship matrix, N represents the number of modal features, and z i and z j represents the feature vector of the joint representation of each fused feature, σ(·) denotes the activation function and T denotes transpose.

[0062] S24, using the KL divergence constraint to generate difference information between the reconstructed Laplace relationship matrix and the original Laplace relationship matrix.

[0063] It can be understood that after obtaining the reconstructed Laplace relationship matrix and the original Laplace relationship matrix, the difference L between the reconstructed Laplace relationship matrix and the original Laplace relationship matrix is ​​generated using the KL (Kullback-Leibler) divergence constraint. h , that is, L h =E q(Z|X，L) [log p(L|Z)]-KL[q(Z|X,L)||p(Z)], where KL(·||·) is the KL divergence, which can exploit the gap between the distribution q(·) and the Gaussian prior p(·), and E q(·) [log p(·)] is the reconstruction error of the hypergraph.

[0064] S26, based on the difference information, uses cross entropy to measure the loss between the semi-supervised label and the fusion representation of the multimodal data to obtain the optimized objective function.

[0065] It can be understood that the semi-supervised labels of multimodal data can be provided by manual pre-labeling to mark the category of the target identified by the corresponding multi-sensor. After obtaining the difference between the reconstructed Laplace relationship matrix and the original Laplace relationship matrix, the cross entropy function is used to measure the loss between the semi-supervised labels of the multimodal data and the obtained fusion features (i.e., the fusion representation Z). The final loss function is the sum of the reconstruction loss and the cross entropy loss L all =λL s +(1-λ)L h , where L s represents the loss function of cross entropy, and λ is the hyperparameter of learning and training. Therefore, the optimized objective function is L all .

[0066] S28, initialize the training parameters of the variational autoencoder and use the Adam algorithm to perform gradient descent processing on the optimized objective function to update the joint characterization result; the joint characterization result is used for multi-sensor joint target recognition.

[0067] It can be understood that the Adam (Adaptive momentum) algorithm is a random optimization method for adaptive momentum. The optimized objective function L of the variational autoencoder is obtained all After that, the training parameters can be initialized and the Adam algorithm can be used to perform gradient descent optimization of the loss function L all, updating the joint representation results, such as but not limited to the representation results of the fusion of the radar's one-dimensional range image features and the laser's point cloud features, the fusion representation results of the image's visual features and the natural language word vector features, etc. Since the variational autoencoder of this embodiment is a hypergraph variational autoencoder, it can mine the high-order correlations of multimodal data based on the hypergraph structure during data processing, and introduces additional prior knowledge to infer the multimodal fusion representation, which can efficiently realize the joint representation of multimodal data features under semi-supervised conditions. Therefore, using these joint representation results can effectively improve the performance of radar and laser feature fusion, thereby improving the accuracy of joint target recognition.

[0068] This method for learning multimodal semi-supervised representations using a hypergraph variational autoencoder (VVA) employs a general framework based on VVAs. First, multimodal data is acquired and a multimodal hypergraph is constructed. The original Laplacian matrix is ​​then calculated, and a single-layer hypergraph convolution is constructed to extract the underlying fused representation. A variational inference model is then used to introduce standard Gaussian distribution prior information into the fused representation, inferring the joint representation of the multimodal data. The joint representation is then used to reconstruct the Laplacian relationship matrix. After generating the difference between the two Laplacian matrices using the KL-divergence constraint, the cross-entropy is used to measure the loss between the semi-supervised labels of the multimodal data and the fused representation, resulting in an optimized objective function. Finally, the VVA training parameters are initialized, and the Adam algorithm is used to perform gradient descent on the optimized objective function. The joint representation is then updated, completing the fusion of the multimodal data.

[0069] Compared to existing technologies, this method proposes a hypergraph variational autoencoder for multimodal semi-supervised representation. This method automatically encodes high-order correlations in multimodal data and effectively learns the underlying fusion representation by introducing standard Gaussian distribution prior information through variational inference. The hypergraph variational autoencoder is more versatile, capable of processing complex multimodal data and fusing feature data from different modalities using only a small number of semi-supervised labels. This significantly improves multimodal data processing performance, effectively enhancing the performance of data fusion, such as radar and laser features, and enhancing the accuracy of joint target recognition.

[0070] The above-mentioned method of the present application can be converted into a hypergraph variational autoencoder through a program, which is used to automatically mine high-order interactions of multimodal data and infer multimodal fusion representation through the introduction of prior knowledge. It has broad application prospects in radar and laser feature fusion and joint target recognition.

[0071] In one embodiment, Figure 2 As shown, the above step S14 may specifically include the following processing steps:

[0072] S141, calculating the distance between each feature data under a single modality in the multimodal data in the Euclidean space;

[0073] S142, using the K nearest neighbor algorithm to calculate the K nearest neighbors of each feature data in the Euclidean space and select the K value;

[0074] S143, constructing a single-modal hyperedge based on each feature data and its K nearest neighbors in the Euclidean space, and generating a single-modal hypergraph index matrix;

[0075] S144: cascade the hypergraph index matrices under each single modality to obtain a multimodal hypergraph.

[0076] Specifically, after obtaining multimodal data X, we first calculate the distance between each feature data in Euclidean space under a single modality. Then, we use the K-nearest neighbor algorithm to calculate the K nearest neighbors of each feature data in Euclidean space, also known as the K nearest neighbors. We select an appropriate K value based on the task requirements. Each feature node and its K nearest neighbors in Euclidean space form a hyperedge. Similarly, we can obtain the hypergraph index matrix for each single modality. Finally, we concatenate the hypergraph index matrices under each single modality to obtain the multimodal hypergraph structure representation. This hypergraph structure representation is also the aforementioned multimodal hypergraph, and its hypergraph index matrix can be denoted as H.

[0077] Through the above steps, a multimodal hypergraph of multimodal data can be efficiently obtained.

[0078] In one embodiment, the above-mentioned step S16 may specifically include the following processing steps:

[0079] Calculate the degree matrix of hyperedges and vertex of multimodal hypergraph;

[0080] According to the degree matrix of the hyperedge and the degree matrix of the vertex, the original Laplace relation matrix is ​​calculated.

[0081] Specifically, first calculate the degree matrix D of the hyperedge of the multimodal hypergraph e and the vertex degree matrix D v , where the degree of the hyperedge is calculated by the hypergraph index matrix H of the multimodal hypergraph The degree of a vertex is calculated as V represents the number of hyperedges, and ε represents the number of vertices. Then calculate the original Laplace relationship matrix of the multimodal hypergraph, Represents the association relationship between each vertex, H T represents the transpose of H, and W is the weight parameter of the hyperedge.

[0082] Through the above steps, the original Laplace relationship matrix of the hypergraph can be efficiently obtained.

[0083] In one embodiment, the above-mentioned step S16 may specifically include the following processing steps:

[0084] Use the original Laplace relation matrix to construct a single-layer hypergraph convolution;

[0085] Use single-layer hypergraph convolution to extract latent variable representations under each modality;

[0086] Each latent variable representation is cascaded and fused to obtain a potential fused representation.

[0087] Specifically, a single-layer hypergraph convolution is defined by the original Laplace relation matrix of the multimodal hypergraph as Where Θ is the set of parameters for learning and training. A single-layer hypergraph convolution is used to extract the latent variable representation Z under each modality. m =hgcn(X m , H), each latent variable representation is cascaded and fused to obtain the potential fusion representation Z = concat(Z1, ..., Z m ).

[0088] Through the above steps, the potential fusion representation of the hypergraph can be efficiently obtained.

[0089] In one embodiment, the above step S20 may specifically include the following processing steps:

[0090] Use a single layer of hypergraph convolution to extract the mean and variance of the fused representation;

[0091] It uses the reparameterization technique to train and sample from the standard Gaussian distribution prior information;

[0092] A joint representation result is generated based on the mean and variance of the sampling results and the fusion representation.

[0093] Specifically, we first use the constructed single-layer hypergraph convolution to extract the mean and variance of the potential fusion representation Z, that is, μ = hgcn μ (Z, H). Then, we use the reparameterization technique in deep learning to sample ε∈N(0, 1) from the standard Gaussian distribution, and the final joint representation result is z i =μ i +ε·σ i , where μi represents the mean of each variable, σ i represents the variance of each variable, N(0,1) represents the standard normal Gaussian distribution, z i Represents the result of the joint representation of each variable.

[0094] Through the above steps, the joint representation results of multimodal data can be obtained efficiently.

[0095] In one embodiment, the process of initializing the training parameters of the variational autoencoder includes:

[0096] Initialize the dropout ratio, the K value of the K-nearest neighbor algorithm, the size of the learning rate, the dimension of the hidden layer, and the training hyperparameters.

[0097] It is understandable that Figure 3 The figure shows a schematic diagram of the process framework of the above method. Each module in the framework is used to implement the corresponding processing steps of the above method. After obtaining the optimized objective function, the training parameters of the hypergraph variational autoencoder can be initialized. For example, the dropout ratio p, the K value of the K-nearest neighbor algorithm, the learning rate η, the dimension h of the hidden layer, and the training hyperparameter λ are initialized. If Θ0 = {p, K, η, h, λ} represents the initialized parameter set, then the set initialization parameters are, for example, but not limited to, Θ0 = {0.5, 10, 0.001, 128, 0.3}.

[0098] Through the above initialization process, the hypergraph variational autoencoder is quickly ready and starts to update the features of the joint representation.

[0099] In one embodiment, in order to more intuitively and comprehensively illustrate the multimodal semi-supervised representation learning method of the above-mentioned hypergraph variational autoencoder, the following is an experimental example of a multimodal semi-supervised representation learning method of a hypergraph variational autoencoder. It should be noted that the implementation cases given in this specification are only illustrative and are not the only limitation on the specific implementation cases of the present invention. Those skilled in the art can adopt the multimodal semi-supervised representation learning method of the above-mentioned hypergraph variational autoencoder under the guidance of the implementation cases provided by the present invention to realize multimodal data fusion processing in different application scenarios.

[0100] The method was validated on the 3D visual object classification task. Experiments were conducted on the existing bimodal datasets ModelNet40 and NTU multi-view dataset. The experimental results show the effectiveness of the above method proposed in this application on the task of semi-supervised classification of visual targets.

[0101] Compared with traditional baseline methods, the above method proposed in this application achieves the best performance under different label ratios. For example, when the semi-supervised label ratio is 30%, it achieves 1.1%, 1.8% and 7% gains on the NTU dataset compared with HGNN (hypergraph neural network), LFM (low-rank fusion network) and GCN (graph neural network), respectively.

[0102] With a semi-supervised label rate of 2%, the proposed method achieves a 1.8% gain over the best baseline method on the NTU dataset and a 6.3% improvement over the graph autoencoder (GAE). For the ModelNet40 dataset, the proposed method outperforms all compared methods, achieving results close to the full label ratio when trained with only 1% semi-supervised labels to fuse multimodal features. Compared to GCN, it achieves gains of 27.8% and 14.4% using only 1% and 0.2% labels.

[0103] In summary, the above method proposed in this application has high effectiveness.

[0104] Table 1 shows the experimental results under different semi-supervised label ratios on the NTU dataset; Table 2 shows the experimental results under different semi-supervised label ratios on the ModelNet40 dataset; Figure 4 These are the t-SNE two-dimensional feature visualizations of the validation set on the ModelNet40 dataset, which are the feature visualizations before and after the fusion representation. (a) is the original feature of modality one, (b) is the original feature of modality two, and (c) is the obtained potential fusion representation.

[0105] Figure 5 It is the accuracy curve of the training set and test set on the ModelNet40 dataset.

[0106] Table 1

[0107]

[0108] Table 2

[0109]

[0110] It should be understood that although Figure 1 and Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Figure 1 and Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0111] See also Figure 6In one embodiment, a multimodal semi-supervised representation learning device 100 for a hypergraph variational autoencoder is provided, comprising a data acquisition module 11, a hypergraph construction module 13, a matrix calculation module 15, a convolution fusion module 17, a joint representation module 19, a matrix reconstruction module 21, a difference calculation module 23, an objective function module 25, and a representation update module 27. The data acquisition module 11 is configured to acquire multimodal data; the multimodal data includes radar feature data, laser feature data, and image feature data. The hypergraph construction module 13 is configured to construct a multimodal hypergraph using the multimodal data. The matrix calculation module 15 is configured to calculate an original Laplace relation matrix based on the multimodal hypergraph. The convolution fusion module 17 is configured to construct a single-layer hypergraph convolution using the original Laplace relation matrix, and extract a potential fused representation based on the single-layer hypergraph convolution. The joint representation module 19 is configured to introduce standard Gaussian distribution prior information into the fused representation using a variational inference model, thereby inferring a joint representation of the multimodal data. The matrix reconstruction module 21 is used to perform inner product calculations on the joint representation results to generate a reconstructed Laplace relation matrix. The difference calculation module 23 is used to generate difference information between the reconstructed Laplace relation matrix and the original Laplace relation matrix using the KL divergence constraint. The objective function module 25 is used to use the cross-entropy metric to measure the loss between the semi-supervised label and the fused representation of the multimodal data based on the difference information to obtain an optimized objective function. The representation update module 27 is used to initialize the training parameters of the variational autoencoder and perform gradient descent on the optimized objective function using the Adam algorithm to update the joint representation results; the joint representation results are used for multi-sensor joint target recognition.

[0112] The above-mentioned multimodal semi-supervised representation learning device 100 of a hypergraph variational autoencoder, through the collaboration of various modules, is based on the general framework of the hypergraph variational autoencoder. First, after acquiring multimodal data, a multimodal hypergraph is constructed. Then, the original Laplacian matrix is ​​calculated, and a single-layer hypergraph convolution is constructed to extract the potential fusion representation. The variational inference model is then used to introduce standard Gaussian distribution prior information into the fusion representation, infer the joint representation result of the multimodal data, and then the joint representation result is used to reconstruct the Laplacian relationship matrix. After generating the difference information of the two Laplacian matrices using the KL divergence constraint, the cross entropy is used to measure the loss between the semi-supervised label of the multimodal data and the fusion representation to obtain the optimized objective function. Finally, the training parameters of the variational autoencoder are initialized and the Adam algorithm is used to perform gradient descent processing on the optimized objective function, and the joint representation result is updated to complete the fusion processing of the multimodal data.

[0113] Compared to existing technologies, this method proposes a hypergraph variational autoencoder for multimodal semi-supervised representation. This method automatically encodes high-order correlations in multimodal data and effectively learns the underlying fusion representation by introducing standard Gaussian distribution prior information through variational inference. The hypergraph variational autoencoder is more versatile, capable of processing complex multimodal data and fusing feature data from different modalities using only a small number of semi-supervised labels. This significantly improves multimodal data processing performance, effectively enhancing the performance of data fusion, such as radar and laser features, and enhancing the accuracy of joint target recognition.

[0114] In one embodiment, the hypergraph construction module 13 includes a distance calculation submodule, a neighbor calculation submodule, a neighbor calculation submodule, an index generation submodule and a hypergraph cascade submodule. Among them, the distance calculation submodule is used to calculate the distance between each feature data in a single mode in the multimodal data in the Euclidean space. The neighbor calculation submodule is used to use the K nearest neighbor algorithm to calculate the K nearest neighbors of each feature data in the Euclidean space and select the K value. The index generation submodule is used to construct a hyperedge in a single mode according to each feature data and the K nearest neighbors of the feature data in the Euclidean space, and generate a hypergraph index matrix in a single mode. The hypergraph cascade submodule is used to cascade the hypergraph index matrix in each single mode to obtain the multimodal hypergraph.

[0115] In one embodiment, the multimodal semi-supervised representation learning device 100 of the above-mentioned hypergraph variational autoencoder can also be used to implement the steps or sub-steps added in other embodiments of the multimodal semi-supervised representation learning method of the above-mentioned hypergraph variational autoencoder.

[0116] Regarding the specific limitations of a multimodal semi-supervised representation learning device 100 for a hypergraph variational autoencoder, please refer to the corresponding limitations of a multimodal semi-supervised representation learning method for a hypergraph variational autoencoder mentioned above, which will not be repeated here. Each module in the multimodal semi-supervised representation learning device 100 for a hypergraph variational autoencoder mentioned above can be implemented in whole or in part by software, hardware, and a combination thereof. The above modules can be embedded in or independent of a device with a specific data processing function in the form of hardware, or can be stored in the memory of the aforementioned device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. The aforementioned device can be, but is not limited to, various types of data processing devices already available in the art.

[0117] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the following processing steps when executing the computer program: obtaining multimodal data; the multimodal data includes radar feature data, laser feature data, and image feature data; constructing a multimodal hypergraph using the multimodal data; calculating the original Laplace relationship matrix based on the multimodal hypergraph; constructing a single-layer hypergraph convolution using the original Laplace relationship matrix, and extracting a potential fusion representation based on the single-layer hypergraph convolution; using a variational inference model to introduce standard Gaussian distribution prior information into the fusion representation, and inferring a joint representation result of the multimodal data; performing inner product calculation on the joint representation result to generate a reconstructed Laplace relationship matrix; using the KL divergence constraint to generate difference information between the reconstructed Laplace relationship matrix and the original Laplace relationship matrix; based on the difference information, using cross entropy to measure the loss between the semi-supervised label of the multimodal data and the fusion representation to obtain an optimized objective function; initializing the training parameters of the variational autoencoder and using the Adam algorithm to perform gradient descent processing on the optimized objective function to update the joint representation result; the joint representation result is used for multi-sensor joint target recognition.

[0118] It can be understood that in addition to the memory and processor mentioned above, the above-mentioned computer device also includes other software and hardware components not listed in this specification. The specific components can be determined according to the model of the specific data processing equipment in different application scenarios. This specification will not list them one by one in detail.

[0119] In one embodiment, when the processor executes the computer program, it can also implement the steps or sub-steps added to each embodiment of the multimodal semi-supervised representation learning method of the hypergraph variational autoencoder described above.

[0120] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which implements the following processing steps when executed by a processor: obtaining multimodal data; the multimodal data includes radar feature data, laser feature data and image feature data; constructing a multimodal hypergraph using the multimodal data; calculating the original Laplace relationship matrix based on the multimodal hypergraph; constructing a single-layer hypergraph convolution using the original Laplace relationship matrix, and extracting a potential fusion representation based on the single-layer hypergraph convolution; using a variational inference model to introduce standard Gaussian distribution prior information into the fusion representation, and inferring the joint representation result of the multimodal data; performing inner product calculation on the joint representation result to generate a reconstructed Laplace relationship matrix; using the KL divergence constraint to generate difference information between the reconstructed Laplace relationship matrix and the original Laplace relationship matrix; based on the difference information, using cross entropy to measure the loss between the semi-supervised label and the fusion representation of the multimodal data to obtain an optimized objective function; initializing the training parameters of the variational autoencoder and using the Adam algorithm to perform gradient descent processing on the optimized objective function to update the joint representation result; the joint representation result is used for multi-sensor joint target recognition.

[0121] In one embodiment, when the computer program is executed by a processor, it can also implement the steps or sub-steps added to each embodiment of the multimodal semi-supervised representation learning method of the hypergraph variational autoencoder described above.

[0122] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (Rambus DRAM, abbreviated as RDRAM) and interface dynamic random access memory (DRDRAM).

[0123] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application. The description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the patent in this application should be based on the attached claims.

Claims

1. A multimodal semi-supervised representation learning method for hypergraph variational autoencoders, characterized by: Including steps: Acquiring multimodal data; the multimodal data includes radar feature data, laser feature data, and image feature data; constructing a multimodal hypergraph using the multimodal data; Calculating an original Laplace relationship matrix according to the multimodal hypergraph; Constructing a single-layer hypergraph convolution using the original Laplace relationship matrix, and extracting a potential fusion representation based on the single-layer hypergraph convolution; Using a variational inference model to introduce standard Gaussian distribution prior information into the fusion representation, and inferring to obtain a joint representation result of the multimodal data; Performing inner product calculation on the joint representation result to generate a reconstructed Laplace relation matrix; Generating difference information between the reconstructed Laplace relationship matrix and the original Laplace relationship matrix using a KL divergence constraint; According to the difference information, cross entropy is used to measure the loss between the semi-supervised label of the multimodal data and the fusion representation to obtain an optimized objective function; Initialize the training parameters of the variational autoencoder and use the Adam algorithm to perform gradient descent processing on the optimized objective function to update the joint characterization result; the joint characterization result is used for multi-sensor joint target recognition.

2. The multimodal semi-supervised representation learning method for hypergraph variational autoencoders according to claim 1, characterized in that: The step of constructing a multimodal hypergraph using the multimodal data comprises: Calculating the distance between each feature data under a single modality in the multimodal data in the Euclidean space; Use the K nearest neighbor algorithm to calculate the K nearest neighbors of each feature data in the Euclidean space and select the K value; Construct a single-modal hyperedge based on each feature data and its K nearest neighbors in Euclidean space, and generate a single-modal hypergraph index matrix; The hypergraph index matrix under each single modality is cascaded to obtain the multimodal hypergraph.

3. The multimodal semi-supervised representation learning method for hypergraph variational autoencoders according to claim 1 or 2, characterized in that: The step of calculating the original Laplace relationship matrix according to the multimodal hypergraph includes: Calculating the degree matrix of the hyperedges and the degree matrix of the vertices of the multimodal hypergraph; The original Laplace relation matrix is ​​calculated based on the degree matrix of the hyperedge and the degree matrix of the vertex.

4. The multimodal semi-supervised representation learning method for hypergraph variational autoencoders according to claim 3, characterized in that: The steps of constructing a single-layer hypergraph convolution using the original Laplace relationship matrix and extracting a potential fusion representation based on the single-layer hypergraph convolution include: Constructing a single-layer hypergraph convolution using the original Laplace relationship matrix; Utilizing the single-layer hypergraph convolution to extract latent variable representations under each modality; Each latent variable representation is cascaded and fused to obtain the potential fused representation.

5. The multimodal semi-supervised representation learning method for hypergraph variational autoencoders according to claim 3, characterized in that: The step of introducing standard Gaussian distribution prior information into the fusion representation using a variational inference model to infer a joint representation result of the multimodal data includes: extracting the mean and variance of the fused representation using the single-layer hypergraph convolution; The reparameterization technique is used for training, sampling from the standard Gaussian distribution prior information; The joint representation result is generated according to the sampling result and the mean and variance of the fused representation.

6. The multimodal semi-supervised representation learning method for hypergraph variational autoencoders according to claim 1, characterized in that: The process of initializing the training parameters of the variational autoencoder includes: Initialize the dropout ratio, the K value of the K-nearest neighbor algorithm, the size of the learning rate, the dimension of the hidden layer, and the training hyperparameters.

7. A multimodal semi-supervised representation learning device for a hypergraph variational autoencoder, characterized in that: include: A data acquisition module, configured to acquire multimodal data, wherein the multimodal data includes radar feature data, laser feature data, and image feature data; A hypergraph construction module, configured to construct a multimodal hypergraph using the multimodal data; A matrix calculation module, configured to calculate an original Laplace relation matrix based on the multimodal hypergraph; A convolution fusion module, configured to construct a single-layer hypergraph convolution using the original Laplace relation matrix, and extract a potential fusion representation based on the single-layer hypergraph convolution; a joint representation module, configured to introduce standard Gaussian distribution prior information into the fusion representation using a variational inference model, and infer a joint representation result of the multimodal data; A matrix reconstruction module, configured to perform inner product calculation on the joint representation result to generate a reconstructed Laplace relation matrix; a difference calculation module, configured to generate difference information between the reconstructed Laplace relationship matrix and the original Laplace relationship matrix using a KL divergence constraint; An objective function module, configured to measure the loss between the semi-supervised label of the multimodal data and the fused representation using cross entropy based on the difference information to obtain an optimized objective function; The characterization update module is used to initialize the training parameters of the variational autoencoder and use the Adam algorithm to perform gradient descent processing on the optimized objective function to update the joint characterization result; the joint characterization result is used for multi-sensor joint target recognition.

8. The multimodal semi-supervised representation learning device for hypergraph variational autoencoder according to claim 7, characterized in that: The hypergraph construction module includes: A distance calculation submodule, configured to calculate the distance between each feature data under a single modality in the multimodal data in the Euclidean space; The nearest neighbor calculation submodule is used to calculate the K nearest neighbors of each feature data in the Euclidean space using the K nearest neighbor algorithm and select the K value; The index generation submodule is used to construct a single-modal hyperedge based on each feature data and its K nearest neighbors in Euclidean space, and generate a single-modal hypergraph index matrix; The hypergraph cascading submodule is used to cascade the hypergraph index matrices under each single modality to obtain the multimodal hypergraph.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multimodal semi-supervised representation learning method of a hypergraph variational autoencoder according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a multimodal semi-supervised representation learning method of a hypergraph variational autoencoder according to any one of claims 1 to 6 are implemented.

Images