Multimodal data analysis method and device for maximizing modal consistency
Through autoencoder and CCA correlation analysis, the multimodal data is encoded and matrix constraints are solved, and the problems of low fusion efficiency and large errors between modals are achieved, achieving more efficient and accurate multimodal data analysis.
Patent Information
- Application Number
- CN202210269022.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-18
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-03-18
AI Technical Summary
The existing multimodal data analysis methods have low fusion efficiency and errors between modes, making it difficult to grasp the overall situation in a timely and accurate manner under a strong confrontation environment.
The autoencoder is used to encode multimodal original data, and the correlation matrix constraint is performed through CCA correlation analysis to obtain the modal consistency data matrix, decoding and error calculation are performed to reconstruct the data, and improve the fusion efficiency of modal data and reduce errors.
The fusion efficiency between modal data is improved, the error of fusion results is reduced, the consistency between modals is enhanced, and multimodal data can be analyzed more accurately.
Smart Images

Figure CN114722916B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a multimodal data analysis method, apparatus, computer equipment and storage medium for maximizing modal consistency. Background Art
[0002] In the modern information warfare environment, there are many types of sensors that can be used for situational awareness and a large amount of data. The data generated by these sensors (such as radar spectrum, target point track, situation, and message description, etc.) are expressed as multimodal data. Each mode may provide certain useful information, but it is difficult to provide relatively complete and accurate information. This information is complementary to a certain extent, but at the same time there are serious redundancies and differences. Therefore, in a strong confrontation environment, effectively analyzing and integrating multimodal data from multiple intelligence sources is a necessary means to timely and accurately grasp the ever-changing overall situation.
[0003] However, the current traditional multimodal data analysis is based on mapping the input to a low dimension through an encoder and then generating an output of the same dimension through a decoder. In this process, due to the large interval between modalities, the fusion efficiency between the modal data is low, and there are losses in the encoding and decoding process, which leads to errors in the modal data fusion results. Summary of the invention
[0004] Based on this, it is necessary to provide a multimodal data analysis method, device, computer equipment and storage medium that can maximize modal consistency and improve the efficiency of modal data fusion in response to the above technical problems.
[0005] A multimodal data analysis method for maximizing modal consistency, the method comprising:
[0006] Acquire a multimodal original data set; the multimodal original data set includes multiple multimodal original data;
[0007] Encode the multimodal original data using an autoencoder to obtain an encoding subspace of the multimodal original data;
[0008] According to CCA correlation analysis, the coding subspace of the multimodal original data is constrained by correlation matrix to obtain the correlation matrix of the multimodal original data;
[0009] The correlation matrix of multimodal raw data is used as the data matrix for maximizing modality consistency;
[0010] Decoding the maximum modal consistency data matrix to obtain the maximum modal consistency modal data;
[0011] The reconstruction error is obtained by calculating the error based on the modal data and the multi-modal original data with the maximum modal consistency;
[0012] The multimodal raw data are analyzed and reconstructed according to the reconstruction error.
[0013] In one embodiment, a correlation matrix constraint is performed on the coding subspace of the multimodal original data according to the CCA correlation analysis to obtain a correlation matrix of the multimodal original data, including:
[0014]
[0015] Among them, U represents the correlation matrix of the x-th mode, V represents the correlation matrix of the y-th mode, f(x) represents the encoding subspace of the x-th mode, g(y) represents the encoding subspace of the y-th mode, θ represents the weight of the regularization factor, n represents the number of samples, tr() represents the trace of the matrix, x and y represent two different modes, Represents the encoded data of modality x, y.
[0016] In one embodiment, the constraints of the correlation matrix include:
[0017]
[0018] Among them, I represents the identity matrix, r represents the random missing rate, and T represents the transpose operation.
[0019] In one embodiment, the constraints also include:
[0020]
[0021] In one embodiment, the reconstruction error obtained by performing error calculation based on the modal data with maximized modal consistency and the multimodal original data includes:
[0022] The error is calculated based on the modal data with the maximum modal consistency and the multimodal original data, and the reconstruction error is obtained as follows:
[0023]
[0024] Among them, LR represents the reconstruction error, H represents the number of modes, v represents the sequence number of the mode, N represents the number of samples, n represents the sample sequence number, and x n represents the multimodal raw data of the xth mode of the nth sample, g v () represents the decoding mapping function, U n represents the correlation matrix of the xth mode of the nth sample, V n represents the correlation matrix of the yth mode of the nth sample, y n Represents the multimodal raw data of the y-th mode of the n-th sample.
[0025] A multimodal data analysis device for maximizing modal consistency, the device comprising:
[0026] An encoding module is used to obtain a multimodal original data set; the multimodal original data set contains multiple multimodal original data; the multimodal original data is encoded by using an autoencoder to obtain an encoding subspace of the multimodal original data;
[0027] A correlation constraint module is used to perform correlation matrix constraints on the encoding subspace of the multimodal original data according to the CCA correlation analysis to obtain the correlation matrix of the multimodal original data; the correlation matrix of the multimodal original data is used as the data matrix for maximizing modal consistency;
[0028] A decoding module, used for decoding the maximum modal consistency data matrix to obtain the maximum modal consistency modal data;
[0029] The reconstruction module is used to calculate the error based on the modal data with the maximum modal consistency and the multimodal original data to obtain the reconstruction error; and analyze and reconstruct the multimodal original data based on the reconstruction error.
[0030] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0031] Acquire a multimodal original data set; the multimodal original data set includes multiple multimodal original data;
[0032] Encode the multimodal original data using an autoencoder to obtain an encoding subspace of the multimodal original data;
[0033] According to CCA correlation analysis, the coding subspace of the multimodal original data is constrained by correlation matrix to obtain the correlation matrix of the multimodal original data;
[0034] The correlation matrix of multimodal raw data is used as the data matrix for maximizing modality consistency;
[0035] Decoding the maximum modal consistency data matrix to obtain the maximum modal consistency modal data;
[0036] The reconstruction error is obtained by calculating the error based on the modal data and the multi-modal original data with the maximum modal consistency;
[0037] The multimodal raw data are analyzed and reconstructed according to the reconstruction error.
[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:
[0039] Acquire a multimodal original data set; the multimodal original data set includes multiple multimodal original data;
[0040] Encode the multimodal original data using an autoencoder to obtain an encoding subspace of the multimodal original data;
[0041] According to CCA correlation analysis, the coding subspace of the multimodal original data is constrained by correlation matrix to obtain the correlation matrix of the multimodal original data;
[0042] The correlation matrix of multimodal raw data is used as the data matrix for maximizing modality consistency;
[0043] Decoding the maximum modal consistency data matrix to obtain the maximum modal consistency modal data;
[0044] The reconstruction error is obtained by calculating the error based on the modal data and the multi-modal original data with the maximum modal consistency;
[0045] The multimodal raw data are analyzed and reconstructed according to the reconstruction error.
[0046] The multimodal data analysis method, device, computer equipment and storage medium for maximizing modal consistency first use an autoencoder to encode the multimodal original data to obtain the encoding subspace of the multimodal original data, perform correlation matrix constraints on the encoding subspace of the multimodal original data according to CCA correlation analysis, and obtain the correlation matrix of the multimodal original data. Use CCA correlation analysis to perform correlation matrix constraints on the encoding subspace of the multimodal original data, which can minimize the similarity of the encoding subspace. Compared with traditional autoencoding, it has a better spatial clustering effect and is more conducive to breaking the interval between modalities, enhancing the consistency between modalities, and obtaining maximized modal consistency data. The multimodal original data is reconstructed using the reconstruction error of the maximized modal consistency data and the multimodal original data. On the one hand, the fusion efficiency between the modal data is improved, and on the other hand, the error of the modal data fusion result is avoided. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 A schematic diagram of a flow chart of a multimodal data analysis method for maximizing modal consistency in one embodiment;
[0048] Figure 2 A structural block diagram of a multimodal data analysis device for maximizing modal consistency in one embodiment;
[0049] Figure 3 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0051] In one embodiment, Figure 1 As shown, a multimodal data analysis method for maximizing modal consistency is provided, comprising the following steps:
[0052] Step 102, obtaining a multimodal original data set; the multimodal original data set includes a plurality of multimodal original data; encoding the multimodal original data using an autoencoder to obtain an encoding subspace of the multimodal original data.
[0053] Multimodal raw data refers to data generated by sensors used for situational awareness, including radar data, remote sensing data, target point track data, situation data, and message description data.
[0054] Step 104, performing correlation matrix constraints on the encoding subspace of the multimodal original data according to the CCA correlation analysis, and obtaining the correlation matrix of the multimodal original data; the correlation matrix of the multimodal original data is used as the data matrix for maximizing modal consistency.
[0055] Using CCA correlation analysis to constrain the correlation matrix of the encoding subspace of multimodal original data can minimize the similarity of the encoding subspace. Compared with traditional autoencoding, it has a better spatial clustering effect, is more conducive to breaking the gap between modalities, enhancing the consistency between modalities, obtaining the maximum modal consistency data, and improving the fusion efficiency between modal data.
[0056] Step 106, decoding the maximum modal consistency data matrix to obtain maximum modal consistency modal data.
[0057] Step 108, performing error calculation based on the modal data with maximized modal consistency and the multimodal original data to obtain a reconstruction error; and analyzing and reconstructing the multimodal original data based on the reconstruction error.
[0058] The multimodal original data is reconstructed by maximizing the reconstruction error between the modal consistency data and the multimodal original data, thus avoiding the generation of errors in the modal data fusion results.
[0059] In the above-mentioned multimodal data analysis method for maximizing modal consistency, the multimodal original data is first encoded using an autoencoder to obtain the encoding subspace of the multimodal original data. The encoding subspace of the multimodal original data is constrained by a correlation matrix according to CCA correlation analysis to obtain the correlation matrix of the multimodal original data. The encoding subspace of the multimodal original data is constrained by a correlation matrix using CCA correlation analysis, which can minimize the similarity of the encoding subspace. Compared with traditional autoencoding, it has a better spatial clustering effect and is more conducive to breaking the interval between modalities, enhancing the consistency between modalities, and obtaining maximized modal consistency data. The multimodal original data is reconstructed using the reconstruction error of the maximized modal consistency data and the multimodal original data. On the one hand, the fusion efficiency between the modal data is improved, and on the other hand, the error of the modal data fusion result is avoided.
[0060] In one embodiment, a correlation matrix constraint is performed on the coding subspace of the multimodal original data according to the CCA correlation analysis to obtain a correlation matrix of the multimodal original data, including:
[0061]
[0062] Among them, U represents the correlation matrix of the x-th mode, V represents the correlation matrix of the y-th mode, f(x) represents the encoding subspace of the x-th mode, g(y) represents the encoding subspace of the y-th mode, θ represents the weight of the regularization factor, n represents the number of samples, tr() represents the trace of the matrix, x and y represent two different modes, Represents the encoded data of modality x, y.
[0063] Uf(x)g(y)V represents the cross-correlation matrix of the code f(x) of mode x and the code g(y) of y. Represents the encoding error of all samples of multimodal data.
[0064] In one embodiment, the constraints of the correlation matrix include:
[0065]
[0066] Among them, I represents the identity matrix, r represents the random missing rate, and T represents the transpose operation.
[0067] In one embodiment, the constraints also include:
[0068]
[0069] In one embodiment, the reconstruction error obtained by performing error calculation based on the modal data with maximized modal consistency and the multimodal original data includes:
[0070] The error is calculated based on the modal data with the maximum modal consistency and the multimodal original data, and the reconstruction error is obtained as follows:
[0071]
[0072] Where LR represents the reconstruction error, H represents the number of modes, v represents the sequence number of the mode, N represents the number of samples, n represents the sample sequence number, and x n represents the multimodal raw data of the xth mode of the nth sample, g v () represents the decoding mapping function, U n represents the correlation matrix of the xth mode of the nth sample, V n represents the correlation matrix of the yth mode of the nth sample, y n Represents the multimodal raw data of the y-th mode of the n-th sample.
[0073] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0074] In one embodiment, Figure 2 As shown, a multimodal data analysis device for maximizing modal consistency is provided, comprising: an encoding module 202, a correlation constraint module 206, a decoding module 208 and a reconstruction module 210, wherein:
[0075] The encoding module 202 is used to obtain a multimodal original data set; the multimodal original data set includes a plurality of multimodal original data; the multimodal original data is encoded by using an autoencoder to obtain an encoding subspace of the multimodal original data;
[0076] A correlation constraint module 206 is used to perform correlation matrix constraints on the coding subspace of the multimodal original data according to the CCA correlation analysis to obtain a correlation matrix of the multimodal original data; the correlation matrix of the multimodal original data is used as a data matrix for maximizing modal consistency;
[0077] A decoding module 208 is used to decode the maximum modal consistency data matrix to obtain maximum modal consistency modal data;
[0078] The reconstruction module 210 is used to perform error calculation based on the modal data with maximized modal consistency and the multimodal original data to obtain a reconstruction error; and analyze and reconstruct the multimodal original data based on the reconstruction error.
[0079] In one embodiment, the correlation constraint module 206 is further used to perform correlation matrix constraints on the coding subspace of the multimodal original data according to the CCA correlation analysis to obtain the correlation matrix of the multimodal original data, including:
[0080]
[0081] Among them, U represents the correlation matrix of the x-th mode, V represents the correlation matrix of the y-th mode, f(x) represents the encoding subspace of the x-th mode, g(y) represents the encoding subspace of the y-th mode, θ represents the weight of the regularization factor, n represents the number of samples, tr() represents the trace of the matrix, x and y represent two different modes, Represents the encoded data of modality x, y.
[0082] In one of the embodiments, in one of the embodiments, the constraints of the correlation matrix include:
[0083]
[0084] Among them, I represents the identity matrix, r represents the random missing rate, and T represents the transpose operation.
[0085] In one embodiment, the constraints also include:
[0086]
[0087] In one embodiment, the reconstruction module 210 is further configured to perform error calculation based on the modal data for maximizing modal consistency and the multimodal original data, and the obtained reconstruction error includes:
[0088] The error is calculated based on the modal data with the maximum modal consistency and the multimodal original data, and the reconstruction error is obtained as follows:
[0089]
[0090] Among them, LR represents the reconstruction error, H represents the number of modes, v represents the sequence number of the mode, N represents the number of samples, n represents the sample sequence number, and x n represents the multimodal raw data of the xth mode of the nth sample, g v () represents the decoding mapping function, U n represents the correlation matrix of the xth mode of the nth sample, Vn represents the correlation matrix of the yth mode of the nth sample, y n Represents the multimodal raw data of the y-th mode of the n-th sample.
[0091] For the specific definition of the multimodal data analysis device for maximizing modal consistency, please refer to the definition of the multimodal data analysis method for maximizing modal consistency in the above text, which will not be repeated here. Each module in the above-mentioned multimodal data analysis device for maximizing modal consistency can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0092] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multimodal data analysis method that maximizes modal consistency is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0093] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0094] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.
[0095] In one embodiment, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.
[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0097] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0098] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A multimodal data analysis method to maximize modal consistency, It is characterized in that The method comprises: Acquire a multimodal original data set; the multimodal original data set includes a plurality of multimodal original data; Encoding the multimodal original data using an autoencoder to obtain an encoding subspace of the multimodal original data; According to CCA correlation analysis, a correlation matrix constraint is performed on the encoding subspace of the multimodal original data to obtain a correlation matrix of the multimodal original data; The correlation matrix of the multimodal raw data is used as a data matrix for maximizing modality consistency; Decoding the maximizing modal consistency data matrix to obtain maximizing modal consistency modal data; Performing error calculation based on the maximized modal consistency modal data and the multimodal original data to obtain a reconstruction error; Analyzing and reconstructing the multimodal raw data according to the reconstruction error; According to the CCA correlation analysis, the coding subspace of the multimodal original data is subjected to correlation matrix constraints to obtain the correlation matrix of the multimodal original data, including: Among them, U represents the correlation matrix of the x-th mode, V represents the correlation matrix of the y-th mode, f(x) represents the encoding subspace of the x-th mode, g(y) represents the encoding subspace of the y-th mode, θ represents the weight of the regularization factor, n represents the number of samples, tr() represents the trace of the matrix, x and y represent two different modes, Represents the encoded data of modality x, y; The constraints of the correlation matrix include: Where I represents the identity matrix, r represents the random missing rate, and T represents the transposition operation; The constraints also include:
2. The method according to claim 1, It is characterized in that The reconstruction error obtained by performing error calculation based on the maximized modal consistency modal data and the multimodal original data includes: The error is calculated based on the modal data with the maximum modal consistency and the multimodal original data, and the reconstruction error is obtained as follows: Where LR represents the reconstruction error, H represents the number of modes, v represents the sequence number of the mode, N represents the number of samples, n represents the sample sequence number, and x n represents the multimodal raw data of the xth mode of the nth sample, g v () represents the decoding mapping function, U n represents the correlation matrix of the xth mode of the nth sample, V n represents the correlation matrix of the yth mode of the nth sample, y n Represents the multimodal raw data of the y-th mode of the n-th sample.
3. A multimodal data analysis device that maximizes modal consistency, It is characterized in that The device comprises: An encoding module is used to obtain a multimodal original data set; the multimodal original data set includes a plurality of multimodal original data; the multimodal original data is encoded by an autoencoder to obtain an encoding subspace of the multimodal original data; A correlation constraint module is used to perform correlation matrix constraints on the encoding subspace of the multimodal original data according to CCA correlation analysis to obtain the correlation matrix of the multimodal original data; the correlation matrix of the multimodal original data is used as the maximized modal consistency data matrix; according to CCA correlation analysis, the encoding subspace of the multimodal original data is subjected to correlation matrix constraints to obtain the correlation matrix of the multimodal original data, including: Among them, U represents the correlation matrix of the x-th mode, V represents the correlation matrix of the y-th mode, f(x) represents the encoding subspace of the x-th mode, g(y) represents the encoding subspace of the y-th mode, θ represents the weight of the regularization factor, n represents the number of samples, tr() represents the trace of the matrix, x and y represent two different modes, Represents the coded data of mode x, y; the constraints of the correlation matrix include: Where I represents the identity matrix, r represents the random missing rate, and T represents the transposition operation; The constraints also include: A decoding module, used for decoding the maximum modal consistency data matrix to obtain maximum modal consistency modal data; A reconstruction module is used to perform error calculation based on the modal data for maximizing modal consistency and the multimodal original data to obtain a reconstruction error; and analyze and reconstruct the multimodal original data based on the reconstruction error.
4. A computer device comprising a memory and a processor, wherein the memory stores a computer program, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.
5. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.
Citation Information
Patent Citations
Cross-modal retrieval method and device based on hash algorithm and neighborhood graph
CN112199531A
Multi-modal electronic document classification
US10223586B1
