Multi-modal data dynamic fusion method and device based on graph optimization

Abstract

The invention relates to the technical field of data fusion, and particularly provides a multi-modal data dynamic fusion method and device based on graph optimization, and the method comprises the steps: collecting multi-source heterogeneous data, and carrying out the preprocessing of the multi-source heterogeneous data; performing feature extraction on the preprocessed multi-source heterogeneous data to obtain features corresponding to the multi-source heterogeneous data; and constructing a data fusion graph based on the features corresponding to the multi-source heterogeneous data to obtain a multi-modal data fusion result. According to the technical scheme provided by the invention, the fusion framework based on the feature similarity graph is constructed so as to realize excavation of the recessive feature relationship.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion, and in particular to a method and device for dynamic fusion of multimodal data based on graph optimization. Background Art

[0002] With the rapid development of artificial intelligence, the data monitoring capabilities of intelligent systems have been rapidly improved. However, traditional systems have significant deficiencies in processing multi-source heterogeneous data. They are unable to effectively pre-process data from visual cameras, lidar, millimeter-wave radar, ultrasonic radar, and GPS positioning modules. Therefore, they lack the ability to obtain multiple key information such as vision, acoustics, distance perception, and positioning.

[0003] Furthermore, the system is unable to represent multimodal data and analyze cross-modal correlations, resulting in inefficient and inaccurate data utilization and difficulty fully understanding and processing complex scenario information. Furthermore, it is unable to fuse multimodal data in real time, making it unable to meet the requirements of scenarios such as automated drone inspections and autonomous driving, which require extremely high real-time information processing. Summary of the Invention

[0004] In order to overcome the above-mentioned defects, the present invention proposes a multimodal data dynamic fusion method and device based on graph optimization.

[0005] In a first aspect, a method for dynamic fusion of multimodal data based on graph optimization is provided, wherein the method comprises:

[0006] Collecting multi-source heterogeneous data and preprocessing the multi-source heterogeneous data;

[0007] Perform feature extraction on the preprocessed multi-source heterogeneous data to obtain the features corresponding to the multi-source heterogeneous data;

[0008] A data fusion graph is constructed based on the features corresponding to the multi-source heterogeneous data to obtain a multimodal data fusion result.

[0009] Preferably, the collecting of multi-source heterogeneous data includes:

[0010] Collecting image data through an image sensor;

[0011] Collect point cloud data through point cloud sensors;

[0012] Collect voice data through voice sensors;

[0013] The positioning data is collected through the GPS positioning module.

[0014] Furthermore, the preprocessing includes at least one of the following: benchmark unification, distortion correction, boundary limitation, correction of sensor errors, data alignment, time synchronization, filtering and denoising, and data synthesis.

[0015] Furthermore, the feature extraction is performed on the pre-processed multi-source heterogeneous data to obtain features corresponding to the multi-source heterogeneous data, including:

[0016] The image data is subjected to feature extraction through a deep residual shrinkage network to obtain image data features;

[0017] Perform feature extraction on point cloud data through UniSeg to obtain point cloud data features;

[0018] Extract features from speech data using an acoustic model to obtain speech data features;

[0019] The probability features corresponding to the positioning data are obtained through the one-hot encoding function.

[0020] Furthermore, constructing a data fusion graph based on features corresponding to the multi-source heterogeneous data includes:

[0021] Use the features corresponding to multi-source heterogeneous data as the attributes of the information source node;

[0022] The target object name observed by the sensor is used as the attribute of the environmental element node;

[0023] When there is an observation relationship between the information source node and the environment element node, an edge between the information source node and the environment element node is constructed;

[0024] When two information source nodes have a connection relationship with the same environmental element node, an edge between the two information source nodes is constructed, and a weight of the edge is set based on the attributes of the two information source nodes.

[0025] Furthermore, the step of setting the weight of the edge based on the attributes of the two information source nodes includes:

[0026] The weight of the edge between two information source nodes is set based on the similarity between the attributes of the two information source nodes.

[0027] In a second aspect, a multimodal data dynamic fusion device based on graph optimization is provided, wherein the multimodal data dynamic fusion device based on graph optimization comprises:

[0028] A preprocessing module, used to collect multi-source heterogeneous data and preprocess the multi-source heterogeneous data;

[0029] The feature extraction module is used to extract features from the preprocessed multi-source heterogeneous data to obtain features corresponding to the multi-source heterogeneous data;

[0030] The analysis module is used to construct a data fusion graph based on the features corresponding to the multi-source heterogeneous data to obtain a multimodal data fusion result.

[0031] In a third aspect, a computer device is provided, comprising: one or more processors;

[0032] The processor is configured to execute one or more programs;

[0033] When the one or more programs are executed by the one or more processors, the multimodal data dynamic fusion method based on graph optimization is implemented.

[0034] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the multimodal data dynamic fusion method based on graph optimization is implemented.

[0035] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0036] The present invention provides a method and device for dynamic fusion of multimodal data based on graph optimization, comprising: collecting and preprocessing multi-source heterogeneous data; extracting features from the preprocessed multi-source heterogeneous data to obtain features corresponding to the multi-source heterogeneous data; and constructing a data fusion graph based on the features corresponding to the multi-source heterogeneous data to obtain a multimodal data fusion result. The technical solution provided by the present invention, by constructing a fusion framework based on a feature similarity graph, achieves multimodal data representation and cross-modal correlation analysis, significantly improving the comprehensive utilization efficiency and accuracy of data and enabling the system to more comprehensively understand and process complex scene information. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the main steps of a multimodal data dynamic fusion method based on graph optimization according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0040] Example 1

[0041] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart showing the main steps of a multimodal data dynamic fusion method based on graph optimization according to an embodiment of the present invention. Figure 1 As shown, the multimodal data dynamic fusion method based on graph optimization in the embodiment of the present invention mainly includes the following steps:

[0042] Step S101: collecting multi-source heterogeneous data and preprocessing the multi-source heterogeneous data;

[0043] Step S102: extracting features from the pre-processed multi-source heterogeneous data to obtain features corresponding to the multi-source heterogeneous data;

[0044] Step S103: constructing a data fusion graph based on the features corresponding to the multi-source heterogeneous data to obtain a multimodal data fusion result.

[0045] In this embodiment, collecting multi-source heterogeneous data includes:

[0046] Collecting image data through an image sensor;

[0047] Collect point cloud data through point cloud sensors;

[0048] Collect voice data through voice sensors;

[0049] The positioning data is collected through the GPS positioning module.

[0050] In one embodiment, the multi-source heterogeneous data may be multi-source heterogeneous data from sources such as visual cameras, laser radars, millimeter-wave radars, ultrasonic radars, and GPS positioning modules.

[0051] In one embodiment, the preprocessing includes at least one of the following: benchmark unification, distortion correction, boundary limitation, correction of sensor errors, data alignment, time synchronization, filtering and denoising, and data synthesis.

[0052] In one embodiment, extracting features from the preprocessed multi-source heterogeneous data to obtain features corresponding to the multi-source heterogeneous data includes:

[0053] The image data is subjected to feature extraction through a deep residual shrinkage network to obtain image data features;

[0054] Perform feature extraction on point cloud data through UniSeg to obtain point cloud data features;

[0055] Extract features from speech data using an acoustic model to obtain speech data features;

[0056] The probability features corresponding to the positioning data are obtained through the one-hot encoding function.

[0057] In one embodiment, constructing a data fusion graph based on features corresponding to the multi-source heterogeneous data includes:

[0058] Use the features corresponding to multi-source heterogeneous data as the attributes of the information source node;

[0059] The target object name observed by the sensor is used as the attribute of the environmental element node;

[0060] When there is an observation relationship between the information source node and the environment element node, an edge between the information source node and the environment element node is constructed;

[0061] When two information source nodes have a connection relationship with the same environmental element node, an edge between the two information source nodes is constructed, and a weight of the edge is set based on the attributes of the two information source nodes.

[0062] In one embodiment, setting the weight of the edge based on the attributes of the two information source nodes includes:

[0063] The weight of the edge between two information source nodes is set based on the similarity between the attributes of the two information source nodes.

[0064] Example 2

[0065] Based on the same inventive concept, the present invention further provides a multimodal data dynamic fusion device based on graph optimization, the multimodal data dynamic fusion device based on graph optimization comprising:

[0066] A preprocessing module, used to collect multi-source heterogeneous data and preprocess the multi-source heterogeneous data;

[0067] The feature extraction module is used to extract features from the preprocessed multi-source heterogeneous data to obtain features corresponding to the multi-source heterogeneous data;

[0068] The analysis module is used to construct a data fusion graph based on the features corresponding to the multi-source heterogeneous data to obtain a multimodal data fusion result.

[0069] Preferably, the collecting of multi-source heterogeneous data includes:

[0070] Collecting image data through an image sensor;

[0071] Collect point cloud data through point cloud sensors;

[0072] Collect voice data through voice sensors;

[0073] The positioning data is collected through the GPS positioning module.

[0074] Furthermore, the preprocessing includes at least one of the following: benchmark unification, distortion correction, boundary limitation, correction of sensor errors, data alignment, time synchronization, filtering and denoising, and data synthesis.

[0075] Furthermore, the feature extraction is performed on the pre-processed multi-source heterogeneous data to obtain features corresponding to the multi-source heterogeneous data, including:

[0076] The image data is subjected to feature extraction through a deep residual shrinkage network to obtain image data features;

[0077] Perform feature extraction on point cloud data through UniSeg to obtain point cloud data features;

[0078] Extract features from speech data using an acoustic model to obtain speech data features;

[0079] The probability features corresponding to the positioning data are obtained through the one-hot encoding function.

[0080] Furthermore, constructing a data fusion graph based on features corresponding to the multi-source heterogeneous data includes:

[0081] Use the features corresponding to multi-source heterogeneous data as the attributes of the information source node;

[0082] The target object name observed by the sensor is used as the attribute of the environmental element node;

[0083] When there is an observation relationship between the information source node and the environment element node, an edge between the information source node and the environment element node is constructed;

[0084] When two information source nodes have a connection relationship with the same environmental element node, an edge between the two information source nodes is constructed, and a weight of the edge is set based on the attributes of the two information source nodes.

[0085] Furthermore, the step of setting the weight of the edge based on the attributes of the two information source nodes includes:

[0086] The weight of the edge between two information source nodes is set based on the similarity between the attributes of the two information source nodes.

[0087] Example 3

[0088] Based on the same inventive concept, the present invention also provides a computer device, which includes a processor and a memory, the memory being used to store a computer program, the computer program including program instructions, and the processor being used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of a multimodal data dynamic fusion method based on graph optimization in the above embodiment.

[0089] Example 4

[0090] Based on the same inventive concept, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of a multimodal data dynamic fusion method based on graph optimization in the above embodiment.

[0091] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0093] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A multimodal data dynamic fusion method based on graph optimization, characterized in that: The method comprises: Collecting multi-source heterogeneous data and preprocessing the multi-source heterogeneous data; Perform feature extraction on the preprocessed multi-source heterogeneous data to obtain the features corresponding to the multi-source heterogeneous data; A data fusion graph is constructed based on the features corresponding to the multi-source heterogeneous data to obtain a multimodal data fusion result.

2. The method according to claim 1, wherein The collecting of multi-source heterogeneous data includes: Collecting image data through an image sensor; Collect point cloud data through point cloud sensors; Collect voice data through voice sensors; The positioning data is collected through the GPS positioning module.

3. The method according to claim 2, wherein The preprocessing includes at least one of the following: benchmark unification, distortion correction, boundary limitation, correction of sensor errors, data alignment, time synchronization, filtering and denoising, and data synthesis.

4. The method according to claim 2, wherein The feature extraction of the pre-processed multi-source heterogeneous data to obtain features corresponding to the multi-source heterogeneous data includes: The image data is subjected to feature extraction through a deep residual shrinkage network to obtain image data features; Perform feature extraction on point cloud data through UniSeg to obtain point cloud data features; Extract features from speech data using an acoustic model to obtain speech data features; The probability features corresponding to the positioning data are obtained through the one-hot encoding function.

5. The method according to claim 2, wherein The constructing of a data fusion graph based on features corresponding to the multi-source heterogeneous data includes: Use the features corresponding to multi-source heterogeneous data as the attributes of the information source node; The target object name observed by the sensor is used as the attribute of the environmental element node; When there is an observation relationship between the information source node and the environment element node, an edge between the information source node and the environment element node is constructed; When two information source nodes have a connection relationship with the same environmental element node, an edge between the two information source nodes is constructed, and a weight of the edge is set based on the attributes of the two information source nodes.

6. The method according to claim 5, wherein The step of setting the weight of the edge based on the attributes of the two information source nodes includes: The weight of the edge between two information source nodes is set based on the similarity between the attributes of the two information source nodes.

7. A device based on the multimodal data dynamic fusion method based on graph optimization according to any one of claims 1 to 6, characterized in that: The device comprises: A preprocessing module, used to collect multi-source heterogeneous data and preprocess the multi-source heterogeneous data; The feature extraction module is used to extract features from the preprocessed multi-source heterogeneous data to obtain features corresponding to the multi-source heterogeneous data; The analysis module is used to construct a data fusion graph based on the features corresponding to the multi-source heterogeneous data to obtain a multimodal data fusion result.

8. A computer device, characterized in that: include: one or more processors; The processor is configured to execute one or more programs; When the one or more programs are executed by the one or more processors, the multimodal data dynamic fusion method based on graph optimization as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed, the multimodal data dynamic fusion method based on graph optimization as described in any one of claims 1 to 6 is implemented.

Images