Multimodal data fusion method, device, medium and electronic equipment

Abstract

The embodiments of the present application disclose a multimodal data fusion method, device, medium and electronic device, which relate to the field of data fusion technology, and aim to solve the problems of poor fusion effect and difficulty of fusion of multimodal data in the prior art. The present application realizes the fusion of multimodal data by inputting the original data into the fusion network. Since the fusion network is obtained by fusion of multiple material networks, the mechanism of its encoder and decoder makes it not affected by the indefinite length sequence of the input data, and effectively combines different modal data, and introduces a global fusion and decoder re-copying strategy, so that it can encode different types of materials separately first, and then encode them as a whole, output more comprehensive global features, and realize high-quality fusion of different modal data.

Description

Technical Field

[0001] The present application relates to the technical field of data fusion, and in particular to a multimodal data fusion method, device, medium and electronic equipment. Background Art

[0002] The existing video coding technologies include H.264, H.265, VP9, ​​AVS, etc. These technologies mainly use the correlation of data in the time domain and space domain to achieve the purpose of data compression. However, different data contain different information, and the existing methods can only extract part of the information in the video. Information coding (Information theory) is a theory that studies the relationship between data and information, and data and signals. It effectively distinguishes the concepts of "data" and "information". Its focus is on the representation and encoding of data, while weakening the focus on the content information represented. Therefore, traditional coding / data compression methods are difficult to fuse and uniformly represent data of different modalities (video, audio, image, voice, text, symbols, etc.).

[0003] The data integrated by traditional solutions is only for a single modality. It usually uses specialized video processing and image processing modules to deal with different data types. For highly integrated devices such as network terminals and mobile terminals with multiple types of data, this solution cannot maximize the advantages of different data. In addition, the use of multiple modules for differentiated processing requires additional maintenance of support for each type, which also places higher requirements on the equipment. Summary of the invention

[0004] The main purpose of the present application is to provide a multimodal data fusion method, device, medium and electronic device, aiming to solve the problems of poor fusion effect and difficulty of multimodal data fusion in the prior art.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a multimodal data fusion method, comprising the following steps:

[0007] Obtain multi-modal raw data;

[0008] The original data is input into the fusion network to obtain fused data; wherein the fusion network is obtained based on global fusion of several material networks, the material networks are used to process different types of data respectively, each material network includes an encoder and a decoder, the weights of the decoder for fusing material features are replicated several times, the weights of each replicated decoder are used to process the material features obtained by an encoder, and the material network is obtained by separate training in end-to-end based on the weights of several merged decoders through training samples.

[0009] In a possible implementation manner of the first aspect, before inputting the original data into the fusion network to obtain the fused data, the method further includes:

[0010] Build encoders and decoders;

[0011] The weight of the decoder used to fuse the material features is copied several times, and the copied decoder weights are used to process the material features obtained by the encoder respectively;

[0012] The weights of several decoders are combined, and based on the combined weights of the decoders, they are trained separately in an end-to-end manner through training samples to obtain a material network;

[0013] Based on several material networks, global fusion is performed to obtain a fusion network.

[0014] In a possible implementation manner of the first aspect, performing global fusion based on multiple material networks to obtain a fusion network includes:

[0015] Based on several material networks, the decoders are globally merged and the encoders are kept unchanged to obtain a fusion network.

[0016] In a possible implementation manner of the first aspect, before inputting the original data into the fusion network to obtain the fused data, the method further includes:

[0017] Based on the encoder, data processing is performed on the multi-modal sample materials respectively to obtain the material features of each sample material;

[0018] The material features are fused and merged with the labeled samples to obtain training samples.

[0019] In a possible implementation manner of the first aspect, the material features include initial features and local features, and based on the encoder, data processing is performed on the multimodal sample materials respectively to obtain the material features of each sample material, including:

[0020] An encoder performs global fusion processing on a sample material to obtain fused material and initial features;

[0021] Each encoder extracts features from the sample material to obtain local features;

[0022] Pack the initial features and local features for transmission to the decoder.

[0023] In a possible implementation manner of the first aspect, each encoder extracts features from the sample material to obtain local features, including:

[0024] Each encoder extracts features from the sample material respectively, and the decoder adopts a cyclic fusion strategy to fuse the extracted features within the module, and then fuses the fused intra-module features between modules to obtain the fusion results as local features.

[0025] In a possible implementation manner of the first aspect, an encoder performs global fusion processing on a sample material to obtain a fused material and obtain an initial feature, including:

[0026] An encoder extracts features from sample materials, and based on a parallel feature fusion network, the extracted features are intra-modally fused and then globally fused to obtain the fusion results as initial features.

[0027] In a second aspect, an embodiment of the present application provides a multimodal data fusion device, comprising:

[0028] An acquisition module, which is used to acquire multi-modal raw data;

[0029] A fusion module, the fusion module is used to input the original data into the fusion network to obtain fused data; wherein the fusion network is obtained based on the global fusion of several material networks, the material networks are used to process different types of data respectively, each material network includes an encoder and a decoder, the weights of the decoder for fusing material features are replicated several times, the weights of each replicated decoder are used to process the material features obtained by an encoder, and the material network is obtained by separate training in end-to-end based on the weights of several merged decoders through training samples.

[0030] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, the multimodal data fusion method provided in any one of the first aspects above is implemented.

[0031] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein:

[0032] The memory is used to store computer programs;

[0033] The processor is used to load and execute a computer program so that the electronic device executes the multimodal data fusion method provided in any one of the first aspects above.

[0034] Compared with the prior art, the beneficial effects of this application are:

[0035] A multimodal data fusion method, device, medium and electronic device proposed in the embodiment of the present application, the method includes: obtaining multimodal original data; inputting the original data into a fusion network to obtain fused data; wherein the fusion network is obtained by global fusion based on several material networks, the material network is used to process different types of data respectively, each material network includes an encoder and a decoder, the weight of the decoder for fusing material features is replicated several times, the weight of each replicated decoder is used to process the material features obtained by an encoder, and the material network is obtained by training the weights of several merged decoders separately in end-to-end through training samples. The present application realizes the fusion of multimodal data by inputting the original data into the fusion network. Since the fusion network is obtained by fusing multiple material networks, the mechanism of its encoder and decoder makes it not affected by the indefinite length sequence of the input data, and effectively combines different modal data, and introduces a global fusion and decoder re-copying strategy, so that it can encode different types of materials separately first, and then encode them as a whole, output more comprehensive global features, and realize high-quality fusion of different modal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of an electronic device of a hardware operating environment involved in an embodiment of the present application;

[0037] Figure 2 A flowchart of a multimodal data fusion method provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a module of a multimodal data fusion device provided in an embodiment of the present application;

[0039] Markings in the figure: 101 - processor, 102 - communication bus, 103 - network interface, 104 - user interface, 105 - memory. DETAILED DESCRIPTION

[0040] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] See attached Figure 1 , attached Figure 1This is a schematic diagram of the structure of an electronic device of the hardware operating environment involved in the embodiment of the present application. The electronic device may include: a processor 101, such as a central processing unit (CPU), a communication bus 102, a user interface 104, a network interface 103, and a memory 105. Among them, the communication bus 102 is used to realize the connection and communication between these components. The user interface 104 may include a display (Display), an input unit such as a keyboard (Keyboard), and the user interface 104 may also include a standard wired interface and a wireless interface. The network interface 103 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 105 may optionally be a storage device independent of the aforementioned processor 101. The memory 105 may be a high-speed random access memory (RAM) memory, or it may be a stable non-volatile memory (NVM), such as at least one disk storage. The processor 101 may be a general-purpose processor, including a central processing unit, a network processor, etc., or it may be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0042] Those skilled in the art will appreciate that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0043] As attached Figure 1 As shown, the memory 105 as a storage medium may include an operating system, a network communication module, a user interface module, and a multimodal data fusion device.

[0044] In the attached Figure 1 In the electronic device shown, the network interface 103 is mainly used for data communication with the network server; the user interface 104 is mainly used for data interaction with the user; the processor 101 and the memory 105 in the present application can be set in the electronic device, and the electronic device calls the multimodal data fusion device stored in the memory 105 through the processor 101, and executes the multimodal data fusion method provided in the embodiment of the present application.

[0045] See attached Figure 2 Based on the hardware device of the foregoing embodiment, the embodiment of the present application provides a multimodal data fusion method, comprising the following steps:

[0046] S10: Acquire multi-modal raw data.

[0047] In the specific implementation process, the original data is the data that needs to be fused, and the modality represents the category of the data, such as video, audio, image, voice, text, symbol, etc. Since fusion is required, at least two modal original data are required to perform the fusion. The embodiments of this application take the three most common types of data as examples for subsequent explanation, namely audio data, text data, and image data.

[0048] S20: Input the original data into the fusion network to obtain fused data; wherein the fusion network is obtained based on global fusion of several material networks, the material networks are used to process different types of data respectively, each material network includes an encoder and a decoder, the weights of the decoder for fusing material features are replicated several times, the weights of each replicated decoder are used to process the material features obtained by an encoder, and the material network is obtained by separate training in end-to-end based on the weights of several merged decoders through training samples.

[0049] In the specific implementation process, both the fusion network and the material network are neural networks. Through the training and learning method of sample plus network, the final fusion network can have the ability to automatically fuse the input data. The material network is designed according to the number of data types processed during the training phase. In this embodiment, three are designed, and each material network processes one type of raw data to ensure the isolation of inter-module data during the training phase. The encoder-decoder belongs to a neural network configuration. The role of the encoder is to transform an input sequence of variable length into a background variable of fixed length, and encode the input sequence information in the background variable, which is no longer affected by the input data sequence, laying the foundation for the subsequent decoder's re-copy strategy fusion.

[0050] The decoder's re-copying strategy mainly involves copying substructures from the input molecule and improving on the copy to generate new molecules with improved properties. This strategy combines skeleton tree generation and adversarial training, and can significantly improve several state-of-the-art molecular optimization methods in multiple tests. In end-to-end training, the complex intermediate process of deep learning is ignored and replaced directly with a single neural network, so that the original sample without any processing can be directly input to output the result. Each material network is trained in this way, which can fully extract the features under the data type, and then obtain the final fusion network through global fusion, and combined with the decoder's re-copying strategy, the material is encoded individually and then encoded as a whole to output more comprehensive global features.

[0051] In this embodiment, multimodal data fusion is achieved by inputting the original data into the fusion network. Since the fusion network is obtained by fusing multiple material networks, the mechanism of its encoder and decoder will not be affected by the indefinite length sequence of the input data, and effectively combine different modal data. In addition, a global fusion and decoder re-copying strategy is introduced to enable it to encode different types of materials separately first, and then encode them as a whole, output more comprehensive global features, and achieve high-quality fusion of different modal data.

[0052] In one embodiment, before inputting the original data into the fusion network to obtain the fused data, the method further includes:

[0053] Build encoders and decoders;

[0054] The weight of the decoder used to fuse the material features is copied several times, and the copied decoder weights are used to process the material features obtained by the encoder respectively;

[0055] The weights of several decoders are combined, and based on the combined weights of the decoders, they are trained separately in an end-to-end manner through training samples to obtain a material network;

[0056] Based on several material networks, global fusion is performed to obtain a fusion network.

[0057] In the specific implementation process, after the decoder weights used to fuse the material features are copied multiple times, the copied decoder weights are not merged and are trained separately; then the detailed features of each modality can be further mined by processing the features of each different modality separately; and the end-to-end training of the merged weights ensures that the detailed features of each material are also learned, and then the final part to be output is cut out separately, thereby realizing the fusion of different modal data. Among them, the global fusion of the material network is to globally merge the decoder and keep the encoder unchanged, that is: based on a number of material networks, global fusion is performed to obtain a fusion network, including:

[0058] Based on several material networks, the decoders are globally merged and the encoders are kept unchanged to obtain a fusion network.

[0059] In fact, it is to first ensure the accuracy of the three material networks under single extraction, and then realize the overall encoding through the merged decoder to improve the comprehensiveness of the global features. In this way, the fused data can take into account the characteristics of single modal data and multi-modal data.

[0060] In one embodiment, before inputting the original data into the fusion network to obtain the fused data, the method further includes:

[0061] Based on the encoder, data processing is performed on the multi-modal sample materials respectively to obtain the material features of each sample material;

[0062] The material features are fused and merged with the labeled samples to obtain training samples.

[0063] In the specific implementation process, the encoder is used to extract features from the sample material, and data processing methods such as semantic segmentation, feature extraction, and feature fusion can be used. The extracted material features are used as part of the training samples and trained together with the labeled samples, that is, other sample materials that have been labeled with the required features, to expand the number of samples and make the trained network more generalizable.

[0064] In one embodiment, the material features include initial features and local features. Based on the encoder, data processing is performed on the multimodal sample materials respectively to obtain the material features of each sample material, including:

[0065] An encoder performs global fusion processing on a sample material to obtain fused material and initial features;

[0066] Each encoder extracts features from the sample material to obtain local features;

[0067] Pack the initial features and local features for transmission to the decoder.

[0068] During the specific implementation process, the encoder first globally fuses the sample materials to obtain an initial fused feature of all the materials, and then each encoder extracts the local features of the materials respectively; the initial features obtained by the encoder are packaged together with the local features and transmitted to the decoder; the decoder receives various types of material features transmitted by the entire encoder, and performs a fusion operation on these feature information in turn, and finally realizes the deep fusion of various types of materials.

[0069] Specifically, the fusion is divided into single feature fusion and global feature fusion:

[0070] On the one hand, each encoder extracts features from the sample material to obtain local features, including:

[0071] Each encoder extracts features from the sample material respectively, and the decoder adopts a cyclic fusion strategy to fuse the extracted features within the module, and then fuses the fused intra-module features between modules to obtain the fusion results as local features.

[0072] In the specific implementation process, for the fusion of a single feature, the encoder extracts the local features of each material, and the decoder is directly connected to the encoder of the material. The received features are directly the local spatial features of each material. At this time, different modalities need to learn from each other. The decoder adopts a cyclic fusion strategy, that is, the features of each modality are fused once in each cycle. In the decoder, intra-modal fusion is first performed, and then the fused intra-modal features are inter-modally fused. The fusion of multiple modal data is achieved through a cycle, and the fusion result of a single feature is obtained after the cycle. Since there are three modalities, three inter-modal fusions are required in each cycle. After the fusion is completed, the fusion result of a single feature is obtained.

[0073] On the other hand, an encoder performs global fusion processing on a sample material to obtain a fused material and obtain initial features, including:

[0074] An encoder extracts features from sample materials, and based on a parallel feature fusion network, the extracted features are intra-modally fused and then globally fused to obtain the fusion results as initial features.

[0075] In the specific implementation process, for the fusion of global features, two parallel feature fusion networks are used: one is a separate encoding network for each modality, and the outputs of the parallel encoding networks of each modality are connected by a parallel decoding network. In the decoder, the features of a single modality are further fused intra-modally, and then the outputs of the parallel decoding networks are globally fused, that is, inter-modal fusion, to obtain the fusion result of the global features.

[0076] Subsequently, in the decoder, the fusion result of the global features obtained by the encoder is used as the initial features of the decoder, and each modality is fused intra-modally, and the features after intra-modal fusion are globally fused again, so that the same partial features are repeated, and the inter-modal fusion and intra-modal fusion in different modalities are more obvious; then after a global copy of the decoder weights, the copied content is segmented, and the segmented decoder weights are matched one-to-one with the encoders of each modality, and finally all decoders are globally fused to achieve global merging of decoders.

[0077] See attached Figure 3 Based on the same inventive concept as in the above-mentioned embodiment, the embodiment of the present application further provides a multimodal data fusion device, comprising:

[0078] An acquisition module, which is used to acquire multi-modal raw data;

[0079] A fusion module, the fusion module is used to input the original data into the fusion network to obtain fused data; wherein the fusion network is obtained based on the global fusion of several material networks, the material networks are used to process different types of data respectively, each material network includes an encoder and a decoder, the weights of the decoder for fusing material features are replicated several times, the weights of each replicated decoder are used to process the material features obtained by an encoder, and the material network is obtained by separate training in end-to-end based on the weights of several merged decoders through training samples.

[0080] Those skilled in the art should understand that the division of the various modules in the embodiment is only a division of logical functions, and can be fully or partially integrated into one or more actual carriers in actual application, and these modules can be all implemented in the form of software called by the processing unit, or all implemented in the form of hardware, or implemented in the form of a combination of software and hardware. It should be noted that each module in the multimodal data fusion device in this embodiment corresponds one by one to each step in the multimodal data fusion method in the aforementioned embodiment. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned multimodal data fusion method, which will not be repeated here.

[0081] Based on the same inventive concept as in the aforementioned embodiment, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, a multimodal data fusion method as provided in the embodiment of the present application is implemented.

[0082] Based on the same inventive concept as in the above-mentioned embodiment, an embodiment of the present application further provides an electronic device, including a processor and a memory, wherein:

[0083] The memory is used to store computer programs;

[0084] The processor is used to load and execute a computer program so that the electronic device executes the multimodal data fusion method provided in the embodiment of the present application.

[0085] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0086] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0087] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0088] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0089] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0090] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0092] In summary, the present application provides a multimodal data fusion method, device, medium and electronic device, the method comprising: obtaining multimodal original data; inputting the original data into a fusion network to obtain fused data; wherein the fusion network is obtained by global fusion based on several material networks, the material network is used to process different types of data respectively, each material network includes an encoder and a decoder, the weight of the decoder for fusing material features is replicated several times, the weight of each replicated decoder is used to process the material features obtained by an encoder, and the material network is obtained by training the weights of several merged decoders separately in end-to-end through training samples. The present application realizes the fusion of multimodal data by inputting the original data into the fusion network. Since the fusion network is obtained by fusing multiple material networks, the mechanism of its encoder and decoder makes it not affected by the indefinite length sequence of the input data, and effectively combines different modal data, and introduces a global fusion and decoder re-copying strategy, so that it can encode different types of materials separately first, and then encode them as a whole, output more comprehensive global features, and realize high-quality fusion of different modal data.

[0093] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A multimodal data fusion method, characterized in that: The following steps are involved: Obtain multi-modal raw data; Input the original data into a fusion network to obtain fused data; wherein the fusion network is obtained by global fusion based on several material networks, the material networks are used to process different types of data respectively, each of the material networks includes an encoder and a decoder, the weight of the decoder used to fuse material features is replicated several times, each replicated decoder weight is used to process the material feature obtained by one encoder, and the material network is obtained by training the weights of several merged decoders separately in end-to-end through training samples; before inputting the original data into the fusion network to obtain the fused data, the method further includes: Based on the encoder, data processing is performed on the multimodal sample materials respectively to obtain the material features of each of the sample materials; Fusing the material features and merging them with the labeled samples to obtain the training samples; The material features include initial features and local features. Based on the encoder, data processing is performed on the multimodal sample materials respectively to obtain the material features of each of the sample materials, including: One of the encoders performs global fusion processing on one of the sample materials to obtain a fused material and the initial features; Each of the encoders extracts features from the sample material to obtain the local features; Packing the initial features and the local features for transmission to the decoder; Each of the encoders extracts features from the sample material to obtain the local features, including: Each of the encoders extracts features from the sample material respectively, and the decoder adopts a cyclic fusion strategy to perform intra-modal fusion on the extracted features, and inter-modal fusion on the fused intra-modal features to obtain a fusion result as the local feature; The one encoder performs global fusion processing on one sample material to obtain a fused material and the initial feature, including: The encoder extracts features from the sample material, and based on a parallel feature fusion network, performs intra-module fusion and then global fusion on the extracted features to obtain a fusion result as the initial feature; Before inputting the original data into the fusion network to obtain the fused data, the method further includes: constructing the encoder and the decoder; Copying the weight of the decoder used for fusing the material features several times, and using the copied weight of the decoder to respectively process the material features obtained by the encoder; Combining the weights of several decoders, and training the combined weights of the decoders separately in an end-to-end manner through the training samples to obtain the material network; Performing global fusion based on several of the material networks to obtain the fusion network; The globally fusing the plurality of material networks to obtain the fused network comprises: The decoders are globally merged based on several of the material networks, and the encoders are kept unchanged to obtain the fusion network.

2. A multimodal data fusion device, characterized in that: include: An acquisition module, wherein the acquisition module is used to acquire multi-modal raw data; A fusion module, the fusion module is used to input the original data into a fusion network to obtain fused data; wherein the fusion network is obtained by global fusion based on several material networks, the material networks are used to process different types of data respectively, each of the material networks includes an encoder and a decoder, the weight of the decoder used to fuse material features is replicated several times, the weight of each replicated decoder is used to process the material features obtained by one encoder, and the material network is obtained by training separately through training samples in end-to-end based on the weights of several merged decoders; before the raw data is input into the fusion network to obtain the fused data, it also includes: Based on the encoder, data processing is performed on the multimodal sample materials respectively to obtain the material features of each of the sample materials; Fusing the material features and merging them with the labeled samples to obtain the training samples; The material features include initial features and local features. Based on the encoder, data processing is performed on the multimodal sample materials respectively to obtain the material features of each of the sample materials, including: One of the encoders performs global fusion processing on one of the sample materials to obtain a fused material and the initial features; Each of the encoders extracts features from the sample material to obtain the local features; Packing the initial features and the local features for transmission to the decoder; Each of the encoders extracts features from the sample material to obtain the local features, including: Each of the encoders extracts features from the sample material respectively, and the decoder adopts a cyclic fusion strategy to perform intra-modal fusion on the extracted features, and inter-modal fusion on the fused intra-modal features to obtain a fusion result as the local feature; The one encoder performs global fusion processing on one sample material to obtain a fused material and the initial feature, including: The encoder extracts features from the sample material, and based on a parallel feature fusion network, performs intra-module fusion and then global fusion on the extracted features to obtain a fusion result as the initial feature; Before inputting the original data into the fusion network to obtain the fused data, the method further includes: constructing the encoder and the decoder; Copying the weight of the decoder used for fusing the material features several times, and using the copied weight of the decoder to respectively process the material features obtained by the encoder; Combining the weights of several decoders, and training the combined weights of the decoders separately in an end-to-end manner through the training samples to obtain the material network; Performing global fusion based on several of the material networks to obtain the fusion network; The globally fusing the plurality of material networks to obtain the fused network comprises: The decoders are globally merged based on several of the material networks, and the encoders are kept unchanged to obtain the fusion network.

3. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is loaded and executed by the processor, the multimodal data fusion method as claimed in claim 1 is implemented.

4. An electronic device, characterized in that: comprising a processor and a memory, wherein: The memory is used to store computer programs; The processor is used to load and execute the computer program so that the electronic device executes the multimodal data fusion method as claimed in claim 1.

Images