A multi-modal data fusion method, system, device, and medium

Abstract

The application relates to the technical field of data processing, and more particularly to a multi-modal data fusion method, system, device and medium. The method comprises the following steps: obtaining target audio and target video; obtaining audio features by inputting the target audio into a pre-trained first Transformer model; obtaining video features by inputting the target video into a pre-trained second Transformer model; performing mid-term fusion on the audio features and the video features by using a fusion model to obtain fusion features; and performing post-term fusion on the fusion features, the audio features and the video features. The multi-modal data fusion method has improved robustness, the accuracy of the fused features is improved, and the efficiency of data processing is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a multimodal data fusion method, system, device, and medium. Background Technology

[0002] Common unimodal data includes text, images, audio, and video, while multimodal data refers to data containing two or more data types simultaneously. With the development of artificial intelligence technology, multimodal data applications are becoming increasingly widespread. In the field of deep learning, various types of information can be abstracted into multiple signals, such as video frames and audio tracks, web images and their subtitles, as well as instructional videos and audio recordings.

[0003] Existing multimodal fusion techniques employ a late-stage fusion approach, which involves encoding each modality using separate models and then simply combining their output representations in a final step. However, this method still suffers from two problems: poor robustness and low accuracy of the fused features. Summary of the Invention

[0004] Based on the above-mentioned technical problems, the present invention aims to achieve the fusion of multimodal data by combining mid-term fusion and late-term fusion.

[0005] The first aspect of this invention provides a multimodal data fusion method, the method comprising:

[0006] Acquire the target audio and target video;

[0007] The target audio is processed through a pre-trained first Transformer model to obtain audio features;

[0008] The target video is processed using a pre-trained second Transformer model to obtain video features;

[0009] The audio features and the video features are fused together using a fusion model to obtain fused features.

[0010] The fusion features, audio features, and video features are then fused together in a later stage.

[0011] In some embodiments of the present invention, the fusion model includes an attention bottleneck module and a feature fusion module.

[0012] In some embodiments of the present invention, the audio features and the video features are fused in an intermediate stage using a fusion model to obtain fused features, including:

[0013] The audio features and video features are passed through the attention bottleneck module to obtain bottlenecked audio features and bottlenecked video features;

[0014] The bottlenecked audio features and the bottlenecked video features are fused in the middle stage using a fusion model to obtain fused features.

[0015] In some embodiments of the present invention, the dimension of the attention bottleneck module is smaller than the dimension of the audio features and smaller than the dimension of the video features.

[0016] In some embodiments of the present invention, the audio features and the video features are passed through an attention bottleneck module to obtain bottlenecked audio features and bottlenecked video features, including:

[0017] The audio features are input into the attention bottleneck module to restrict the attention of the first preset part, and the audio features after the bottleneck are obtained.

[0018] The video features are input into the attention bottleneck module to restrict the second preset part of the attention, resulting in video features after the bottleneck.

[0019] In some embodiments of the present invention, the fusion of the fusion features, the audio features, and the video features is performed in a post-processing fusion, including:

[0020] The final layer of features of the fusion features, the audio features, and the video features are then fused together in a later stage.

[0021] In some embodiments of the present invention, after the post-fusion of the fused features, the audio features, and the video features, the method further includes:

[0022] Each classification loss is added to the output after fusion, wherein the classification loss adopts the cross-entropy function.

[0023] A second aspect of the present invention provides a multimodal data fusion system, the system comprising:

[0024] The acquisition module is configured to acquire target audio and target video.

[0025] The first single-mode extraction module is configured to obtain audio features from the target audio through a pre-trained first Transformer model.

[0026] The second single-mode extraction module is configured to obtain video features from the target video through a pre-trained second Transformer model.

[0027] The intermediate fusion module is configured to perform intermediate fusion of the audio features and the video features through a fusion model to obtain fused features;

[0028] The post-fusion module is configured to perform post-fusion of the fusion features, the audio features, and the video features.

[0029] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the multimodal data fusion method described in the embodiments of the present invention.

[0030] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal data fusion method described in various embodiments of the present invention.

[0031] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0032] The multimodal data fusion described in this application improves robustness, accuracy of fused features, and data processing efficiency. In particular, the attention bottleneck module of the fusion model effectively limits the attention of some single-modal data, reducing fusion pressure and allowing mid-stage and late-stage fusion to work together effectively, thereby improving the robustness and accuracy of the fusion.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0034] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0035] Figure 1 This illustration shows a schematic diagram of the steps of a multimodal data fusion method according to an exemplary embodiment of this application;

[0036] Figure 2 A schematic diagram of the attention bottleneck module of the fusion model in an exemplary embodiment of this application is shown;

[0037] Figure 3 This illustration shows a schematic diagram of another multimodal data fusion method process in an exemplary embodiment of this application;

[0038] Figure 4 A schematic diagram of a multimodal data fusion system structure according to an exemplary embodiment of this application is shown;

[0039] Figure 5 This illustration shows a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of this application. Detailed Implementation

[0040] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of this application. It will be apparent to those skilled in the art that this application can be implemented without one or more of these details. In other instances, to avoid confusion with this application, some technical features well-known in the art have not been described.

[0041] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or combinations thereof.

[0042] Exemplary embodiments according to this application will now be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments may be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. The drawings are not drawn to scale, and some details may be enlarged and omitted for clarity. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0043] The following is in conjunction with the instruction manual appendix. Figure 1 -Appendix Figure 5 Several embodiments are given to describe exemplary implementations according to this application. It should be noted that the following application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the implementations of this application are not limited in any way. Rather, the implementations of this application can be applied to any applicable scenario.

[0044] In some exemplary embodiments of this application, a multimodal data fusion method is provided, such as... Figure 1 As shown, the method includes:

[0045] S1. Obtain the target audio and target video;

[0046] S2. Obtain audio features from the target audio through a pre-trained first Transformer model;

[0047] S3. Obtain video features from the target video through a pre-trained second Transformer model;

[0048] S4. Perform mid-term fusion on the audio features and the video features through a fusion model to obtain fusion features;

[0049] S5. Perform late-stage fusion on the fusion features, the audio features, and the video features.

[0050] One of the first Transformer model and the second Transformer model processes audio, and the other processes video. Structurally, the first Transformer model includes a feature extraction module, a tokenizer, a Transformer encoder, and a classifier, while the second Transformer model includes a feature extraction module, a Transformer encoder, and a classifier. The first Transformer model and the second Transformer model need to be pre-trained before they can be used to identify unimodal data. The feature extraction module includes a feature input layer, a convolutional structure, and a splicing layer connected in sequence. During training, a conventional training method in the prior art can be adopted, such as obtaining video image training samples, configuring the parameters of the second Transformer model, initializing the iteration times of the second Transformer model, defining the loss function of the second Transformer model, inputting the OCT image training samples into the second Transformer model for training, and terminating the training when the training reaches the preset iteration times. The first Transformer model needs to convert audio samples into text, so a tokenizer is required here, and other aspects are the same as the conventional training methods adopted in the prior art.

[0051] In a preferred implementation manner, the fusion model includes an attention bottleneck module (as shown in Figure 2 ), and a feature fusion module. Performing mid-term fusion on the audio features and the video features through the fusion model to obtain fusion features includes: obtaining post-bottleneck audio features and post-bottleneck video features from the audio features and the video features through the attention bottleneck module; performing mid-term fusion on the post-bottleneck audio features and the post-bottleneck video features through the fusion model to obtain fusion features. The dimension of the attention bottleneck module here is smaller than the dimension of the audio features and smaller than the dimension of the video features. Assuming the dimension of the two modalities is N and the dimension of the fusion bottleneck is M, that is, M < N, then the computational complexity can be reduced from quadratic complexity N 2The complexity decreases to linear MN. The attention bottleneck module organizes and compresses information from each modality (audio, video) before sharing it with other modalities, while still preserving the attention mechanism within the modality.

[0052] In a preferred implementation, obtaining bottlenecked audio features and bottlenecked video features by passing the audio features and video features through an attention bottleneck module includes: inputting the audio features into the attention bottleneck module to restrict a first preset portion of attention to obtain bottlenecked audio features; and inputting the video features into the attention bottleneck module to restrict a second preset portion of attention to obtain bottlenecked video features.

[0053] In some embodiments of this application, such as Figure 3 As shown, the post-fusion of the fused features, audio features, and video features includes: performing post-fusion on the last layer of features of the fused features, audio features, and video features. Further, after performing post-fusion on the fused features, audio features, and video features, the method further includes: adding their respective classification losses to the output after post-fusion, wherein the classification loss adopts the cross-entropy function.

[0054] Currently, two common approaches to single-modal data fusion are early fusion and late fusion. Early fusion involves all layers in the transformer being cross-modal, concatenating inputs from all modalities at input time. Late fusion involves all layers in the transformer being single-modal, without exchanging cross-modal information in the transformer encoder; instead, the individual features are concatenated at the end for downstream task training. This application uses mid-term and late-term fusion to restrict cross-modal flow to the mid-to-late layers of the network, allowing early layers to focus on learning and extracting single-modal information. In particular, mid-term fusion utilizes an attention bottleneck module structure, preserving the features of each modality while fusing multimodal features. Furthermore, compared to early fusion, mid-term fusion reduces computational complexity to half, offering simplicity and speed while saving on model training and fusion costs.

[0055] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention.

[0056] In some exemplary embodiments of this application, a multimodal data fusion system is also provided, which executes the multimodal data fusion method described in various embodiments of this application, such as... Figure 4 As shown, the system includes:

[0057] The acquisition module is configured to acquire target audio and target video.

[0058] The first single-mode extraction module is configured to obtain audio features from the target audio through a pre-trained first Transformer model.

[0059] The second single-mode extraction module is configured to obtain video features from the target video through a pre-trained second Transformer model.

[0060] The intermediate fusion module is configured to perform intermediate fusion of the audio features and the video features through a fusion model to obtain fused features;

[0061] The post-fusion module is configured to perform post-fusion of the fusion features, the audio features, and the video features.

[0062] The fusion model includes an attention bottleneck module and a feature fusion module, which performs intermediate fusion of audio features and video features through the fusion model. The multimodal data fusion system described in this application improves the robustness of multimodal data fusion, enhances the accuracy of fused features, and increases data processing efficiency. In particular, the attention bottleneck module of the fusion model effectively limits the attention of some single-modal data, reducing fusion pressure and allowing intermediate and final fusion to cooperate appropriately, thereby improving the robustness and accuracy of the fusion.

[0063] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application.

[0064] It should also be emphasized that the system provided in this application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0065] Please refer to the following. Figure 5 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 5As shown, the electronic device 2 includes: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected via the bus 202. The memory 201 stores a computer program that can run on the processor 200. When the processor 200 runs the computer program, it executes the multimodal data fusion method provided in any of the foregoing embodiments of this application.

[0066] The memory 201 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 203 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0067] Bus 202 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs. After receiving an execution instruction, the processor 200 executes the program. The multimodal data fusion method disclosed in any of the foregoing embodiments of this application can be applied to the processor 200, or implemented by the processor 200.

[0068] The processor 200 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 200 or by instructions in software form. The processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 201. The processor 200 reads the information in memory 201 and, in conjunction with its hardware, completes the steps of the above method.

[0069] This application also provides a computer-readable storage medium corresponding to the multimodal data fusion method provided in the foregoing embodiments, wherein a computer program is stored thereon, and the computer program, when run by a processor, executes the multimodal data fusion method provided in any of the foregoing embodiments.

[0070] In addition, examples of the computer-readable storage medium may include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be described in detail here.

[0071] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimodal data fusion method provided in any of the foregoing embodiments. The method includes: acquiring target audio and target video; obtaining audio features from the target audio using a pre-trained first Transformer model; obtaining video features from the target video using a pre-trained second Transformer model; performing intermediate fusion of the audio features and the video features using a fusion model to obtain fused features; and performing post-fusion of the fused features, the audio features, and the video features.

[0072] It should be noted that the algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used in conjunction with the teachings herein. The required structure for constructing such devices is obvious from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application. Numerous specific details are set forth in the specification provided herein. However, it is to be understood that embodiments of this application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0073] Those skilled in the art will understand that the various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art should understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the virtual machine creation apparatus according to embodiments of this application.

[0074] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal data fusion method, characterized in that, The method includes: Acquire the target audio and target video; The target audio is processed through a pre-trained first Transformer model to obtain audio features; The target video is processed using a pre-trained second Transformer model to obtain video features; The audio features and the video features are fused in an intermediate stage using a fusion model to obtain fused features; The fusion features, audio features, and video features are then fused together in a later stage. The fusion model includes an attention bottleneck module and a feature fusion module; The audio features and video features are fused in the middle stage using a fusion model to obtain fused features, including: The audio features and video features are passed through the attention bottleneck module to obtain bottlenecked audio features and bottlenecked video features; The bottlenecked audio features and the bottlenecked video features are fused in the middle stage using a fusion model to obtain fused features; In particular, when processed by the attention bottleneck module, the intramodal attention mechanisms of the audio features and video features are preserved.

2. The multimodal data fusion method according to claim 1, characterized in that, The dimension of the attention bottleneck module is smaller than the dimension of the audio features and smaller than the dimension of the video features.

3. The multimodal data fusion method according to claim 2, characterized in that, The audio features and video features are passed through an attention bottleneck module to obtain bottlenecked audio features and bottlenecked video features, including: The audio features are input into the attention bottleneck module to restrict the attention of the first preset part, and the audio features after the bottleneck are obtained. The video features are input into the attention bottleneck module to restrict the second preset part of the attention, resulting in video features after the bottleneck.

4. The multimodal data fusion method according to claim 1, characterized in that, The post-processing fusion of the fused features, the audio features, and the video features includes: The final layer of features of the fusion features, the audio features, and the video features are then fused together in a later stage.

5. The multimodal data fusion method according to claim 1, characterized in that, After the post-fusion of the fused features, the audio features, and the video features, the process further includes: Each classification loss is added to the output after fusion, wherein the classification loss adopts the cross-entropy function.

6. A multimodal data fusion system, characterized in that, The system includes: The acquisition module is configured to acquire target audio and target video. The first single-mode extraction module is configured to obtain audio features from the target audio through a pre-trained first Transformer model. The second single-mode extraction module is configured to obtain video features from the target video through a pre-trained second Transformer model. The intermediate fusion module is configured to perform intermediate fusion of the audio features and the video features through a fusion model to obtain fused features; The post-fusion module is configured to perform post-fusion of the fusion features, the audio features, and the video features; The fusion model includes an attention bottleneck module and a feature fusion module; The audio features and video features are fused in the middle stage using a fusion model to obtain fused features, including: The audio features and video features are passed through the attention bottleneck module to obtain bottlenecked audio features and bottlenecked video features; The bottlenecked audio features and the bottlenecked video features are fused in the middle stage using a fusion model to obtain fused features; In particular, when processed by the attention bottleneck module, the intramodal attention mechanisms of the audio features and video features are preserved.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal data fusion method according to any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal data fusion method according to any one of claims 1-5.

Images