Deep learning framework based on multi-modal data fusion and adaptive learning

Abstract

The invention relates to a deep learning framework based on multi-modal data fusion and adaptive learning, which comprises a multi-modal data fusion module, an adaptive learning module and a deep learning model module, fusing in different stages of feature extraction by adopting a plurality of fusion strategies to obtain fused features; the deep learning model module is used for performing feature extraction on data of different modalities according to different depth models, performing prediction according to fusion features obtained by the multi-modal data fusion module, and outputting a prediction result; and the adaptive learning module is used for dynamically adjusting hyper-parameters of the depth model by adopting a learning mechanism. Compared with the prior art, the method has the advantages of providing more comprehensive information representation, self-adaptive learning ability, wider application prospect and the like.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a deep learning framework based on multimodal data fusion and adaptive learning. Background Art

[0002] With the rapid development of big data and artificial intelligence technology, deep learning has become an important tool for solving complex problems. However, existing deep learning models often have limitations when processing multimodal data and cannot fully utilize the complementary information between different modal data. The existing problems are mainly reflected in the following aspects:

[0003] 1. Data requirements: Deep learning models require a large amount of data for effective training. For multidimensional data, especially high-dimensional data, it can be a challenge to obtain enough labeled data. Lack of sufficient data may result in the model being unable to fully learn the intrinsic structure and characteristics of the data, thus affecting the performance of the model.

[0004] 2 Computing resources: Deep learning models require a lot of computing resources, including GPUs and high-performance computing clusters. For multi-dimensional data, the complexity and computational effort of the model usually increase, so more computing resources are needed to support model training and reasoning. This may increase the training time and cost of the model.

[0005] 3. Model interpretability: Deep learning models are generally less interpretable, especially when dealing with multidimensional data. Due to the complex network structure and nonlinear transformations within the model, it is difficult to intuitively understand how the model extracts features and makes predictions from the input data. This may lead to distrust in the model's prediction results and limit its use in certain application scenarios that require a high degree of interpretability.

[0006] 4. Dimensional curse: When the dimension of the data is very high, you may encounter the so-called "dimensionality curse". In high-dimensional space, the sparsity of data increases, making traditional distance and similarity measurement methods invalid. This may cause the performance of deep learning models to degrade when processing high-dimensional data, because the model may not be able to effectively capture the useful information in the data.

[0007] 5. Hyperparameter Tuning: For multi-dimensional data, the complexity and number of hyperparameters of deep learning models usually increase. Hyperparameter tuning is crucial to the performance of the model, but it is also a time-consuming and experience-intensive process. Inappropriate hyperparameter settings may lead to poor model performance or unstable training process.

[0008] 6. Large-scale, high-quality multimodal datasets are difficult to collect: Due to the limitations of different imaging devices and sensors, it is difficult to obtain large amounts of high-quality multimodal data.

[0009] 7. High cost of data labeling: The processing and labeling of multimedia data such as images and videos requires a lot of time and manpower, resulting in insufficient labeled data and affecting the training effect of the model.

[0010] 8 Multimodal data need to be matched in pairs for research: In practical applications, the absence of any modality will result in a reduction in available data, making data fusion difficult.

[0011] 9. Selection of fusion strategy: How to effectively fuse multiple data sources remains a challenge and needs to be optimized in combination with specific application scenarios to obtain comprehensive and accurate model output.

[0012] 10. The structure of multimodal deep learning models is complex: The model usually consists of multiple branches, each of which is responsible for processing one type of data, which increases the complexity and computational cost of model training.

[0013] 11. Data imbalance problem: Different types of data may appear with huge differences in frequency in videos, causing the model to over-rely on certain data sources, thereby reducing the robustness and accuracy of the model.

[0014] 12. Model sensitivity to specific data distribution: Due to the diversity and complexity of multimodal data, deep learning models are often prone to overfitting to specific data distributions, resulting in insufficient generalization capabilities.

[0015] 13 Challenges of model robustness: In some cases, a single data source may be interfered with or fail, but the model needs to be able to comprehensively utilize information from other data sources to make up for this lack and improve robustness.

[0016] 14 High training cost: Due to the complex structure and numerous parameters of multimodal deep learning models, a lot of computing resources and time are required to train and optimize the models.

[0017] 15. The need for real-time processing: In some application scenarios, such as autonomous driving and smart security, the model needs to be able to process multimodal data in real time and respond quickly, which places higher requirements on the performance and efficiency of the model.

[0018] In addition, existing models often require a lot of parameter adjustments when adapting to new environments and new tasks, and lack sufficient adaptive capabilities. M. Ben-Ezra's "Segmentation with invisible keying signal" proposed a new deep learning framework, but the training of multi-dimensional data is insufficient and it cannot effectively learn the intrinsic structure and characteristics of the data. PW Patil, KM Bradar, A. Dudhane and S. Murala, "An End-to-End Edge Aggregation Network for Moving Object Segmentation," proposed data requirements and ways to deal with multi-dimensional disasters, but did not consider the impact of the complexity of multi-dimensional data on the learning model. J. Singh, S. Murala and GSR Kosuru, in "Multi Domain Learning for Motion Magnification," proposed a multi-dimensional and multi-domain learning model, but the application scenarios it set have certain limitations and cannot take into account the characteristics of a wider range of application scenarios. PN Williams and K. Li, in "Black-Box Sparse Adversarial Attack via Multi-Objective Optimisation CVPR Proceedings," proposed a multi-angle, multi-dimensional optimization CVPR process, but the interpretability of the learning model is usually poor, especially when dealing with multi-dimensional data. Due to the complex network structure and nonlinear transformation within the model, it is difficult to intuitively understand how the model extracts features and makes predictions from the input data.

[0019] In summary, deep learning models face constraints in terms of data requirements, computing resources, model interpretability, dimensionality curse, and hyperparameter adjustment when processing multidimensional data. Summary of the invention

[0020] The purpose of the present invention is to provide a deep learning framework based on multimodal data fusion and adaptive learning to improve the robustness and prediction accuracy of deep learning models.

[0021] The purpose of the present invention can be achieved by the following technical solutions:

[0022] A deep learning framework based on multimodal data fusion and adaptive learning, including a multimodal data fusion module, an adaptive learning module and a deep learning model module.

[0023] The multimodal data fusion module is used to obtain data of different modes and fuse them at different stages of feature extraction using a variety of fusion strategies to obtain fusion features;

[0024] The deep learning model module is used to extract features of data of different modalities according to different deep models, and to make predictions according to the fusion features obtained by the multimodal data fusion module, and output prediction results;

[0025] The adaptive learning module is used to dynamically adjust the hyperparameters of the deep model using a learning mechanism.

[0026] Furthermore, the multimodal data fusion module includes:

[0027] Data acquisition submodule: used to acquire data of different modalities, including video data, text data, and audio data;

[0028] Preprocessing submodule: used for preprocessing the data of different modalities;

[0029] Fusion submodule: It is used to adopt multiple fusion strategies to fuse at different stages of feature extraction to obtain fusion features.

[0030] Furthermore, the fusion submodule includes:

[0031] Early fusion unit: used to fuse the extracted early features of different modalities to obtain early fusion features;

[0032] Mid-term fusion unit: used to fuse the early fusion features and the extracted mid-term features of different modalities to obtain mid-term fusion features;

[0033] Late fusion unit: used to fuse the mid-term fusion features and the extracted late features of different modalities to obtain the final fusion features.

[0034] Furthermore, the multimodal data fusion module further includes an adversarial generation submodule, and the adversarial generation submodule includes:

[0035] Early adversarial feature generation unit: used to generate early feature adversarial samples and add them to the early features of the corresponding modality to perform early feature fusion together;

[0036] Mid-term adversarial feature generation unit: used to generate mid-term feature adversarial samples and add them to the mid-term features of the corresponding modality to perform mid-term feature fusion together;

[0037] Late adversarial feature generation unit: used to generate late feature adversarial samples and add them to the late features of the corresponding modality to perform late feature fusion together.

[0038] Furthermore, the step of preprocessing the video data includes:

[0039] The acquired video data is uniformly converted into a standard format and decomposed into frame sequences, and each frame image is uniformly scaled and normalized.

[0040] Furthermore, the step of preprocessing the text data includes:

[0041] The acquired text data is cleaned to remove noise information, wherein the text cleaning includes removing irrelevant characters, punctuation errors, special symbols, extra spaces and line breaks in the text data;

[0042] The text data after text cleaning is segmented and converted into vectors using a pre-trained word embedding model.

[0043] Furthermore, the step of preprocessing the audio data includes:

[0044] Using an audio noise reduction algorithm to remove background noise from the acquired audio data to obtain noise-reduced audio data;

[0045] The denoised audio data is segmented to obtain independent speech segment data.

[0046] Furthermore, the deep learning model includes one of CNN, RNN, and Transformer.

[0047] Furthermore, the learning mechanism includes one or more of meta-learning, reinforcement learning and data enhancement.

[0048] Furthermore, the feature extraction stage of the deep learning model module includes three stages, namely, an early stage, a mid-stage and a late stage, which are used to extract early features, mid-stage features and late features, respectively.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] (1) Since data of different modalities may be subject to different noises and interferences, the multimodal data fusion module of the present invention can combine information from different perceptual modalities by adopting a variety of fusion strategies to provide more comprehensive and accurate input information for the deep learning model. Moreover, by performing fusion at different stages of feature extraction, the deep learning model can capture more detailed and comprehensive information, making up for the limitations of single modality data in expressing information, helping the model to better understand the meaning and contextual relationship behind the data, thereby obtaining a more comprehensive information representation and improving the robustness and prediction accuracy of the deep learning model.

[0051] (2) In the fusion submodule of the present invention, the early fusion unit, the mid-term fusion unit and the late fusion unit are used to perform early, mid-term and late fusion on the features of different modalities, respectively, so as to combine the features of each stage and obtain more representative and discriminative information.

[0052] (3) The multimodal data fusion module of the present invention includes an adversarial generation submodule to generate feature adversarial samples in each feature extraction stage. By adding them to the features of the corresponding stage for joint fusion, the model can learn to identify and resist such false data during the training process, thereby further improving the prediction accuracy and reliability of the model.

[0053] (4) The multimodal data fusion of the present invention not only provides richer input features, but also accelerates the training process of the neural network. Traditional neural network training requires a lot of computing resources, while multimodal data fusion can provide more information, thereby reducing the amount of computing in the network training process. At the same time, it can also improve the generalization performance of the model and enhance the model's adaptability to unknown data.

[0054] (5) The multimodal data fusion of the present invention can be applied in many fields, including autonomous driving, medical diagnosis, intelligent speech recognition, etc. These fields require processing of a large amount of multimodal data, and the deep learning framework of the present invention can provide a more efficient and accurate solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the deep learning framework of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0057] This embodiment provides a deep learning framework based on multimodal data fusion and adaptive learning. The framework aims to provide efficient and accurate solutions for various complex tasks by integrating multimodal data (such as images, text, audio, etc.) and adaptive learning mechanisms. The framework combines multimodal data fusion technology and adaptive learning algorithms to automatically adjust the model structure and parameters to adapt to different tasks and data sets. Figure 1 As shown, the framework includes a multimodal data fusion module 1, an adaptive learning module 2 and a deep learning model module 3.

[0058] Multimodal data fusion module 1: responsible for receiving and fusing data of different modalities.

[0059] Adaptive learning module 2: Automatically adjust the parameters and structure of the deep learning model according to the characteristics of the task and environment.

[0060] Deep learning model module 3: perform feature extraction, process fused data, and output prediction results.

[0061] Specifically, the framework includes:

[0062] The multimodal data fusion module 1 includes a data acquisition submodule 11 , a preprocessing submodule 12 , a fusion submodule 13 , and an adversarial generation submodule 14 .

[0063] The data acquisition submodule 11 is used to acquire data of different modes, including video data, text data, and audio data. Video data: video clips that can contain various scenes and events, such as sports, traffic accidents, daily activities, etc. Audio data: audio streams synchronized with video data, including background noise, human voice, music, etc. Text data: text information such as video title, description, tags or subtitles.

[0064] The preprocessing submodule 12 is used to preprocess the data of the different modes:

[0065] Video: Convert video files of various formats into a standard format suitable for learning system processing, such as the common MP4 format, to ensure compatibility for subsequent processing; use video processing libraries (such as OpenCV) to decompose the video into frame sequences and perform necessary preprocessing on each frame, such as scaling and normalization.

[0066] Audio: Use audio noise reduction algorithms to remove background noise from audio and improve audio quality. For example, spectral subtraction and other techniques are used to estimate the spectral characteristics of the noise and subtract it from the original audio spectrum to obtain the denoised audio.

[0067] Speech segmentation, which divides a continuous audio stream into independent speech segments, for example, separating the speech parts of different speakers in a conversation for subsequent analysis and processing. Segmentation can be performed using energy-based or silence detection methods; audio feature extraction, which extracts the characteristic parameters of the audio. These features can reflect important information such as the timbre and pitch of the audio, providing a basis for subsequent learning and analysis.

[0068] Text: Text cleaning, removing irrelevant characters, punctuation errors, special symbols and other noise information in the text. For example, removing extra spaces and line breaks in the text; word segmentation, dividing continuous text into individual words, which is the basic step of text analysis. You can use Chinese word segmentation tools, etc., to perform word segmentation based on the grammar and vocabulary rules of the language; text vectorization, using pre-trained word embedding models (such as Word2Vec, GloVe) to convert the segmented text into a digital vector representation so that the computer can process it.

[0069] The fusion submodule 13 supports multiple fusion strategies, including early fusion, late fusion and intermediate fusion, so as to fuse the features extracted by the deep learning model three times according to the multiple fusion strategies to obtain the final fusion features. Therefore, it includes an early fusion unit 131, a mid-term fusion unit 132, and a late fusion unit 133.

[0070] Early fusion unit 131: Fusion is performed in the early stage of information processing. The specific stage is to directly combine information from different data sources or different modalities in the early stage of raw data or feature extraction. For example, in a multi-sensor system, when multiple sensors have just collected data, the data is immediately integrated and processed. Its key feature is that it retains the most original detailed information of the data, but it also contains a large amount of noise and redundant information, which requires subsequent processing for screening and optimization. This embodiment is described using the early stage of feature extraction, and the early fusion unit 131 fuses the extracted early features of different modalities to obtain early fusion features.

[0071] Mid-term fusion unit 132: A stage between early fusion and late fusion. The stages between early and late stages can be called mid-term stages, and mid-term fusion can be performed. The mid-term fusion features are obtained by fusion based on the early fusion features and the extracted mid-term features of different modalities. Fusion is performed in the middle stage of the feature extraction process. At this time, the information of each data source or modality has undergone a certain degree of preliminary processing, but the final feature representation or decision result has not yet been fully formed. This not only retains a certain degree of original information association, but also has a certain degree of abstraction and generalization ability, and can balance the advantages and disadvantages of early fusion and late fusion to a certain extent. The feature level selection and design of the intermediate fusion are relatively flexible, and need to be reasonably adjusted and optimized according to the specific application scenarios and task requirements to achieve the best fusion effect.

[0072] Late fusion unit 133: In the late stage of information processing, that is, after each data source or modality has been independently processed in depth and obtained its own highest-level feature representation or decision result, fusion is performed. For example, in the multimodal recognition task of image and text, the image and text are firstly independently feature extracted and classified, and finally the classification results of the two are comprehensively judged. Its advantage is that after each data source or modality is independently processed, the noise and redundant information are filtered to a certain extent, and the fusion is more representative and discriminative information. However, some related information in the early stage may be lost, and the performance requirements for each independent processing module are high. If a large deviation occurs in one of the modules, it may affect the final fusion effect. This embodiment adopts the late stage of feature extraction, and the late fusion unit 133 is based on the mid-term fusion feature and the extracted late features of different modalities to obtain the final fusion feature.

[0073] The adversarial generation submodule 14 generates adversarial samples and adds the adversarial samples to the fusion submodule 13 for fusion. Specifically, the adversarial generation network GAN is used for generation, which is a deep learning model architecture. GAN consists of two main parts, the generator and the discriminator. The task of the generator is to generate realistic data in an attempt to deceive the discriminator. It receives random noise or latent variables as input and tries to generate data samples that look similar to real data. Adversarial samples can be used as a means of data enhancement in the fusion step or to evaluate the robustness of the fusion model. The adversarial generation submodule 14 includes an early adversarial feature generation unit 141, a mid-term adversarial feature generation unit 142, and a late adversarial feature generation unit 143. The early adversarial feature generation unit 141 generates early feature adversarial samples and adds them to the early features of the corresponding modality to perform early feature fusion together. For example, in image and audio fusion applications, adversarial samples of images and audio are generated and merged and processed early to increase data diversity and complexity and improve model generalization capabilities; the mid-term adversarial feature generation unit 142 generates mid-term feature adversarial samples and adds them to the mid-term features of the corresponding modality to perform mid-term feature fusion together. The intermediate fusion stage can be injected into the partially fused model to observe the processing power and performance changes; the late adversarial feature generation unit 143 generates late feature adversarial samples and adds them to the late features of the corresponding modality to perform late feature fusion together. The stability and anti-attack capability of the fusion system can be evaluated during late fusion. The purpose of adversarial generation is to enhance the model's ability to resist various potential malicious attacks or abnormal data by generating adversarial samples and allowing the fusion model to learn how to deal with these challenging data. For example, in the multi-data source risk assessment fusion system in the financial field, generating adversarial samples can simulate the characteristics of fraudulent data, prompting the model to learn to identify and resist such false data during training, thereby improving the accuracy and reliability of risk assessment.

[0074] The deep learning model module 3 is the core part of the framework, which is responsible for extracting features from data of different modalities using different deep learning models (for example, for video frames, convolutional neural network CNN is used to extract spatial features, for audio data, recurrent neural network RNN ​​or convolutional neural network CNN is used to extract temporal features, for text data, Transformer or recurrent neural network RNN ​​is used to extract text features), and combines the above-mentioned fusion submodule 13 to perform feature fusion, and makes predictions based on the fused features to output the prediction results.

[0075] In addition, the deep learning model module 3 is used in various stages of the deep learning model, including training, testing, and verification.

[0076] Adaptive learning module 2 actively introduces learning mechanisms (meta-learning, reinforcement learning, and data enhancement, etc.) to enable the model to gradually adapt to the ever-changing new environment and new tasks, thereby significantly improving its generalization performance. In addition, this module also has powerful functions, and can dynamically adjust hyperparameters such as learning rate and optimization algorithm according to the performance feedback of the model, thereby achieving a more efficient learning process. For example, when facing complex and changing task scenarios, through the meta-learning mechanism, the model can quickly learn the commonalities and characteristics between different tasks, so as to better adjust its own parameters to adapt to new tasks. Reinforcement learning allows the model to continuously optimize its own strategy in the interaction with the environment and improve its performance under specific tasks. Data enhancement can provide the model with more diverse training data and further enhance the generalization ability of the model. At the same time, by dynamically adjusting hyperparameters such as learning rate, the model can avoid falling into the local optimal solution, improve learning efficiency, and enable the model to converge to a better performance state faster. In short, this module plays a vital role in the optimization and adaptation process of deep learning models.

[0077] The adaptive learning module 2 can also guide the training phase of the model, during which the model is trained through the back-propagation algorithm and gradient descent optimizer (such as Adam, RMSprop, etc.). During the training process, hyperparameters such as learning rate and batch size are dynamically adjusted according to the performance of the model (such as accuracy, loss function value, etc.). Regularization techniques (such as Dropout, L1 / L2 regularization) are introduced to prevent overfitting.

[0078] Adaptive strategy: Adaptively adjust the weights and fusion strategies of different modalities based on the performance of the validation set or user feedback from online learning. For example, if the data of a certain modality performs poorly in a specific task, its weight can be reduced or the fusion method can be changed.

[0079] In addition, evaluation indicators and comparative experiments are used to evaluate and verify:

[0080] Evaluation metrics: Use metrics such as accuracy, recall, and F1 score to evaluate the performance of the model on video understanding and event detection tasks.

[0081] Comparative experiment: Compare with the model that only uses single modal data to demonstrate the advantages of multimodal data fusion and adaptive learning.

[0082] The framework of this embodiment can be applied in multiple scenarios, including:

[0083] Intelligent interaction system: combines visual and auditory information to achieve more natural human-computer interaction.

[0084] Autonomous driving: Improving the vehicle’s environmental perception capabilities by integrating camera, radar, and lidar data.

[0085] Medical diagnosis: Combine medical images with medical records to assist doctors in diagnosing diseases.

[0086] Sentiment analysis: Combine the emotional color of text and speech to more fully understand the user's emotions and intentions.

[0087] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0088] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0089] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0092] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0093] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A deep learning framework based on multimodal data fusion and adaptive learning, characterized in that: It includes a multimodal data fusion module (1), an adaptive learning module (2) and a deep learning model module (3). The multimodal data fusion module (1) is used to obtain data of different modalities and fuse them at different stages of feature extraction using a variety of fusion strategies to obtain fusion features; The deep learning model module (3) is used to extract features from data of different modalities according to different deep models, and to make predictions according to the fusion features obtained by the multimodal data fusion module (1), and output prediction results; The adaptive learning module (2) is used to dynamically adjust the hyperparameters of the deep model using a learning mechanism.

2. A deep learning framework based on multimodal data fusion and adaptive learning according to claim 1, characterized in that: The multimodal data fusion module (1) comprises: Data acquisition submodule (11): used to acquire data of different modalities, including video data, text data, and audio data; Preprocessing submodule (12): used for preprocessing the data of different modalities; Fusion submodule (13): used to adopt multiple fusion strategies to fuse at different stages of feature extraction to obtain fusion features.

3. A deep learning framework based on multimodal data fusion and adaptive learning according to claim 2, characterized in that: The fusion submodule (13) comprises: An early fusion unit (131): used for fusing the extracted early features of different modalities to obtain early fusion features; A mid-term fusion unit (132): used for fusing the early fusion features with the extracted mid-term features of different modalities to obtain mid-term fusion features; Late fusion unit (133): used to fuse the mid-term fusion features and the extracted late features of different modalities to obtain the final fusion features.

4. A deep learning framework based on multimodal data fusion and adaptive learning according to claim 3, characterized in that: The multimodal data fusion module (1) further comprises an adversarial generation submodule (14), wherein the adversarial generation submodule (14) comprises: An early adversarial feature generation unit (141) is used to generate early feature adversarial samples and add them to the early features of the corresponding modality to perform early feature fusion together; A mid-term adversarial feature generation unit (142): used to generate mid-term feature adversarial samples and add them to the mid-term features of the corresponding modality to perform mid-term feature fusion together; Late adversarial feature generation unit (143): used to generate late feature adversarial samples and add them to the late features of the corresponding modality to perform late feature fusion together.

5. A deep learning framework based on multimodal data fusion and adaptive learning according to claim 2, characterized in that: The steps of video data preprocessing include: The acquired video data is uniformly converted into a standard format and decomposed into frame sequences, and each frame image is uniformly scaled and normalized.

6. A deep learning framework based on multimodal data fusion and adaptive learning according to claim 2, characterized in that: The steps of preprocessing the text data include: The acquired text data is cleaned to remove noise information, wherein the text cleaning includes removing irrelevant characters, punctuation errors, special symbols, extra spaces and line breaks in the text data; The text data after text cleaning is segmented and converted into vectors using a pre-trained word embedding model.

7. A deep learning framework based on multimodal data fusion and adaptive learning according to claim 2, characterized in that: The step of preprocessing the audio data comprises: Using an audio noise reduction algorithm to remove background noise from the acquired audio data to obtain noise-reduced audio data; The denoised audio data is segmented to obtain independent speech segment data.

8. The deep learning framework based on multimodal data fusion and adaptive learning according to claim 1, characterized in that: The deep learning model includes one of CNN, RNN, and Transformer.

9. A deep learning framework based on multimodal data fusion and adaptive learning according to claim 1, characterized in that: The learning mechanism includes one or more of meta-learning, reinforcement learning and data enhancement.

10. A deep learning framework based on multimodal data fusion and adaptive learning according to claim 1, characterized in that: The feature extraction stage of the deep learning model module (3) includes three stages, namely, an early stage, a middle stage and a late stage, which are used to extract early features, middle stage features and late features respectively.

Images