Multi-modal data fusion model system based on convolutional neural network and training method thereof

Abstract

The invention belongs to the technical field of big data, and discloses a multi-modal data fusion model system based on a convolutional neural network and a training method thereof, and the model comprises an input module which is used for receiving multi-modal data; the single-modal feature extraction module is used for performing feature extraction on each modal data by adopting a convolutional neural network (CNN); the inter-modal feature interaction module is used for realizing inter-modal depth feature interaction through a cross-modal attention mechanism and enhancing inter-modal correlation; the feature fusion module dynamically adjusts and fuses different modal features by using a weighted adaptive fusion layer; and the classification and optimization module is used for performing learning optimization on the fused features by using a full connection layer and a classifier to finish a final task, such as classification or prediction. The invention breaks through the limitation of the traditional method, provides a systematized and scientific multi-modal data fusion scheme, brings new technical progress for the fields of data mining, artificial intelligence and information processing, and has important industrial application value and popularization prospect.

Description

Technical Field

[0001] The present invention belongs to, but is not limited to, the field of big data technology, and particularly relates to a multi-modal data fusion model system based on a convolutional neural network and a training method thereof. Background Art

[0002] Current multi-modal data processing technologies usually rely on methods of single-modal feature extraction and subsequent fusion. These methods usually process the data of each modality (such as images, texts, and sounds) independently. After extracting the features of each modality, simple feature concatenation or weighted fusion is performed in the subsequent stage. However, this processing method has the following main problems:

[0003] The correlation between features of different modalities is not fully utilized: Independent extraction of single-modal features fails to capture the interaction information between modalities, resulting in unsatisfactory fusion effects.

[0004] The fusion strategy lacks flexibility: Traditional methods usually adopt fixed weighting or simple concatenation strategies and cannot adapt to the complex non-linear relationships of multi-modal data.

[0005] The training efficiency and model performance are limited: Due to the independent processing of modality data, the accuracy of feature extraction directly limits the subsequent fusion effect.

[0006] The problem that the existing technology urgently needs to solve is how to design a unified multi-modal data fusion model system that can:

[0007] 1. Efficiently extract the correlation features within and between modalities;

[0008] 2. Dynamically adjust the weights of each modality data to achieve deep fusion;

[0009] 3. Improve the generalization ability and training efficiency of the fusion model. Summary of the Invention

[0010] In view of the problems existing in the prior art, the present invention provides a multi-modal data fusion model system based on a convolutional neural network and a training method thereof.

[0011] The present invention is implemented as follows. A multi-modal data fusion model system based on a convolutional neural network, the model comprising:

[0012] An input module, configured to receive multi-modal data, such as images, texts, and sounds;

[0013] A single-modal feature extraction module, connected to the input module, and respectively extracting features of each modality data by using a convolutional neural network CNN;

[0014] The inter-modal feature interaction module is connected to the single-modal feature extraction module, and realizes in-depth feature interaction between modalities through cross-modal attention mechanism to enhance the correlation between modalities;

[0015] The feature fusion module is connected to the inter-modal feature interaction module, and uses a weighted adaptive fusion layer to dynamically adjust and fuse features of different modalities;

[0016] The classification and optimization module is connected to the feature fusion module, and uses a fully connected layer and a classifier to learn and optimize the fused features to complete the final tasks such as classification or prediction.

[0017] Another object of the present invention is to provide a training method for a multi-modal data fusion model based on a convolutional neural network, and the method includes the following steps:

[0018] S1: Preprocess the multi-modal data;

[0019] S2: Respectively adopt a convolutional neural network to extract features from each modality data;

[0020] S3: Perform interaction calculation on features of different modalities;

[0021] S4: Fuse the interacted features;

[0022] S5: Calculate the fused features.

[0023] Further, the S1 specifically includes:

[0024] Normalize and adjust the size of the image data;

[0025] Perform word segmentation and vectorization processing on the text data;

[0026] Perform short-time Fourier transform on the sound data.

[0027] Further, the S2 specifically includes:

[0028] Adopt a convolutional neural network to perform convolution and pooling operations on the image data;

[0029] Adopt a one-dimensional convolutional neural network to perform convolution operation on the text data;

[0030] Adopt a two-dimensional convolutional neural network to perform convolution operation on the spectrogram of the sound data.

[0031] Further, the S3 specifically includes:

[0032] Adopt a cross-modal attention mechanism to calculate the mutual correlation of features of different modalities;

[0033] Calculate the weights of features of different modalities based on the weighted attention matrix.

[0034] Further, S4 specifically includes:

[0035] Using a weighted adaptive fusion layer to fuse different modal features;

[0036] Using weighted summation, concatenation, or bilinear pooling for fusion calculation.

[0037] Further, the multi-modal data in S1 specifically includes:

[0038] Image data: Using standard image preprocessing techniques, such as normalization and data augmentation, to uniformly adjust the images to a fixed size;

[0039] Text data: Converting the text into word vectors or embedding representations through a tokenizer, such as Word2Vec or BERT;

[0040] Sound data: Generating a spectrogram through short-time Fourier transform (STFT) as the input.

[0041] Further, S2 specifically includes:

[0042] Constructing a CNN network for each type of modal data to extract local features within the modality:

[0043] Image modality: Using standard convolutional layers and pooling layers to extract spatial features;

[0044] Text modality: Extracting sequence features through a one-dimensional convolutional network;

[0045] Sound modality: Extracting the time-frequency features of the spectrogram through a two-dimensional convolutional network;

[0046] Assume the input modality is X i , and its feature extraction process can be expressed as:

[0047] F i = GNN(X i )

[0048] where F i is the feature representation of the i-th modality, and GNN is a convolutional neural network.

[0049] Further, S3 specifically includes:

[0050] Constructing a cross-modal attention mechanism to generate an interactive feature representation between modalities by calculating the correlation weights of features between different modalities, specifically:

[0051]

[0052] where A ijis the attention weight between modalities i and j, d is the feature dimension, and through the bidirectional attention mechanism, the interaction feature F is generated ij :

[0053] F ij＝ A ij .F j

[0054] Through the bidirectional attention module, the deep interaction information between modalities is further mined.

[0055] Furthermore, the S4 specifically includes:

[0056] Dynamically fuse the interaction features between modalities and the unimodal features, and adopt a weighted adaptive fusion strategy;

[0057] Fusion layer formula:

[0058] Among them, F i represents the feature of the i-th modality, w i represents the weight, and the weight is optimized through adaptive adjustment.

[0059] Furthermore, the S5 specifically includes:

[0060] Map the fused feature F 融合 to the classification space through a fully connected layer, and finally the classifier completes the classification or prediction task. The model uses the cross-entropy loss function to optimize the classification result, and the formula is:

[0061]

[0062] Among them, y c and are the probability distributions of the true label and the predicted label respectively, C is the number of categories, the cross-entropy loss function is used for optimization, and the Adam optimizer is used to improve the model convergence speed.

[0063] Furthermore, the S6 specifically includes:

[0064] Adopt a step-by-step training strategy, first train the feature extraction module of each modality separately, then jointly train the inter-modal interaction module and the feature fusion module, and finally perform end-to-end training of the entire model;

[0065] Evaluate the model performance through the validation set and adjust the hyperparameters.

[0066] Another object of the present invention is to provide a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the training method of the multi-modal data fusion model system based on the convolutional neural network.

[0067] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which when executed by a processor causes the processor to execute the steps of the training method of the multi-modal data fusion model system based on a convolutional neural network.

[0068] Combined with the above technical solutions and solved technical problems, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0069] Through the cross-modal attention mechanism and the inter-modal interaction module, the present invention can fully explore the correlation features between multi-modal data. This design breaks through the problem of poor fusion effect caused by independently processing the features of each modality in traditional technologies, ensuring closer and deeper interaction at the data dimension and feature level, and improving the overall performance and stability of multi-modal data processing.

[0070] In order to further improve the fusion effect, the present invention introduces a weighted adaptive fusion strategy. Different from traditional fixed weighting, this strategy can dynamically optimize and allocate the weights of the features of each modality based on a deep learning framework, so as to better adapt to complex and changeable multi-modal data. This flexibility not only improves the robustness of the model, but also significantly enhances the generalization ability in different application scenarios.

[0071] The present invention adopts a mode combining step-by-step training and end-to-end optimization, so that each stage (such as feature extraction, modal interaction and feature fusion) can be fully optimized. By locking the key feature extraction elements through prior step-by-step training and then fine-tuning the model parameters as a whole in end-to-end training, it is ensured that higher levels can be achieved in terms of accuracy, robustness and efficiency, meeting the requirements of various complex applications.

[0072] Different from traditional methods that are mostly applied in single-modal or simple cross-modal scenarios, the methods and systems of the present invention can be applied to various cross-modal tasks, such as video content analysis, combined understanding of speech and images, etc., and have broader industrial implementation value. This general multi-modal data processing solution significantly expands the application field and provides more systematic technical support for the artificial intelligence and information processing industries.

[0073] Through the step-by-step training strategy, the present invention effectively reduces the overall training time and achieves better optimization effects in the feature extraction and fusion stages. At the same time, multiple iterations and deep fine-tuning ensure that the model can still maintain good generalization ability and stability when facing different types of multi-modal data, further improving the practicality of the model in the real environment.

[0074] The present invention aims to break through the limitations of traditional multi-modal fusion methods, and proposes a systematic and scientific fusion solution on the basis of the original processing framework, which has important industrial application value and broad promotion prospects. At the technical level, the present invention can not only fill the technical gap of multi-modal deep interaction at home and abroad, but also bring innovation impetus to the fields of data mining, artificial intelligence and information processing. Its expected benefits are reflected in many application scenarios such as efficient human-computer interaction systems, intelligent security monitoring, multi-modal medical diagnosis and video content analysis, making a powerful contribution to promoting the further development of multi-modal data processing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 FIG. is a structural diagram of a multi-modal data fusion model system based on a convolutional neural network provided by an embodiment of the present invention;

[0076] Figure 2 FIG. is a flowchart of a training method for a multi-modal data fusion model system based on a convolutional neural network provided by an embodiment of the present invention;

[0077] Figure 3 FIG. is a flowchart of a model training and verification method provided by an embodiment of the present invention;

[0078] In the figure: 1. Input module; 2. Single-modal feature extraction module; 3. Inter-modal feature interaction module; 4. Feature fusion module; 5. Classification and optimization module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0079] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0080] As Figure 1 shown, an embodiment of the present invention provides a multi-modal data fusion model system based on a convolutional neural network, and the model includes:

[0081] An input module 1 for receiving multi-modal data, such as images, texts, and sounds;

[0082] A single-modal feature extraction module 2, connected to the input module 1, and using a convolutional neural network CNN to extract features from each modal data respectively;

[0083] An inter-modal feature interaction module 3, connected to the single-modal feature extraction module 2, and realizing deep feature interaction between modalities through a cross-modal attention mechanism to enhance the correlation between modalities;

[0084] A feature fusion module 4, connected to the inter-modal feature interaction module 3, and using a weighted adaptive fusion layer to dynamically adjust and fuse features of different modalities;

[0085] The classification and optimization module 5, which is connected to the feature fusion module 4, uses a fully connected layer and a classifier to learn and optimize the fused features, and complete the final tasks, such as classification or prediction.

[0086] This model first receives multimodal data through the input module, including different types of signals such as images, texts, and sounds. The data of each modality is preprocessed, such as normalization, noise reduction, word segmentation (for texts), etc., to adapt to the model input. Then, it is separately transmitted to the single-modal feature extraction module, which uses a convolutional neural network (CNN) to extract deep features from the data of each modality. For example, image data passes through multiple convolutional layers and pooling layers to extract its spatial structure information; text data passes through an embedding layer and a one-dimensional convolutional layer to extract key semantic features; audio data is input into the CNN to extract time-frequency features after spectral conversion through the short-time Fourier transform (STFT).

[0087] After the single-modal feature extraction is completed, the data enters the inter-modal feature interaction module. This module uses a cross-modal attention mechanism to calculate the mutual correlation of features of different modalities and enhance the complementary information between modalities. For example, in the image-text combination task, this module can establish semantic correspondence between local regions in the image and keywords in the text, enabling the model to better understand the association between different modalities. For audio-text data, the model will identify the mapping relationship between speech features and the text to improve the feature matching degree. In this stage, the importance weights between modalities are mainly calculated through a weighted attention matrix, so as to highlight key features and suppress redundant information.

[0088] Subsequently, the processed cross-modal features enter the feature fusion module, which uses an adaptive weighted fusion layer to dynamically adjust the fusion strategy according to the importance of different modalities. Specifically, the model will calculate the corresponding fusion weights based on the feature distributions of different modalities, and integrate the information through weighted summation, concatenation, or bilinear pooling. For example, in the multimodal sentiment analysis task, this module will assign a higher weight to the text, while in the visual classification task, it will rely more on image features. Through this adaptive adjustment mechanism, the model can flexibly adapt to different task requirements and improve the feature expression ability after fusion.

[0089] Finally, the fused features are transmitted to the classification and optimization module, which includes a fully connected layer (FC) and a classifier (such as Softmax, SVM, or regression model) for further learning and optimizing the features. During the training process, the model adopts gradient descent optimization (such as Adam, SGD) to minimize the loss function, such as cross-entropy loss or mean squared error loss, to optimize the accuracy of the feature representation. Through the end-to-end training process, the model can continuously adjust the convolutional kernel parameters, attention weights, and fusion strategies to improve the task performance. Finally, the module outputs prediction results, such as class labels, sentiment tendencies, or regression values, to achieve tasks such as classification, object recognition, or prediction.

[0090] In the embodiment of the present invention, the input module 1 first receives data from different modalities such as images, texts, and sounds, and performs basic preprocessing operations on them. For image data, it includes steps such as size adjustment, normalization, or data augmentation; for text data, operations such as word segmentation, stop word removal, and word vector mapping can be adopted; for sound data, filtering, framing, and feature extraction (such as Mel-frequency cepstral coefficients) can be performed. The processing methods at this stage can be adjusted according to the specific multi-modal application requirements, aiming to provide a suitable and consistent input format for subsequent feature extraction.

[0091] The preprocessed multi-modal data are respectively input into the single-modal feature extraction module 2. This module uses deep learning methods such as convolutional neural networks (CNNs) to extract features from each modality of data. For example, for image data, CNNs can gradually extract low-level features such as edges and textures through multiple convolutional and pooling operations and combine them into high-level semantic information; for text data, CNNs can use filters to capture local context information of short text segments or phrases; for audio data, it focuses on identifying spectral features, energy distribution, etc. The resulting multi-group single-modal feature vectors lay the foundation for subsequent cross-modal interaction and fusion.

[0092] After the single-modal feature extraction is completed, the inter-modal feature interaction module 3 receives the features of each modality and realizes deep interaction through a cross-modal attention mechanism. Specifically, the system calculates the similarity or dependence degree between different modality features, and accordingly assigns larger weights to the modalities or feature channels with higher importance, so that they receive more attention in subsequent fusion. This cross-modal attention mechanism iterates repeatedly among multiple modalities, which can significantly enhance the modality correlation degree and explore potential complementary information from different angles, avoiding the problem of information isolation generated when each modality is processed independently.

[0093] The weighted feature matrix generated by the cross-modal feature interaction is passed to the feature fusion module 4, which is provided with a weighted adaptive fusion layer. The weighted adaptive fusion layer can dynamically allocate the feature contribution ratios of each modality, and expand or shrink the influence degrees of different modalities through a series of learnable parameters (or weights generated based on a deep network). Different from traditional fixed-weight fusion, the present invention adopts an adaptive method, enabling the model to flexibly allocate resources according to the task requirements and data distribution, so as to adapt to complex and changeable multi-modal scenarios and improve the overall fusion effect.

[0094] After the feature fusion is completed, the fused feature vector is transmitted to the classification and optimization module 5. This module usually includes a fully connected layer and a classifier, further integrating the fused features through the fully connected layer, and completing the prediction or discrimination for a specific task (such as image classification, text sentiment analysis, or speech recognition, etc.) in the classifier. During this process, the model will cooperate with the backpropagation algorithm to update and fine-tune the parameters of the previous parts to maximize the classification accuracy or minimize the prediction error. As the training iterates, the model's adaptive fusion and feature extraction capabilities will continue to improve, and finally converge to a stable and efficient solution.

[0095] In the entire signal data processing flow, there are close feedback and iterative relationships among the various modules. The single-modal feature extraction module 2 and the cross-modal feature interaction module 3 will continuously fine-tune the extraction and interaction strategies according to the training feedback of the classification and optimization module 5; the feature fusion module 4 will also gradually evolve through adaptive parameter learning. This closed-loop iteration not only ensures that the optimal feature representation can be learned at each stage, but also improves the ability to depict complex correlation relationships in cross-modal data, ultimately ensuring better performance and robustness in multi-modal tasks.

[0096] As Figure 2 shown, an embodiment of the present invention provides a training method for a multi-modal data fusion model system based on the convolutional neural network as described in claim 1. The method specifically includes:

[0097] S1: Multi-modal data preprocessing;

[0098] S2: Single-modal feature extraction;

[0099] S3: Cross-modal feature interaction;

[0100] S4: Feature fusion;

[0101] S5: Classification and optimization;

[0102] S6: Model training and verification.

[0103] In the above S1, the multi-modal data specifically includes:

[0104] Image data: Using standard image preprocessing techniques such as normalization and data augmentation, the images are uniformly adjusted to a fixed size;

[0105] Text data: The text is converted into word vectors or embedding representations through a tokenizer, such as Word2Vec or BERT;

[0106] Sound data: A spectrogram is generated through the short-time Fourier transform (STFT) as the input.

[0107] Specifically, S2 includes:

[0108] Construct a CNN network for each type of modal data to extract local features within the modality:

[0109] Image modality: Standard convolutional layers and pooling layers are used to extract spatial features;

[0110] Text modality: One-dimensional convolutional networks are used to extract sequence features;

[0111] Sound modality: Two-dimensional convolutional networks are used to extract the time-frequency features of the spectrogram;

[0112] Assume the input modality is X i , and its feature extraction process can be expressed as:

[0113] F i = GNN(X i )

[0114] where F i is the feature representation of the i-th modality, and GNN is a convolutional neural network.

[0115] Specifically, S3 includes:

[0116] Construct a cross-modal attention mechanism. By calculating the correlation weights of features between different modalities, an interactive feature representation between modalities is generated, specifically:

[0117]

[0118] where A ij is the attention weight between modalities i and j, d is the feature dimension, and through a bidirectional attention mechanism, an interactive feature F ij is generated:

[0119] F ij＝ A ij .F j

[0120] Further explore the deep interactive information between modalities through a bidirectional attention module.

[0121] Specifically, S4 includes:

[0122] Dynamically fuse the cross-modal interaction features and single-modal features, and adopt a weighted adaptive fusion strategy;

[0123] Fusion layer formula:

[0124] where F i represents the features of the i-th modality, and w i represents the weight, and the weight is optimized through adaptive adjustment.

[0125] The specific content of S5 includes:

[0126] Map the fused features F 融合 to the classification space through a fully connected layer, and finally the classifier completes the classification or prediction task. The model uses the cross-entropy loss function to optimize the classification result, and the formula is:

[0127]

[0128] where y c and are the probability distributions of the true label and the predicted label respectively, C is the number of classes, and the cross-entropy loss function is used for optimization. The Adam optimizer is used to improve the model convergence speed.

[0129] As Figure 3 shown, the specific content of S6 includes:

[0130] S61: Adopt a step-by-step training strategy. First, train the feature extraction module of each modality separately, then jointly train the cross-modal interaction module and the feature fusion module, and finally perform end-to-end training of the entire model;

[0131] S62: Evaluate the model performance through the validation set and adjust the hyperparameters.

[0132] An embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the training method of the multi-modal data fusion model system based on the convolutional neural network.

[0133] An embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the training method of the multi-modal data fusion model system based on the convolutional neural network.

[0134] The present invention proposes a multi-modal data fusion model system based on a convolutional neural network and its training method, including steps such as data preprocessing, single-modal feature extraction, inter-modal feature interaction, feature fusion, classification and optimization, and model training and verification. This solution effectively mines the correlation information between different modalities through the combination of deep learning and the attention mechanism, improves the expression ability and fusion accuracy of multi-modal data, and thus provides an efficient and reliable technical solution for the processing of multi-type data such as images, texts, and voices.

[0135] During the training process, the system first performs preprocessing on multi-modal data such as normalization, word segmentation, and noise reduction (S1); then, uses neural networks such as CNN to independently extract features from images, texts, or voices (S2); subsequently, completes inter-modal feature interaction through a cross-modal attention mechanism (S3); performs weighted adaptive fusion on multi-modal features in the feature fusion stage (S4); then, in the classification and optimization link (S5), learns the fusion result through a fully connected layer and a classifier; finally, in the model training and verification (S6) step, comprehensively verifies the accuracy and robustness of the model and continuously iterates to obtain the optimal result.

[0136] The multi-modal data fusion method of the present invention is applicable to a variety of industries and scenarios, and has broad prospects especially in the fields of intelligent security, industrial inspection, medical diagnosis, and public opinion analysis. For example, in intelligent security, the system can simultaneously process surveillance video and audio information to achieve more accurate anomaly detection; in medical diagnosis, fusing images (such as CT, MRI, etc.) and medical record text data can provide reference assistance for doctors from multiple dimensions; in the field of industrial inspection, combining visible light images, infrared images, and sensor measurement data can identify potential faults or hidden dangers from multiple angles.

[0137] To facilitate the implementation of this method in actual applications, the present invention provides a computer device, which includes a memory and a processor. The memory pre-stores a computer program including the training method, and the processor can execute each step of the above multi-modal fusion algorithm when running this program. By deploying key algorithm modules on a high-performance processor or a GPU acceleration card, it is possible to efficiently complete operations such as input of large-scale multi-modal data, feature extraction, and model training.

[0138] In addition to the computer device itself, the present invention also provides a computer-readable storage medium for storing the above multi-modal fusion training program. When the processor reads and executes the program in this medium, it can also complete the entire algorithm process from data preprocessing to result prediction. This storage medium can be in various forms such as hard disks, solid-state drives, USB flash drives, or optical discs, which is convenient for rapid deployment and cross-platform migration in various hardware environments to meet the business needs of different users.

[0139] By combining the multi-modal data fusion method with general computer hardware devices, collaborative training and inference can be carried out in a distributed environment or the cloud, enhancing the scalability of the system. Whether it is large-scale real-time analysis of surveillance videos or batch offline processing of images and texts, the processor and storage resources can be flexibly scheduled to accelerate the training process. In addition, in the scenarios of transfer learning and federated learning, the method of the present invention can also make full use of the existing model weights and data collected by different edge nodes to provide richer knowledge support for multi-modal fusion.

[0140] The model obtained by the present invention shows higher accuracy and robustness in multi-modal fusion, significantly exceeding traditional single-modal processing or simple modal splicing methods. At the same time, the cross-modal attention mechanism ensures full interaction between multi-source information such as images, texts, and voices, avoiding the problem of information fragmentation. The weighted adaptive fusion strategy can dynamically assign weights according to the importance of different modal data, not only improving the classification and prediction performance, but also effectively adapting to the changing scene requirements and enhancing the generalizability and practical value of the model.

[0141] In multiple experiments and actual tests, the method of the present invention has achieved obvious performance improvements in tasks such as image-text combination recognition and audio-video synchronization detection. Compared with the single-modal processing method, the convolutional neural network fusion strategy proposed by the present invention has stable and significant improvements in the accuracy, recall rate, and AUC metrics of multi-modal tasks, proving its feasibility and superiority in the actual production environment. In addition, benefiting from the flexible training framework and the support of the readable storage medium, the present invention has broad industrial application prospects, which can not only help enterprises efficiently process massive data, but also provide strong technical support for the intelligent upgrading of various industries.

[0142] It should be noted that the implementation mode of the present invention can be realized by hardware, software, or a combination of software and hardware. The hardware part can be realized by using special logic; the software part can be stored in the memory and executed by an appropriate instruction execution system, such as a microprocessor or special designed hardware. Those of ordinary skill in the art can understand that the above-mentioned devices and methods can be realized by using computer executable instructions and / or included in the processor control code. For example, such code is provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be realized by the hardware circuit of programmable hardware devices such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips and transistors, or programmable logic devices such as field programmable gate arrays, and can also be realized by software executed by various types of processors, or can be realized by a combination of the above hardware circuits and software such as firmware.

[0143] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be covered by the protection scope of the present invention.

Claims

1. A multimodal data fusion model system based on convolutional neural network, characterized in that: The model includes: An input module is used to receive multimodal data, including images, texts, and sounds, and perform normalization, noise reduction, or word segmentation on the input data; A single-modal feature extraction module is connected to the input module and uses a convolutional neural network (CNN) to extract features from each modality data, wherein the image data is subjected to a plurality of convolutional layers and a pooling layer to extract spatial structural features, the text data is subjected to an embedding layer and a one-dimensional convolutional layer to extract semantic features, and the audio data is subjected to a spectrum conversion through a short-time Fourier transform (STFT) and then input into the CNN to extract time-frequency features; An inter-modal feature interaction module is connected to the unimodal feature extraction module, and calculates the correlation between different modal features based on the cross-modal attention mechanism to enhance the complementary information between modalities. The module calculates the importance weights between modalities through a weighted attention matrix to highlight key features and suppress redundant information. A feature fusion module is connected to the inter-modal feature interaction module, and uses a weighted adaptive fusion layer to dynamically weight and fuse features of different modalities, wherein the module calculates corresponding fusion weights according to feature distributions of different modalities, and implements fusion by weighted summation, concatenation or bilinear pooling to improve feature expression capabilities; The classification and optimization module is connected to the feature fusion module, and includes a fully connected layer (FC) and a classifier, and is used to learn and optimize the fused features. The classification and optimization module minimizes the loss function based on the gradient descent optimization algorithm to optimize the accuracy of the feature representation, and finally outputs the prediction results, including category labels, sentiment tendencies or regression values, to complete the classification, target recognition or prediction tasks.

2. A training method for a multimodal data fusion model system based on a convolutional neural network as claimed in claim 1, characterized in that: The method specifically includes: S1: Multimodal data preprocessing; S2: unimodal feature extraction; S3: Inter-modality feature interactions; S4: feature fusion; S5: Classification and optimization; S6: Model training and validation.

3. The training method of the multimodal data fusion model system based on convolutional neural network as claimed in claim 2, characterized in that: The multimodal data in S1 specifically includes: Image data: Use standard image preprocessing techniques, such as normalization and data augmentation, to uniformly resize images to a fixed size; Text data: Convert text into word vectors or embedding representations through a tokenizer, such as Word2Vec or BERT; Sound data: The spectrum is generated by short-time Fourier transform (STFT) as input.

4. The training method of the multimodal data fusion model system based on convolutional neural network as claimed in claim 2, characterized in that: The S2 specifically includes: Construct a CNN network for each modality of data to extract local features within the modality: Image modality: Standard convolutional layers and pooling layers are used to extract spatial features; Text modality: extract sequence features through one-dimensional convolutional network; Sound modality: Extract the time-frequency features of the spectrogram through a two-dimensional convolutional network; Assume the input modality X i , its feature extraction process can be expressed as: F i =GNN(X i ) Among them, F i is the feature representation of the i-th modality, and GNN is a convolutional neural network.

5. The training method of the multimodal data fusion model system based on convolutional neural network as claimed in claim 2, characterized in that: The S3 specifically includes: Construct a cross-modal attention mechanism to generate interactive feature representations between modalities by calculating the correlation weights between features of different modalities. Specifically: Among them, A ij is the attention weight between modalities i and j, d is the feature dimension, and the interactive feature F is generated through the bidirectional attention mechanism. ij : F ij＝ A ij .F j Through the bidirectional attention module, the deep interaction information between modalities is further explored.

6. The training method of the multimodal data fusion model system based on convolutional neural network as claimed in claim 2, characterized in that: The S4 specifically includes: Dynamically fuse the inter-modal interaction features and single-modal features, using a weighted adaptive fusion strategy; Fusion layer formula: Among them, F i represents the characteristics of the i-th mode, w i Represents weights, which are optimized through adaptive adjustment.

7. The training method of the multimodal data fusion model system based on convolutional neural network as claimed in claim 2, characterized in that: The S5 specifically includes: For the fusion feature F 融合 Through the full connection layer mapping to the classification space, the classifier finally completes the classification or prediction task. The model uses the cross entropy loss function to optimize the classification results. The formula is: Among them, y c and are the probability distributions of the true label and the predicted label respectively, C is the number of categories, the cross entropy loss function is used for optimization, and the Adam optimizer is used to improve the model convergence speed.

8. The training method of the multimodal data fusion model system based on convolutional neural network as claimed in claim 2, characterized in that: The S6 specifically includes: S61: A step-by-step training strategy is adopted. The feature extraction module of each modality is trained separately first, and then the inter-modality interaction module and the feature fusion module are trained jointly. Finally, the end-to-end training of the whole model is performed. S62: Evaluate model performance and tune hyperparameters using the validation set.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the training method of the multimodal data fusion model system based on the convolutional neural network as described in any one of claims 2 to 8.

10. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the training method of a multimodal data fusion model system based on a convolutional neural network as described in any one of claims 2 to 8.

Images