A method and system for multimodal data fusion based on large models

Abstract

This invention discloses a multimodal data fusion method and system based on a large model, belonging to the field of data fusion technology. The method includes: receiving multimodal data and performing noise layering filtering and spatiotemporal alignment; mapping the data to the latent space of the large model through lightweight encoders for each modality, extracting initial features using a single-modality pre-trained model, and generating latent space vectors through adapter transformation; receiving the latent space vectors of each modality and task prompts, and dynamically aggregating them to generate a fusion feature matrix through the self-attention and cross-attention mechanisms of the large model to achieve semantic integration of multimodal information; using the fusion feature matrix as a soft label, learning cross-modal semantic mapping capabilities through knowledge distillation; after training, the model is dynamically quantized and pruned for optimization, deployed to the edge, and the optimized feature matrix is ​​input into a task-customized lightweight head network, combining multimodal context information to output the final task result. This invention can bridge the semantic gap between modalities and improve fusion efficiency.

Description

Technical Field

[0001] This invention relates to the field of data fusion technology, specifically a multimodal data fusion method and system based on a large model. Background Technology

[0002] The existing technology has the following drawbacks:

[0003] Modal heterogeneity: Traditional multimodal fusion methods (such as feature concatenation and weighted averaging) struggle to capture cross-modal semantic relationships, resulting in insufficient information utilization.

[0004] Poor model generalization: Existing algorithms (such as CNN+Attention) rely on specific task design and are not adaptable to unknown modal combinations or complex scenarios.

[0005] High computational cost: When large models directly process multimodal data, the number of parameters is huge, making it difficult to meet real-time requirements (such as deployment on edge devices). Summary of the Invention

[0006] This invention addresses the needs and shortcomings of current technological development by providing a multimodal data fusion method and system based on a large model. By bridging the semantic gap between modalities, it improves fusion efficiency and generalization ability, while also supporting lightweight deployment.

[0007] Firstly, the present invention provides a multimodal data fusion method based on a large model, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0008] A multimodal data fusion method based on a large model includes the following steps:

[0009] S1. Receive multimodal data and perform noise layering filtering and time-space alignment;

[0010] S2. The data output from step S1 is mapped to the latent space of the large model through the lightweight encoders of each modality. After extracting the initial features using the single-modality pre-trained model, the latent space vector is generated by the adapter transformation and used for downstream cross-modal retrieval and multimodal fusion.

[0011] S3. Receive latent space vectors and task prompts from each modality, extract semantic associations within a single modality through the self-attention mechanism of the large model, calculate cross-modal semantic weights using the cross-attention mechanism, and finally dynamically aggregate to generate a fusion feature matrix to achieve semantic integration of multimodal information.

[0012] S4. Receive the fused feature matrix output by the large model as a soft label, and learn cross-modal semantic mapping capability through knowledge distillation; after training, the model is optimized by dynamic quantization and pruning, deployed to the edge or front-end environment, and the optimized feature matrix is ​​input into the task-customized lightweight head network, which combines multimodal context information to output the final task result.

[0013] Optionally, the large model involved refers to a high-parameter neural network pre-trained on large-scale cross-modal data, capable of mapping multimodal information to a unified latent space, and supporting cross-modal semantic association and task reasoning through attention mechanisms. Its core features include:

[0014] The pre-training phase learns cross-modal semantic alignment;

[0015] It provides a standardized latent space dimension as a unified representation basis for multimodal features;

[0016] It supports dynamically adjusting cross-modal attention weights based on task prompts to generate a fused feature matrix.

[0017] Optionally, step S1 specifically includes:

[0018] S1.1. Access multimodal data, including video data, text data, and image data, to provide raw materials for subsequent processing;

[0019] S1.2. A hierarchical noise filtering framework is adopted. First, intramodal primary filtering is performed, that is, according to the characteristics of different modal data, the corresponding basic filtering algorithm is used to initially screen out candidate clean data. Then, the candidate clean data is input into the adaptive noise judgment algorithm. Through intelligent analysis and judgment, noise data is further filtered to improve data purity.

[0020] S1.3. Based on the timestamp information of the data itself, match the multimodal data to ensure that the data of different modalities correspond in the time dimension, so that the changes of each modal data in the time series can be presented and analyzed synchronously.

[0021] S1.4. By using multimodal coordinate mapping technology, the coordinate information of different modal data in the spatial dimension is transformed and matched to eliminate spatial differences caused by data acquisition equipment and perspective factors, and to achieve accurate spatial alignment of multimodal data.

[0022] Preferably, in step S1.2, a hierarchical noise filtering framework is used to perform primary intramodal filtering:

[0023] For video data, use any one of the following filtering algorithms: inter-frame filtering, median / Gaussian filtering, and outlier detection to initially screen out clean candidate data.

[0024] For text data, any one of the following filtering algorithms is used: regular expression filtering, stop word filtering, text similarity detection, and language model error correction, to initially screen out clean candidate data.

[0025] For image data, one of the following filtering algorithms is used: spatial filtering, deblurring algorithm, and anomaly detection, to initially screen out clean candidate data.

[0026] Optionally, step S2 specifically includes:

[0027] S2.1 The lightweight encoders of each modality perform preliminary dimensionality reduction and feature compression on the data output from step S1, and map it to the basic dimension of the latent space that can be processed by the large model.

[0028] S2.2. Employ a text encoder using a single-modal pre-trained model BERT or a cross-modal pre-trained model CLIP to perform semantic encoding on the mapped text data and extract feature vectors h containing contextual relationships. text By pre-training a single-modal visual model ResNet or ViT, convolutional operations and attention mechanisms are performed on image data to generate feature vectors h containing spatial structure information. image ;

[0029] S2.3. For the initial feature vectors of text and images, a linear transformation is performed using learnable adapter parameters to generate a text latent vector Z adapted to cross-modal space. text and the latent vector Z of the image aligned with the text space image :

[0030] Z text =W text *h text +b text ,

[0031] Z image =W image *h image +b image ,

[0032] In the formula, W text b text W image b image These are the learnable parameters of the adapter.

[0033] Secondly, this invention provides a multimodal data fusion system based on a large model, and the technical solution adopted to solve the above-mentioned technical problems is as follows:

[0034] A multimodal data fusion system based on a large model, comprising:

[0035] The data receiving and preprocessing module is used to receive multimodal data and perform noise layering filtering and spatiotemporal alignment.

[0036] The multimodal feature mapping and transformation module is used to map the preprocessed data to the latent space of a large model through lightweight encoders of each modality. After extracting the initial features using a single-modal pre-trained model, the latent space vector is generated by the adapter transformation and used for downstream cross-modal retrieval and multimodal fusion.

[0037] The vector receiving and dynamic aggregation module is used to receive latent space vectors and task prompts from each modality. It extracts semantic associations within a single modality through the self-attention mechanism of the large model, calculates cross-modal semantic weights using the cross-attention mechanism, and finally dynamically aggregates them to generate a fusion feature matrix, thereby realizing the semantic integration of multimodal information.

[0038] The lightweight model deployment module receives the fused feature matrix output by the large model as a soft label and learns cross-modal semantic mapping capabilities through knowledge distillation. The trained model is dynamically quantized and pruned for optimization, then deployed to the edge or front-end environment. The optimized feature matrix is ​​then input into a task-customized lightweight head network, which combines multimodal context information to output the final task result.

[0039] Optionally, the large model involved refers to a high-parameter neural network pre-trained on large-scale cross-modal data, capable of mapping multimodal information to a unified latent space, and supporting cross-modal semantic association and task reasoning through attention mechanisms. Its core features include:

[0040] The pre-training phase learns cross-modal semantic alignment;

[0041] It provides a standardized latent space dimension as a unified representation basis for multimodal features;

[0042] It supports dynamically adjusting cross-modal attention weights based on task prompts to generate a fused feature matrix.

[0043] Optionally, the data receiving and preprocessing modules involved specifically include:

[0044] The multimodal data receiving unit is used to receive multimodal data, including video data, text data, and image data, to provide raw materials for subsequent processing;

[0045] The hierarchical noise filtering unit is used to perform primary filtering within a modality using a hierarchical noise filtering framework. This involves using a basic filtering algorithm tailored to the characteristics of different modal data to initially select candidate clean data. The candidate clean data is then input into an adaptive noise judgment algorithm, which further filters out noisy data and improves data purity through intelligent analysis and judgment.

[0046] The time alignment unit is used to match multimodal data based on the timestamp information of the data itself, ensuring that the data of different modalities correspond in the time dimension, so that the changes of each modality data in the time series can be presented and analyzed synchronously.

[0047] The spatial alignment unit is used to transform and match the coordinate information of different modal data in the spatial dimension by means of multimodal coordinate mapping technology, eliminate spatial differences caused by data acquisition equipment and perspective factors, and achieve accurate spatial alignment of multimodal data.

[0048] Preferably, the hierarchical noise filtering unit adopts a hierarchical noise filtering framework, first performing intra-modal primary filtering:

[0049] For video data, use any one of the following filtering algorithms: inter-frame filtering, median / Gaussian filtering, and outlier detection to initially screen out clean candidate data.

[0050] For text data, any one of the following filtering algorithms is used: regular expression filtering, stop word filtering, text similarity detection, and language model error correction, to initially screen out clean candidate data.

[0051] For image data, one of the following filtering algorithms is used: spatial filtering, deblurring algorithm, and anomaly detection, to initially screen out clean candidate data.

[0052] Optionally, the multimodal feature mapping and transformation module involved specifically includes:

[0053] The data mapping unit is used to perform preliminary dimensionality reduction and feature compression on the preprocessed data using lightweight encoders of each modality, and then map it to the basic dimension of the latent space that can be processed by large models.

[0054] The semantic encoding unit is used by a text encoder employing either the single-modal pre-trained model BERT or the cross-modal pre-trained model CLIP to perform semantic encoding on the mapped text data, extracting a feature vector h containing contextual associations. text ;

[0055] The image processing unit is used to perform convolution operations and attention mechanisms on image data using a single-modal pre-trained visual model ResNet or ViT to generate a feature vector h containing spatial structure information. image ;

[0056] The linear transformation unit is used to perform a linear transformation on the initial feature vectors of text and images through learnable adapter parameters, generating a text latent vector Z that adapts to the cross-modal space. text and the latent vector Z of the image aligned with the text space image :

[0057] Z text =W text *h text +b text ,

[0058] Z image =W image *h image +b image ,

[0059] In the formula, W text b text W image b image These are the learnable parameters of the adapter.

[0060] The multimodal data fusion method and system based on a large model of the present invention have the following advantages compared with the prior art:

[0061] 1. This invention achieves dynamic calculation of semantic weights for multimodal data such as images, text, and videos through the self-attention and cross-attention mechanisms of large models, solves the modality gap problem, generates a fusion feature matrix containing cross-modal semantic associations, and supports downstream tasks in jointly understanding multimodal information;

[0062] 2. The lightweight header of this invention can be customized for specific tasks without modifying the core multimodal processing flow, supporting rapid business iteration; the standardized process from preprocessing to feature fusion can be applied to scenarios such as disaster emergency response, intelligent diagnosis, and industrial inspection. Attached Figure Description

[0063] Appendix Figure 1 This is a flowchart of the method according to Embodiment 1 of the present invention;

[0064] Appendix Figure 2 This is a module connection block diagram of Embodiment 2 of the present invention. Detailed Implementation

[0065] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0066] Example 1:

[0067] Combined with appendix Figure 1 This embodiment proposes a multimodal data fusion method based on a large model, which includes the following steps:

[0068] S1. Receive multimodal data, perform noise layering filtering and time-space alignment, specifically including:

[0069] S1.1. Access multimodal data, including video data, text data, and image data, to provide raw materials for subsequent processing.

[0070] S1.2. A hierarchical noise filtering framework is adopted. First, primary filtering within the modality is performed. That is, based on the characteristics of different modal data, the corresponding basic filtering algorithm is used to initially screen out candidate clean data. Then, the candidate clean data is input into the adaptive noise judgment algorithm. Through intelligent analysis and judgment, noise data is further filtered to improve data purity.

[0071] In this step, during intramodal primary filtering:

[0072] For video data, use any one of the following filtering algorithms: inter-frame filtering, median / Gaussian filtering, and outlier detection to initially screen out clean candidate data.

[0073] For text data, any one of the following filtering algorithms is used: regular expression filtering, stop word filtering, text similarity detection, and language model error correction, to initially screen out clean candidate data.

[0074] For image data, one of the following filtering algorithms is used: spatial filtering, deblurring algorithm, and anomaly detection, to initially screen out clean candidate data.

[0075] S1.3. Based on the timestamp information of the data itself, match the multimodal data to ensure that the data of different modalities correspond in the time dimension, so that the changes of each modal data in the time series can be presented and analyzed synchronously.

[0076] S1.4. By using multimodal coordinate mapping technology, the coordinate information of different modal data in the spatial dimension is transformed and matched to eliminate spatial differences caused by data acquisition equipment and perspective factors, and to achieve accurate spatial alignment of multimodal data.

[0077] S2. The data output from step S1 is mapped to the latent space of a large model (such as CLIP) using lightweight encoders for each modality. Initial features are extracted using a single-modality pre-trained model, and then transformed by an adapter to generate latent space vectors for downstream cross-modal retrieval and multimodal fusion. Specifically, this includes:

[0078] S2.1 The lightweight encoders of each modality (such as 3D convolutional networks for video, FastText for text, and MobileNet for images) perform preliminary dimensionality reduction and feature compression on the data output from step S1, and map it to the basic dimension of the latent space that can be processed by large models (such as CLIP).

[0079] S2.2. Employ a text encoder using a single-modal pre-trained model BERT or a cross-modal pre-trained model CLIP to perform semantic encoding on the mapped text data and extract feature vectors h containing contextual relationships. text By pre-training a single-modal visual model ResNet or ViT, convolutional operations and attention mechanisms are performed on image data to generate feature vectors h containing spatial structure information. image ;

[0080] S2.3. For the initial feature vectors of text and images, a linear transformation is performed using learnable adapter parameters to generate a text latent vector Z adapted to cross-modal space. text and the latent vector Z of the image aligned with the text space image :

[0081] Z text =W text *h text +b text ,

[0082] Z image =W image *h image +b image ,

[0083] In the formula, W text b text W image b image These are the learnable parameters of the adapter.

[0084] S3. Receive latent space vectors and task prompts from each modality, extract semantic associations within a single modality through the self-attention mechanism of a large model (such as BLIP-2), calculate cross-modal semantic weights using a cross-attention mechanism, and finally dynamically aggregate to generate a fusion feature matrix to achieve semantic integration of multimodal information.

[0085] S4. The TinyBERT lightweight model architecture is adopted. It receives the fused feature matrix output by a large model (such as BLIP-2) as a soft label and learns cross-modal semantic mapping ability through knowledge distillation. The trained model is optimized by dynamic quantization and pruning, deployed to the edge or front-end environment, and the optimized feature matrix is ​​input into the task-customized lightweight head network. Combined with multimodal context information, the final task result is output.

[0086] It should be added that the large-scale models involved refer to high-parameter neural networks pre-trained on large-scale cross-modal data, capable of mapping multimodal information to a unified latent space, and supporting cross-modal semantic association and task reasoning through attention mechanisms. These are not limited to the CLIP and BLIP-2 large-scale models listed above. The core characteristics of large-scale models include:

[0087] The pre-training phase learns cross-modal semantic alignment;

[0088] It provides a standardized latent space dimension as a unified representation basis for multimodal features;

[0089] It supports dynamically adjusting cross-modal attention weights based on task prompts to generate a fused feature matrix.

[0090] Example 2:

[0091] Combined with appendix Figure 2 This embodiment proposes a multimodal data fusion system based on a large model, which includes:

[0092] The data receiving and preprocessing module is used to receive multimodal data and perform noise layering filtering and spatiotemporal alignment.

[0093] The multimodal feature mapping and transformation module is used to map the preprocessed data to the latent space of a large model (such as CLIP) through lightweight encoders of each modality. After extracting the initial features using a single-modal pre-trained model, the latent space vector is generated by the adapter transformation and used for downstream cross-modal retrieval and multimodal fusion.

[0094] The vector receiving and dynamic aggregation module is used to receive latent space vectors and task prompts from each modality. It extracts semantic associations within a single modality through the self-attention mechanism of a large model (such as BLIP-2), and then uses a cross-attention mechanism to calculate cross-modal semantic weights. Finally, it dynamically aggregates and generates a fusion feature matrix to achieve semantic integration of multimodal information.

[0095] The lightweight model deployment module is used to adopt the TinyBERT lightweight model architecture, receive the fused feature matrix output by a large model (such as BLIP-2) as a soft label, and learn cross-modal semantic mapping capabilities through knowledge distillation. The trained model is dynamically quantized and pruned for optimization, deployed to the edge or front-end environment, and the optimized feature matrix is ​​input into the task-customized lightweight head network. Combined with multimodal context information, the final task result is output.

[0096] In this embodiment, the data receiving and preprocessing module specifically includes:

[0097] The multimodal data receiving unit is used to receive multimodal data, including video data, text data, and image data, to provide raw materials for subsequent processing;

[0098] The hierarchical noise filtering unit is used to perform primary filtering within a modality using a hierarchical noise filtering framework. This involves using a basic filtering algorithm tailored to the characteristics of different modal data to initially select candidate clean data. The candidate clean data is then input into an adaptive noise judgment algorithm, which further filters out noisy data and improves data purity through intelligent analysis and judgment.

[0099] The time alignment unit is used to match multimodal data based on the timestamp information of the data itself, ensuring that the data of different modalities correspond in the time dimension, so that the changes of each modality data in the time series can be presented and analyzed synchronously.

[0100] The spatial alignment unit is used to transform and match the coordinate information of different modal data in the spatial dimension by means of multimodal coordinate mapping technology, eliminate spatial differences caused by data acquisition equipment and perspective factors, and achieve accurate spatial alignment of multimodal data.

[0101] It should be added that the hierarchical noise filtering unit adopts a hierarchical noise filtering framework, first performing intra-modal primary filtering:

[0102] For video data, use any one of the following filtering algorithms: inter-frame filtering, median / Gaussian filtering, and outlier detection to initially screen out clean candidate data.

[0103] For text data, any one of the following filtering algorithms is used: regular expression filtering, stop word filtering, text similarity detection, and language model error correction, to initially screen out clean candidate data.

[0104] For image data, one of the following filtering algorithms is used: spatial filtering, deblurring algorithm, and anomaly detection, to initially screen out clean candidate data.

[0105] In this embodiment, the multimodal feature mapping and transformation module specifically includes:

[0106] The data mapping unit is used to perform preliminary dimensionality reduction and feature compression on the preprocessed data using lightweight encoders of various modalities (such as 3D convolutional networks for video, FastText for text, and MobileNet for images), and then map it to the underlying dimension of the latent space that can be processed by large models.

[0107] The semantic encoding unit is used by a text encoder employing either the single-modal pre-trained model BERT or the cross-modal pre-trained model CLIP to perform semantic encoding on the mapped text data, extracting a feature vector h containing contextual associations. text ;

[0108] The image processing unit is used to perform convolution operations and attention mechanisms on image data using a single-modal pre-trained visual model ResNet or ViT to generate a feature vector h containing spatial structure information. image ;

[0109] The linear transformation unit is used to perform a linear transformation on the initial feature vectors of text and images through learnable adapter parameters, generating a text latent vector Z that adapts to the cross-modal space. text and the latent vector Z of the image aligned with the text space image :

[0110] Z text =W text *h text +b text ,

[0111] Z image =W image *h image +b image ,

[0112] In the formula, W text b text W image b image These are the learnable parameters of the adapter.

[0113] It should be added that the large-scale models involved refer to high-parameter neural networks pre-trained on large-scale cross-modal data, capable of mapping multimodal information to a unified latent space, and supporting cross-modal semantic association and task reasoning through attention mechanisms. These are not limited to the CLIP and BLIP-2 large-scale models listed above. The core characteristics of large-scale models include:

[0114] The pre-training phase learns cross-modal semantic alignment;

[0115] It provides a standardized latent space dimension as a unified representation basis for multimodal features;

[0116] It supports dynamically adjusting cross-modal attention weights based on task prompts to generate a fused feature matrix.

[0117] For Embodiment 1 and Embodiment 2, the large model fusion features are used as soft labels to train a lightweight model, which can improve the inference speed by 3-5 times while retaining more than 90% of the cross-modal capability; the model size is reduced by 40%-60% through model compression technology (such as INT8 quantization), making it suitable for deployment on edge devices or browsers.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal data fusion method based on a large model, characterized in that, Includes the following steps: S1. Receive multimodal data, perform noise layering filtering and time-space alignment; specifically including: S1.

1. Access multimodal data, including video data, text data, and image data, to provide raw materials for subsequent processing; S1.

2. A hierarchical noise filtering framework is adopted. First, intramodal primary filtering is performed, that is, according to the characteristics of different modal data, the corresponding basic filtering algorithm is used to initially screen out candidate clean data. Then, the candidate clean data is input into the adaptive noise judgment algorithm. Through intelligent analysis and judgment, noise data is further filtered to improve data purity. S1.

3. Based on the timestamp information of the data itself, match the multimodal data to ensure that the data of different modalities correspond in the time dimension, so that the changes of each modal data in the time series can be presented and analyzed synchronously. S1.

4. Using multimodal coordinate mapping technology, the coordinate information of different modal data in the spatial dimension is transformed and matched to eliminate spatial differences caused by data acquisition equipment and perspective factors, and to achieve accurate spatial alignment of multimodal data. S2. The data output from step S1 is mapped to the latent space of a large model using lightweight encoders for each modality. After extracting initial features using a single-modality pre-trained model, the latent space vector is generated through an adapter transformation. This vector is used for downstream cross-modal retrieval and multimodal fusion. Specifically, this includes: S2.1 The lightweight encoders of each modality perform preliminary dimensionality reduction and feature compression on the data output from step S1, and map it to the basic dimension of the latent space that can be processed by the large model. S2.

2. Employ a text encoder using a single-modal pre-trained model BERT or a cross-modal pre-trained model CLIP to perform semantic encoding on the mapped text data and extract feature vectors h containing contextual relationships. text By pre-training a single-modal visual model ResNet or ViT, convolutional operations and attention mechanisms are performed on image data to generate feature vectors h containing spatial structure information. image ; S2.

3. For the initial feature vectors of text and images, a linear transformation is performed using learnable adapter parameters to generate a text latent vector Z adapted to cross-modal space. text and the latent vector Z of the image aligned with the text space image : Z text =W text *h text +b text , Z image =W image *h image +b image , In the formula, W text b text W image b image These are the learnable parameters of the adapter; S3. Receive latent space vectors and task prompts from each modality, extract semantic associations within a single modality through the self-attention mechanism of the large model, calculate cross-modal semantic weights using the cross-attention mechanism, and finally dynamically aggregate to generate a fusion feature matrix to achieve semantic integration of multimodal information. S4. Receive the fused feature matrix output by the large model as a soft label, and learn cross-modal semantic mapping capability through knowledge distillation; after training, the model is optimized by dynamic quantization and pruning, deployed to the edge or front-end environment, and the optimized feature matrix is ​​input into the task-customized lightweight head network, which combines multimodal context information to output the final task result.

2. The multimodal data fusion method based on a large model according to claim 1, characterized in that, The large model refers to a high-parameter neural network pre-trained on large-scale cross-modal data. It has the ability to map multimodal information to a unified latent space and supports cross-modal semantic association and task reasoning through attention mechanisms. Its core features include: The pre-training phase learns cross-modal semantic alignment; It provides a standardized latent space dimension as a unified representation basis for multimodal features; It supports dynamically adjusting cross-modal attention weights based on task prompts to generate a fused feature matrix.

3. The multimodal data fusion method based on a large model according to claim 1, characterized in that, Execute step S1.2, using a hierarchical noise filtering framework, and first perform intra-modal primary filtering: For video data, use any one of the following filtering algorithms: inter-frame filtering, median / Gaussian filtering, and outlier detection to initially screen out clean candidate data. For text data, any one of the following filtering algorithms is used: regular expression filtering, stop word filtering, text similarity detection, and language model error correction, to initially screen out clean candidate data. For image data, one of the following filtering algorithms—spatial filtering, deblurring algorithm, or anomaly detection—is used to initially screen out clean candidate data.

4. A multimodal data fusion system based on a large model, characterized in that, It includes: The data receiving and preprocessing module is used to receive multimodal data and perform noise layering filtering and spatiotemporal alignment. The multimodal feature mapping and transformation module is used to map the preprocessed data to the latent space of a large model through lightweight encoders of each modality. After extracting the initial features using a single-modal pre-trained model, the latent space vector is generated by the adapter transformation and used for downstream cross-modal retrieval and multimodal fusion. The vector receiving and dynamic aggregation module is used to receive latent space vectors and task prompts from each modality. It extracts semantic associations within a single modality through the self-attention mechanism of the large model, calculates cross-modal semantic weights using the cross-attention mechanism, and finally dynamically aggregates them to generate a fusion feature matrix, thereby realizing the semantic integration of multimodal information. The lightweight model deployment module is used to receive the fused feature matrix output by the large model as a soft label and learn cross-modal semantic mapping capabilities through knowledge distillation. The trained model is dynamically quantized and pruned for optimization, then deployed to edge or front-end environments. The optimized feature matrix is ​​then input into a task-customized lightweight head network, which combines multimodal contextual information to output the final task result. Specifically, the data receiving and preprocessing module includes: The multimodal data receiving unit is used to receive multimodal data, including video data, text data, and image data, to provide raw materials for subsequent processing; The hierarchical noise filtering unit is used to perform primary filtering within a modality using a hierarchical noise filtering framework. This involves using a basic filtering algorithm tailored to the characteristics of different modal data to initially select candidate clean data. The candidate clean data is then input into an adaptive noise judgment algorithm, which further filters out noisy data and improves data purity through intelligent analysis and judgment. The time alignment unit is used to match multimodal data based on the timestamp information of the data itself, ensuring that the data of different modalities correspond in the time dimension, so that the changes of each modality data in the time series can be presented and analyzed synchronously. The spatial alignment unit is used to transform and match the coordinate information of different modal data in the spatial dimension by means of multimodal coordinate mapping technology, eliminate spatial differences caused by data acquisition equipment and perspective factors, and achieve accurate spatial alignment of multimodal data; The multimodal feature mapping and transformation module specifically includes: The data mapping unit is used to perform preliminary dimensionality reduction and feature compression on the preprocessed data using lightweight encoders of each modality, and then map it to the basic dimension of the latent space that can be processed by large models. The semantic encoding unit is used by a text encoder employing either the single-modal pre-trained model BERT or the cross-modal pre-trained model CLIP to perform semantic encoding on the mapped text data and extract feature vectors h containing contextual relationships. text ; The image processing unit is used to perform convolution operations and attention mechanisms on image data using a single-modal pre-trained visual model ResNet or ViT to generate a feature vector h containing spatial structure information. image ; The linear transformation unit is used to perform a linear transformation on the initial feature vectors of text and images through learnable adapter parameters, generating a text latent vector Z that adapts to the cross-modal space. text and the latent vector Z of the image aligned with the text space image : Z text =W text *h text +b text , Z image =W image *h image +b image , In the formula, W text b text W image b image These are the learnable parameters of the adapter.

5. A multimodal data fusion system based on a large model according to claim 4, characterized in that, The large model refers to a high-parameter neural network pre-trained on large-scale cross-modal data. It has the ability to map multimodal information to a unified latent space and supports cross-modal semantic association and task reasoning through attention mechanisms. Its core features include: The pre-training phase learns cross-modal semantic alignment; It provides a standardized latent space dimension as a unified representation basis for multimodal features; It supports dynamically adjusting cross-modal attention weights based on task prompts to generate a fused feature matrix.

6. A multimodal data fusion system based on a large model according to claim 4, characterized in that, The hierarchical noise filtering unit adopts a hierarchical noise filtering framework, first performing intra-modal primary filtering: For video data, use any one of the following filtering algorithms: inter-frame filtering, median / Gaussian filtering, and outlier detection to initially screen out clean candidate data. For text data, any one of the following filtering algorithms is used: regular expression filtering, stop word filtering, text similarity detection, and language model error correction, to initially screen out clean candidate data. For image data, one of the following filtering algorithms—spatial filtering, deblurring algorithm, or anomaly detection—is used to initially screen out clean candidate data.

Images