Multi-modal data fusion method and system based on large model agent

Abstract

The invention discloses a multi-modal data fusion method and system based on a large-model intelligent agent, belongs to the technical field of artificial intelligence, big data processing and intelligent agents, and aims to solve the technical problem of how to effectively integrate data of different modalities such as texts, images and voices by using the large-model intelligent agent. According to the technical scheme, the method comprises the steps that data collection and preprocessing are conducted, specifically, various modal data of texts, images and voice are collected through a web crawler, an API interface, a camera and microphone equipment, the collected data are preprocessed, the preprocessed multi-modal data are obtained, and the data quality is ensured; feature extraction and mapping: extracting corresponding modal features from the preprocessed multi-modal data through CNN and Transform models, mapping the different modal features to the same space, and combining the aligned features to form comprehensive feature representation; carrying out multi-modal fusion processing; and performing intelligent decision and feedback.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, big data processing and intelligent agent technology, and specifically to a multimodal data fusion method and system based on a large model intelligent agent. Background Art

[0002] With the rapid development of artificial intelligence technology, large-model intelligent agents are increasingly being used in various fields. Multimodal data fusion technology can integrate data from different modalities (such as text, images, voice, etc.) to enhance the perception and cognitive ability of intelligent agents. However, existing multimodal data fusion methods still have problems such as low efficiency and poor fusion effect when dealing with massive and complex data faced by large-model intelligent agents.

[0003] Therefore, how to use large-model intelligent agents to effectively integrate data from different modalities such as text, images, and voice to improve the perception, cognition, and decision-making capabilities of intelligent agents is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The technical task of the present invention is to provide a multimodal data fusion method and system based on a large model intelligent agent to solve the problem of how to use the large model intelligent agent to effectively integrate data of different modalities such as text, image, voice, etc., and improve the perception, cognition and decision-making ability of the intelligent agent.

[0005] The technical task of the present invention is achieved in the following way: a multimodal data fusion method based on a large model intelligent agent, the method is as follows:

[0006] Data collection and preprocessing: Collect text, image and voice data in multiple modalities through web crawlers, API interfaces, cameras and microphone devices, and preprocess the collected data to obtain preprocessed multimodal data to ensure data quality;

[0007] Feature extraction and mapping: Use CNN and Transformer models to extract the corresponding features of each modality from the preprocessed multimodal data, map the features of different modalities to the same space, and combine the aligned features to form a comprehensive feature representation;

[0008] Multimodal fusion processing: effectively fuse the features of different modalities to generate a comprehensive feature representation;

[0009] Intelligent decision-making and feedback: Generate decision results based on the fused features, and optimize the performance of the entire system through a feedback mechanism.

[0010] As a preferred method, the collection of text, image and voice multi-modal data through web crawlers, API interfaces, cameras and microphone devices is as follows:

[0011] Collect text data: Collect text information from Internet and database sources through web crawlers and API interfaces;

[0012] Collect image data: Use cameras and image sensors to obtain image data, and support JPEG and PNG formats;

[0013] Collect voice data: Record voice signals through microphones, and support WAV and MP3 audio formats;

[0014] Collect video and sensor data and other modal data, and select appropriate collection devices and methods according to specific application scenarios for video and sensor data collection.

[0015] Preferably, preprocess the collected data as follows:

[0016] Data cleaning: Include deleting duplicate data, filling missing data, and removing outliers; among them, deleting duplicate data means removing duplicate records through hash algorithms or unique identifiers; filling missing data means filling missing values using mean, median, or interpolation methods; removing outliers means identifying and processing outliers through statistical methods (Z-score) or machine learning algorithms;

[0017] Data transformation: Include format conversion and dimension adjustment; among them, format conversion means converting data in different formats into a unified format, for example, converting image data into RGB format and text data into UTF-8 encoding; dimension adjustment means performing dimensionality reduction on high-dimensional data, for example, using PCA (Principal Component Analysis) or t-SNE (t-Distributed Stochastic Neighbor Embedding);

[0018] Data standardization: Include normalization and standardization; among them, normalization means scaling data to the [0,1] interval, and the formula is: Standardization means converting data into a distribution with a mean of 0 and a standard deviation of 1, and the formula is where μ is the mean; σ is the standard deviation;

[0019] Data augmentation: Include image augmentation, text augmentation, and voice augmentation; among them, image augmentation means increasing data diversity by means of random cropping, flipping, rotating, translating, warping, color transformation, and adding noise; text augmentation means expanding the text dataset by means of synonym replacement, sentence restructuring, and generating synthetic text; voice augmentation means enriching voice data by adding background noise, changing the speech rate, and adjusting the pitch;

[0020] Data annotation: including automatic annotation and manual annotation; among them, automatic annotation refers to using pre-trained models (image classification models, text sentiment analysis models) to preliminarily annotate data; manual annotation refers to for data with complex or high-precision requirements, combining manual annotation to ensure the accuracy of annotation.

[0021] Preferably, the corresponding features of each modality are extracted from the preprocessed multi-modal data through CNN and Transformer models as follows:

[0022] Image data feature extraction: Common methods include convolutional neural networks (CNNs), such as ResNet, which can automatically learn high-level semantic features in images; when using the ResNet-50 model to extract image features, the last fully connected layer is removed, and the output of the global average pooling layer is directly used as the feature representation of the image;

[0023] Text data feature extraction: Common methods include word embedding Word2Vec and Transformer-based models (BERT); when using the BERT model to extract text features, the text is input into the pre-trained BERT model, and the vector corresponding to the [CLS] token is taken as the feature representation of the text;

[0024] Speech data feature extraction: Common methods include Mel Frequency Cepstral Coefficients (MFCC) and deep learning-based methods, and the long short-term memory network (LSTM) is used to extract the time-frequency features of speech.

[0025] Preferably, the features of different modalities are mapped to the same space as follows:

[0026] Dimension alignment: The feature vectors of different modalities are mapped to the same dimension through linear transformation or non-linear transformation, that is, the features of each modality are transformed through a multi-layer perceptron (MLP) to make the multi-modal dimensions consistent;

[0027] Time alignment: For time series data, that is, video and audio data, synchronization is performed through timestamps;

[0028] Semantic alignment: By learning a shared semantic space, the features of different modalities have similar semantic meanings in the corresponding space. Common methods include Canonical Correlation Analysis (CCA) and its extended version KCCA;

[0029] The aligned features are combined to form a comprehensive feature representation as follows:

[0030] Concatenation: The feature vectors of different modalities are directly concatenated into a long vector;

[0031] Weighted summation: Perform weighted summation on the feature vectors of different modalities, where the weights are fixed or learned;

[0032] Attention mechanism: Use an attention model to perform weighted combination of features of different modalities. Use the self-attention mechanism in Transformer to enable the attention model to automatically learn the importance between features of different modalities.

[0033] Preferably, the multi-modal fusion processing is as follows:

[0034] Early fusion: At the data level, splice the features of different modalities to form a comprehensive feature vector, and then input it into a single model for processing; for example, splice image pixels and text word vectors and then input them into the model;

[0035] Mid-term fusion: Interact and fuse the features of different modalities in the middle layer of the model; common methods include cross-modal attention mechanisms; for example, in a vision-language model, use a cross-modal attention mechanism to model the relationship between different modalities;

[0036] Late fusion: Independently analyze the data of different modalities, and finally fuse them at the decision-making level;

[0037] Hybrid fusion: Combine the advantages of early fusion and mid-term fusion, and connect a cross-modal attention mechanism after early fusion; the specific hybrid fusion method in a multi-modal Transformer model is as follows: Large pre-trained models based on Transformer dominate in multi-modal data fusion tasks. Large pre-trained models based on Transformer effectively capture the relationships within and between multi-modalities through self-attention and cross-attention mechanisms; among them, the self-attention mechanism: The core of Transformer is the self-attention mechanism, which can capture the global dependencies between elements in a sequence; the calculation formula of the self-attention mechanism is: where Q, K, and V are the query, key, and value matrices respectively, and d k is the dimension of the key; multi-head attention: Through the multi-head mechanism, large pre-trained models based on Transformer learn different relationship patterns to enhance the representation ability; modality encoding: Add modality feature encoding for each modality to help the model distinguish different modality information; the fusion methods include concatenation fusion and interactive attention; concatenation fusion directly concatenates the embedding sequences of each modality and then inputs them into Transformer; interactive attention introduces a cross-modal attention mechanism in Transformer to enable the information of different modalities to interact and fuse with each other;

[0038] LMF (Low-rank Multimodal Fusion): Through low-rank matrix factorization technology, LMF transforms the high-complexity tensor outer product and fully connected (FC) operations in TFN (TensorFusion Network) into more efficient individual modal linear transformations and multi-dimensional dot products, thus significantly reducing the number of model parameters;

[0039] MCB (Multimodal Compact Bilinear Pooling): MCB improves CBP to make it applicable to fusing features of different modalities; The calculation method of MCB is as follows: First, the feature vectors of two modalities are respectively passed through the Count Sketch mapping function to obtain the Count Sketch of the features, and then through FFT and inverse FFT to obtain the fused features;

[0040] Multimodal Transformer: Based on the sequence encoding of Transformers, using Tsai's multiple cross-modal attention blocks module, and then encoding the multimodal sequence for classification through self-attention; Cross-modal attention: A way to fuse cross-modal information, providing a potential cross-modal adaptation; For example, cross-modal attention from modality β to modality α; Cross-modal Transformer: Based on the cross-modal attention module, a cross-modal Transformer module can be designed to enable one modality to receive information from another modality.

[0041] More preferably, the intelligent decision-making and feedback are as follows:

[0042] Generate decisions through comprehensive feature representation by large models, deep reinforcement learning, and decision fusion techniques; Among them, the large model refers to using a pre-trained large model (LLM) for task planning and reasoning, and the pre-trained large model is used to handle complex tasks and generate specific action instructions; Deep reinforcement learning refers to combining multimodal perception and reinforcement learning algorithms (DQN) to enable the agent to learn the optimal strategy in a complex environment; Decision fusion refers to fusing the independent decision results of different modalities to improve the overall decision-making performance, and common methods include voting mechanisms, probability averaging, and learning-based fusion;

[0043] Optimize perception, decision-making, and actions through environmental feedback to form a closed loop; Specifically as follows:

[0044] Environmental feedback processing: Collect feedback data on task completion and the accuracy of user instructions from the environment, and use the feedback data as the basis for optimization;

[0045] Model optimization: Adjust model parameters based on the feedback data to optimize the decision-making strategy;

[0046] Closed-loop optimization: Feedback the results of perception, decision-making, and action to form a closed loop for continuous learning and dynamic adaptation;

[0047] Decision generation: Based on the fused features, the large model agent uses the knowledge and model structure inside the large model agent to generate decision results or response content;

[0048] Feedback adjustment: Collect the user feedback and performance data of the task completion degree of the agent in actual applications as feedback information to optimize the fusion strategy and model parameters.

[0049] A multi-modal data fusion system based on a large model agent, which is used to implement the multi-modal data fusion method based on a large model agent as described above; the system includes:

[0050] Data acquisition and preprocessing module, which is used to collect various modal data such as text, images, and voices through web crawlers, API interfaces, cameras, and microphones, and perform preprocessing operations such as cleaning, conversion, standardization, enhancement, and annotation to obtain preprocessed data and ensure data quality;

[0051] Feature extraction and mapping module, which is used to extract modal features from the preprocessed data through CNN and Transformer models, map different modal features to the same space by means of dimension alignment, time alignment, and semantic alignment, and then fuse the aligned features through concatenation, weighted summation, and attention mechanism;

[0052] Multi-modal fusion processing module, which is used to provide early fusion, mid-term fusion, late fusion, and hybrid fusion strategies, and generate comprehensive feature representations by combining Transformer, LMF, and MCB technologies to enhance the performance and robustness of the model;

[0053] Intelligent decision-making and feedback module, which is used to generate decisions by means of large models, deep reinforcement learning, and decision fusion technologies, and continuously improve the system performance by collecting environmental feedback, adjusting model parameters, and forming closed-loop optimization, ensuring that the agent shows good robustness in actual applications, effectively resisting adversarial attacks, and providing safe and reliable intelligent decision-making services.

[0054] An electronic device, including: a memory and at least one processor;

[0055] Wherein, a computer program is stored on the memory;

[0056] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the multi-modal data fusion method based on a large model agent as described above.

[0057] A computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the multimodal data fusion method based on a large model agent as described above.

[0058] The multimodal data fusion method and system based on a large model agent of the present invention have the following advantages:

[0059] (1) The present invention can significantly improve the perception, cognition, and decision-making abilities of a large model agent, promote the development of artificial intelligence technology towards a more intelligent and reliable direction, and improve the agent's processing and decision-making abilities for multi-source heterogeneous data;

[0060] (2) The present invention can effectively integrate data from different modalities (such as text, image, voice, etc.), improve the perception, cognition, and decision-making abilities of the agent; and through steps such as data collection and preprocessing, feature extraction and mapping, multimodal fusion processing, and intelligent decision-making and feedback, it can achieve efficient fusion of multimodal data and optimal decision-making of the agent;

[0061] (3) Through the comprehensive data collection and preprocessing module of the present invention, it can collect raw data from multiple modality sources, and perform cleaning, conversion, and standardization to ensure the quality and consistency of the data, laying a good foundation for subsequent processing; at the same time, it can adapt to multiple modality data sources and implement an automated data preprocessing process;

[0062] (4) The feature extraction and mapping module of the present invention uses advanced models and technologies to extract useful information from each modality data, and maps the feature vectors of different modalities to the same semantic space for effective fusion, enabling the model to learn richer feature representations; at the same time, deep learning models are used for feature extraction, and specific mapping mechanisms are adopted to ensure effective alignment of different modality features;

[0063] (5) The multimodal fusion processing module of the present invention adopts a variety of fusion strategies and advanced technologies to integrate different modality data, generate comprehensive feature representations, improve the robustness, performance, and generalization ability of the model, and make it perform better in complex data and tasks; at the same time, it has the ability to adaptively adjust the fusion strategy to adapt to different task scenarios and data distributions;

[0064] (6) The intelligent decision-making and feedback module of the present invention uses a variety of technologies to generate decisions, and optimizes the system performance through a feedback mechanism, collects environmental feedback to adjust model parameters, forms a closed-loop optimization, and continuously improves the decision-making ability of the agent; at the same time, it can generate high-quality decision results based on the fused features, and use the feedback information to achieve closed-loop optimization of the entire fusion processing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention will be further described below with reference to the accompanying drawings.

[0066] Appendix Figure 1 It is a flowchart of a multi-modal data fusion method based on a large model agent. Specific implementation manners

[0067] The multi-modal data fusion method and system based on a large model agent of the present invention will be described in detail below with reference to the accompanying drawings of the specification and specific embodiments.

[0068] Embodiment 1:

[0069] As shown in the appendix Figure 1 This embodiment provides a multi-modal data fusion method based on a large model agent, and the method is as follows:

[0070] S1. Data collection and preprocessing: Collect various modal data such as text, images, and voices through web crawlers, API interfaces, cameras, and microphone devices, and preprocess the collected data to obtain preprocessed multi-modal data to ensure data quality;

[0071] S2. Feature extraction and mapping: Extract corresponding modal features from the preprocessed multi-modal data through CNN and Transformer models, map different modal features to the same space, and combine the aligned features to form a comprehensive feature representation;

[0072] S3. Multi-modal fusion processing: Effectively fuse features of different modalities to generate a comprehensive feature representation; the purpose of multi-modal fusion processing is to improve the performance and robustness of the model by integrating data from different modalities;

[0073] S4. Intelligent decision-making and feedback: Generate decision results based on the fused features, and optimize the performance of the entire system through a feedback mechanism.

[0074] The specific method of collecting various modal data such as text, images, and voices through web crawlers, API interfaces, cameras, and microphone devices in step S1 of this embodiment is as follows:

[0075] S1-101. Collect text data: Collect text information from Internet and database sources through web crawlers and API interfaces;

[0076] S1-102. Collect image data: Use cameras and image sensors to obtain image data, and support JPEG and PNG formats;

[0077] S1-103. Collect voice data: Record voice signals through microphones, and support WAV and MP3 audio formats;

[0078] S1-104. Collect video and other modal data of sensor data, and select appropriate collection devices and methods according to specific application scenarios to collect video and sensor data.

[0079] The preprocessing of the collected data in step S1 of this embodiment is specifically as follows:

[0080] S1-201. Data cleaning: including deleting duplicate data, filling missing data, and removing outliers; among them, deleting duplicate data means removing duplicate records through a hash algorithm or a unique identifier; filling missing data means filling missing values using the mean, median, or interpolation method; removing outliers means identifying and processing outliers through statistical methods (Z-score) or machine learning algorithms;

[0081] S1-202. Data conversion: including format conversion and dimension adjustment; among them, format conversion means converting data in different formats into a unified format, such as converting image data into RGB format and text data into UTF-8 encoding; dimension adjustment means performing dimensionality reduction on high-dimensional data, such as using PCA (Principal Component Analysis) or t-SNE (t-Distributed Stochastic Neighbor Embedding);

[0082] S1-203. Data standardization: including normalization and standardization; among them, normalization means scaling the data to the interval [0,1], and the formula is: Standardization means converting the data into a distribution with a mean of 0 and a standard deviation of 1, and the formula is where μ is the mean; σ is the standard deviation;

[0083] S1-204. Data augmentation: including image augmentation, text augmentation, and speech augmentation; among them, image augmentation means increasing data diversity by means of random cropping, flipping, rotating, translating, warping, color transformation, and adding noise; text augmentation means expanding the text dataset by means of synonym replacement, sentence restructuring, and generating synthetic text; speech augmentation means enriching speech data by adding background noise, changing the speech rate, and adjusting the pitch;

[0084] S1-205. Data annotation: including automatic annotation and manual annotation; among them, automatic annotation means using a pre-trained model (image classification model, text sentiment analysis model) to perform preliminary annotation on the data; manual annotation means for data with complex or high-precision requirements, combining manual annotation to ensure the accuracy of the annotation.

[0085] The extraction of corresponding modal features from the preprocessed multi-modal data in step S2 of this embodiment by the CNN and Transformer models is specifically as follows:

[0086] S2-101. Image data feature extraction: Common methods include convolutional neural networks (CNNs), such as ResNet, which can automatically learn high-level semantic features in images; when using the ResNet-50 model to extract image features, the last fully connected layer is removed, and the output of the global average pooling layer is directly used as the feature representation of the image;

[0087] S2-102. Text data feature extraction: Common methods include word embedding Word2Vec and Transformer-based models (BERT); when using the BERT model to extract text features, the text is input into the pre-trained BERT model, and the vector corresponding to the [CLS] token is taken as the feature representation of the text;

[0088] S2-103. Speech data feature extraction: Common methods include Mel Frequency Cepstral Coefficients (MFCC) and deep learning-based methods. The long short-term memory network (LSTM) is used to extract the time-frequency features of speech.

[0089] In this embodiment, the mapping of different modality features to the same space in step S2 is specifically as follows:

[0090] S2-101. Dimension alignment: The feature vectors of different modalities are mapped to the same dimension through linear transformation or non-linear transformation, that is, the features of each modality are transformed through a multi-layer perceptron (MLP) to make the multi-modal dimensions consistent;

[0091] S2-102. Temporal alignment: For time series data, that is, video and audio data, synchronization is performed through timestamps;

[0092] S2-103. Semantic alignment: By learning a shared semantic space, the features of different modalities have similar semantic meanings in the corresponding space. Common methods include Canonical Correlation Analysis (CCA) and its extended version KCCA.

[0093] In this embodiment, the combined aligned features to form a comprehensive feature representation in step S2 is specifically as follows:

[0094] S2-301. Concatenation: The feature vectors of different modalities are directly concatenated into a long vector;

[0095] S2-302. Weighted summation: The feature vectors of different modalities are weighted and summed, and the weights are fixed or learned;

[0096] S2-303. Attention mechanism: The attention model is used to weight and combine the features of different modalities. The self-attention mechanism in Transformer is used to let the attention model automatically learn the importance between the features of different modalities.

[0097] The multimodal fusion processing in step S3 of this embodiment is specifically as follows:

[0098] S301. Early fusion: At the data level, the features of different modalities are concatenated to form a comprehensive feature vector, which is then input into a single model for processing; for example, the image pixels and text word vectors are concatenated and then input into the model;

[0099] S302. Intermediate fusion: The features of different modalities are interacted and fused in the middle layer of the model; common methods include cross-modal attention mechanisms; for example, in a vision-language model, a cross-modal attention mechanism is used to model the relationships between different modalities;

[0100] S303. Late fusion: The data of different modalities are analyzed independently, and finally fused at the decision-making level;

[0101] S304. Hybrid fusion: Combining the advantages of early fusion and intermediate fusion, a cross-modal attention mechanism is connected after early fusion; the specific fusion method in the multimodal Transformer model for hybrid fusion is as follows: Large pre-trained models based on Transformer dominate in multimodal data fusion tasks. Large pre-trained models based on Transformer effectively capture the relationships within and between modalities through self-attention and cross-attention mechanisms; among them, the self-attention mechanism: The core of Transformer is the self-attention mechanism, which can capture the global dependencies between elements in a sequence; the calculation formula of the self-attention mechanism is: where Q, K, and V are the query, key, and value matrices respectively, and d k is the dimension of the key; multi-head attention: Through the multi-head mechanism, large pre-trained models based on Transformer learn different relationship patterns to enhance the representation ability; modality encoding: Add modality feature encodings for each modality to help the model distinguish different modality information; the fusion methods include concatenation fusion and interactive attention; concatenation fusion directly concatenates the embedding sequences of each modality and then inputs them into Transformer; interactive attention introduces a cross-modal attention mechanism in Transformer to make the information of different modalities interact and fuse with each other;

[0102] S305. LMF (Low-rank Multimodal Fusion): LMF uses low-rank matrix factorization technology to transform the high-complexity tensor outer product and fully connected (FC) operations in TFN (Tensor Fusion Network) into more efficient modality-specific linear transformations and multi-dimensional dot products, thereby significantly reducing the number of model parameters;

[0103] S306, MCB (Multimodal Compact Bilinear Pooling): MCB improves CBP to make it applicable to fusing features of different modalities; The calculation method of MCB is as follows: First, the feature vectors of the two modalities are respectively passed through the CountSketch mapping function to obtain the Count Sketch of the features, and then through FFT and inverse FFT to obtain the fused features;

[0104] S307, Multimodal Transformer: Based on the sequence encoding of Transformers, using Tsai's multiple cross-modal attention blocks module, and then through self-attention to encode the multimodal sequence for classification; Cross-modal attention: A way to fuse cross-modal information, providing a potential cross-modal adaptation; For example, cross-modal attention from modality β to modality α; Cross-modal Transformer: Based on the cross-modal attention module, a cross-modal Transformer module can be designed to enable one modality to receive information from another modality.

[0105] The intelligent decision-making and feedback in step S4 of this embodiment are specifically as follows:

[0106] S401, Generate decisions through large models, deep reinforcement learning, and decision fusion technologies based on the comprehensive feature representation; Among them, the large model refers to using a pre-trained large model (LLM) for task planning and reasoning, and the pre-trained large model is used to process complex tasks and generate specific action instructions; Deep reinforcement learning refers to combining multimodal perception and reinforcement learning algorithms (DQN) to enable the intelligent agent to learn the optimal strategy in a complex environment; Decision fusion refers to fusing the independent decision results of different modalities to improve the overall decision-making performance, and common methods include voting mechanisms, probability averaging, and learning-based fusion;

[0107] S402, Optimize perception, decision-making, and actions through environmental feedback to form a closed loop; Specifically as follows:

[0108] S40201, Environmental feedback processing: Collect feedback data on task completion and the accuracy of user instructions from the environment, and use the feedback data as the basis for optimization;

[0109] S40202, Model optimization: Adjust model parameters based on the feedback data to optimize the decision-making strategy;

[0110] S40203, Closed-loop optimization: Feedback the results of perception, decision-making, and actions to form a closed loop of continuous learning and dynamic adaptation;

[0111] S40204. Decision Generation: The large - model agent generates decision results or response content based on the fused features, utilizing the knowledge and model structure within the large - model agent.

[0112] S40205. Feedback Adjustment: Collect the user feedback and performance data of the task completion degree of the agent in actual applications as feedback information to optimize the fusion strategy and model parameters.

[0113] Embodiment 2:

[0114] This embodiment provides a multi - modal data fusion system based on a large - model agent, which is used to implement the multi - modal data fusion method based on a large - model agent in Embodiment 1. The system includes:

[0115] Data Acquisition and Pre - processing Module: It is used to collect various modal data such as text, images, and voices through web crawlers, API interfaces, cameras, and microphones, and perform pre - processing operations such as cleaning, conversion, standardization, enhancement, and annotation to obtain pre - processed data and ensure data quality.

[0116] Feature Extraction and Mapping Module: It is used to extract modal features from the pre - processed data through CNN and Transformer models, map different modal features to the same space by means of dimension alignment, time alignment, and semantic alignment, and then fuse the aligned features through concatenation, weighted summation, and attention mechanisms.

[0117] Multi - modal Fusion Processing Module: It is used to provide early fusion, mid - term fusion, late fusion, and hybrid fusion strategies, and generate comprehensive feature representations by combining Transformer, LMF, and MCB technologies to enhance the model performance and robustness.

[0118] Intelligent Decision - making and Feedback Module: It is used to generate decisions by means of large - model, deep reinforcement learning, and decision fusion technologies, and continuously improve the system performance by collecting environmental feedback, adjusting model parameters, and forming a closed - loop optimization to ensure that the agent shows good robustness in actual applications, effectively resists adversarial attacks, and provides safe and reliable intelligent decision - making services.

[0119] Embodiment 3:

[0120] This embodiment also provides an electronic device, including: a memory and a processor;

[0121] Among them, the memory stores computer - executable instructions;

[0122] The processor executes the computer - executable instructions stored in the memory, so that the processor executes the multi - modal data fusion method based on a large - model agent in any embodiment of the present invention.

[0123] The processor can be a central processing unit (CPU), or can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0124] The memory can be used to store computer programs and / or modules. By running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory, the processor realizes various functions of the electronic device. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include high-speed random access memory, and can also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash memory card, at least one magnetic disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0125] Embodiment 4:

[0126] This embodiment also provides a computer-readable storage medium, which stores multiple instructions. The instructions are loaded by the processor to enable the processor to execute the multi-modal data fusion method based on the large model agent in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided. Software program codes for realizing the functions of any one of the above embodiments are stored on the storage medium, and the computer (or CPU or MPU) of the system or device reads and executes the program codes stored in the storage medium.

[0127] In this case, the program code read from the storage medium itself can realize the functions of any one of the above embodiments. Therefore, the program code and the storage medium storing the program code constitute a part of the present invention.

[0128] Embodiments of the storage medium for providing program codes include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code can be downloaded from a server computer via a communication network.

[0129] In addition, it should be clear that not only can the partial or all of the actual operations be completed by executing the program code read by a computer, but also by the operating system operating on the computer based on the instructions of the program code, thereby implementing the functions of any one of the above embodiments.

[0130] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer. Subsequently, based on the instructions of the program code, the CPU etc. installed on the expansion board or the expansion unit execute the partial and all actual operations, thereby implementing the functions of any one of the above embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal data fusion method based on large model agents, characterized in that, The method is as follows: Data collection and preprocessing: Collect various modal data such as text, images, and voices through web crawlers, API interfaces, cameras, and microphone devices, and preprocess the collected data to obtain preprocessed multi-modal data to ensure data quality; Feature extraction and mapping: Extract corresponding modal features from the preprocessed multi-modal data through CNN and Transformer models, map different modal features to the same space, and combine the aligned features to form a comprehensive feature representation; Multi-modal fusion processing: Effectively fuse features of different modalities to generate a comprehensive feature representation; Intelligent decision-making and feedback: Generate decision results based on the fused features and optimize the performance of the entire system through a feedback mechanism.

2. The multimodal data fusion method based on the large model agent according to claim 1, wherein Collecting various modal data such as text, images, and voices through web crawlers, API interfaces, cameras, and microphone devices is as follows: Collect text data: Collect text information from Internet and database sources through web crawlers and API interfaces; Collect image data: Use cameras and image sensors to obtain image data, and support JPEG and PNG formats; Collect voice data: Record voice signals through microphones, and support WAV and MP3 audio formats; Collect other modal data such as video and sensor data, and select appropriate collection devices and methods according to specific application scenarios for video and sensor data collection.

3. The multimodal data fusion method based on a large model agent according to claim 1 or 2, wherein Preprocessing the collected data is as follows: Data cleaning: Include deleting duplicate data, filling missing data, and removing outliers; among them, deleting duplicate data means removing duplicate records through a hash algorithm or unique identifier; filling missing data means filling missing values using mean, median, or interpolation methods; removing outliers means identifying and processing outliers through statistical methods or machine learning algorithms; Data conversion: Include format conversion and dimension adjustment; among them, format conversion means converting data in different formats into a unified format; dimension adjustment means performing dimensionality reduction on high-dimensional data; Data standardization: including normalization and standardization; among them, normalization refers to scaling the data to the interval [0, 1], and the formula is: Standardization refers to converting the data into a distribution with a mean of 0 and a standard deviation of 1, and the formula is where μ is the mean; σ is the standard deviation; Data augmentation: Include image augmentation, text augmentation, and voice augmentation; among them, image augmentation means increasing data diversity by means of random cropping, flipping, rotating, translating, distorting, color transformation, and adding noise; text augmentation means expanding the text dataset by means of synonym replacement, sentence restructuring, and generating synthetic text; voice augmentation means enriching voice data by adding background noise, changing the speaking speed, and adjusting the pitch; Data annotation: Include automatic annotation and manual annotation; among them, automatic annotation means using a pre-trained model to perform preliminary annotation on the data; manual annotation means for data with complex or high-precision requirements, combining manual annotation to ensure the accuracy of annotation.

4. The multimodal data fusion method based on a large model agent according to claim 3, wherein Extracting corresponding modal features from the preprocessed multi-modal data through CNN and Transformer models is as follows: Image data feature extraction: When using the ResNet-50 model to extract image features, remove the last fully connected layer and directly use the output of the global average pooling layer as the feature representation of the image; Text data feature extraction: When using the BERT model to extract text features, the text is input into the pre-trained BERT model, and the vector corresponding to the [CLS] token is taken as the feature representation of the text; Speech data feature extraction: Use long short-term memory networks to extract the time-frequency features of speech.

5. The multimodal data fusion method based on the large model agent according to claim 4, wherein, Mapping different modality features to the same space is as follows: Dimension alignment: Map the feature vectors of different modalities to the same dimension through linear transformation or non-linear transformation, that is, transform the features of each modality through a multi-layer perceptron to make the multi-modal dimensions consistent; Time alignment: For time series data, that is, video and audio data, synchronize through timestamps; Semantic alignment: By learning a shared semantic space, make the features of different modalities have similar semantic meanings in the corresponding space; Combine the aligned features to form a comprehensive feature representation as follows: Concatenation: Directly concatenate the feature vectors of different modalities into a long vector; Weighted summation: Perform weighted summation on the feature vectors of different modalities, and the weights are fixed or obtained through learning; Attention mechanism: Use the attention model to perform weighted combination on the features of different modalities, and use the self-attention mechanism in Transformers to let the attention model automatically learn the importance between different modality features.

6. The multimodal data fusion method based on a large model agent according to claim 5, wherein, Multi-modal fusion processing is as follows: Early fusion: At the data level, concatenate the features of different modalities to form a comprehensive feature vector, and then input it into a single model for processing; Mid-term fusion: Interact and fuse the features of different modalities in the middle layer of the model; Late fusion: Independently analyze the data of different modalities, and finally fuse at the decision-making level; Hybrid Fusion: Combining the advantages of early fusion and mid-term fusion, followed by a cross-modal attention mechanism after early fusion; The specific hybrid fusion method in the multi-modal Transformer model is as follows: Large pre-trained models based on Transformer dominate in multi-modal data fusion tasks. Large pre-trained models based on Transformer effectively capture the relationships within and between multi-modalities through self-attention and cross-attention mechanisms; Among them, the self-attention mechanism: The core of Transformer is the self-attention mechanism, which can capture the global dependencies between elements in a sequence; The calculation formula of the self-attention mechanism is: where Q, K, and V are the query, key, and value matrices respectively, and d k is the dimension of the key; Multi-head attention: Through the multi-head mechanism, large pre-trained models based on Transformer learn different relationship patterns to enhance the representation ability; Modal encoding: Add modal feature encoding for each modality to help the model distinguish different modal information; The fusion methods include concatenation fusion and interactive attention; Concatenation fusion directly concatenates the embedding sequences of each modality and then inputs them into Transformer; Interactive attention introduces a cross-modal attention mechanism in Transformer to enable the information of different modalities to interact and fuse with each other; LMF: LMF transforms the high-complexity tensor outer product and fully connected operation in TFN into more efficient modality-specific linear transformations and multi-dimensional dot products through low-rank matrix factorization technology, thus significantly reducing the number of model parameters; MCB: MCB improves CBP to make it suitable for fusing features of different modalities; The calculation method of MCB is: First, the feature vectors of the two modalities are respectively obtained through the Count Sketch mapping function to get the Count Sketch of the features, and then the fused features are obtained through FFT and inverse FFT; Multi-modal Transformer: Based on the sequence encoding of Transformers, use Tsai's multiple cross-modal attention blocks module, and then encode the multi-modal sequence through self-attention for classification.

7. The multimodal data fusion system based on the large model agent according to claim 6, wherein Intelligent decision-making and feedback are as follows: Generate decisions through integrated feature representation using large models, deep reinforcement learning, and decision fusion techniques; among them, the large model refers to using a pre-trained large model for task planning and reasoning, and the pre-trained large model is used to process complex tasks and generate specific action instructions; deep reinforcement learning refers to combining multi-modal perception and reinforcement learning algorithms to enable the agent to learn the optimal strategy in a complex environment; decision fusion refers to fusing the independent decision results of different modalities to improve the overall decision-making performance; Optimize perception, decision-making, and actions through environmental feedback to form a closed loop; specifically as follows: Environmental feedback processing: Collect feedback data on task completion and the accuracy of user instructions from the environment, and use the feedback data as the basis for optimization; Model optimization: Adjust model parameters based on the feedback data to optimize the decision-making strategy; Closed-loop optimization: Feed back the results of perception, decision-making, and actions to form a closed loop of continuous learning and dynamic adaptation; Decision generation, the large model agent generates decision results or response content based on the fused features, using the knowledge and model structure inside the large model agent; Feedback adjustment, collect the performance data of user feedback and task completion of the agent in actual applications as feedback information to optimize the fusion strategy and model parameters.

8. A multi-modal data fusion system based on large model agents, characterized in that, This system is used to implement the multi-modal data fusion method based on the large model agent as described in any one of claims 1 to 7; this system includes: A data acquisition and preprocessing module, used to collect various modal data such as text, images, and voices through web crawlers, API interfaces, cameras, and microphones, and perform preprocessing operations such as cleaning, conversion, standardization, enhancement, and annotation to obtain preprocessed data; A feature extraction and mapping module, used to extract modal features from the preprocessed data through CNN and Transformer models, map different modal features to the same space using dimension alignment, time alignment, and semantic alignment methods, and then fuse the aligned features through concatenation, weighted summation, and attention mechanisms; A multi-modal fusion processing module, used to provide early fusion, mid-term fusion, late fusion, and hybrid fusion strategies, and generate an integrated feature representation by combining Transformer, LMF, and MCB technologies; An intelligent decision-making and feedback module, used to generate decisions by means of large models, deep reinforcement learning, and decision fusion techniques, and continuously improve the system performance by collecting environmental feedback, adjusting model parameters, and forming closed-loop optimization.

9. An electronic device, characterized in that, Including: A memory and at least one processor; Wherein, a computer program is stored on the memory; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the multi-modal data fusion method based on the large model agent as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, and the computer program can be executed by the processor to implement the multi-modal data fusion method based on the large model agent as described in any one of claims 1 to 7.

Images