Multi-modal data feature fusion processing system and method thereof

Abstract

The invention relates to the technical field of processing systems, and discloses a multi-modal data feature fusion processing system and method, and the system comprises an intelligent acquisition subsystem, a self-adaptive extraction subsystem, a dynamic fusion training subsystem, an intelligent generation subsystem, and a cross-modal AI data center. The intelligent acquisition subsystem is integrated with a multi-modal data perception engine, realizes dynamic data acquisition and preprocessing, and comprises an AI intelligent acquisition module, a first AI big data module and an intelligent communication interaction module. According to the technical scheme, the multi-modal data processing efficiency and quality are comprehensively improved, and the lightweight preprocessing model group performs exclusive optimization for different modals, so that the quality of the data input to the extraction subsystem is improved by 40%, and the feature extraction accuracy is averagely improved by 22%; the intelligent communication interaction module supports cross-subsystem real-time instruction interaction and load balancing, the data transmission efficiency is improved by 60%, and the high-concurrency scene processing delay is reduced to be within 150ms.

Description

Technical Field

[0001] The present invention belongs to the technical field of processing systems, and in particular, relates to a multimodal data feature fusion processing system and method thereof. Background Art

[0002] Modality refers to the way something happens or exists, while multimodality refers to the combination of two or more modalities in various forms. To further explain modality and multimodality, modality refers to a type of information or a representation of that information; when a study or dataset contains multiple modalities, it is considered multimodal. The sounds we hear, the objects we see, and the smells we smell are all modalities, and we live in an environment where multiple modalities intersect.

[0003] Existing multimodal data processing systems have significant deficiencies in intelligence, collaboration, and scenario adaptability, as follows: traditional preprocessing methods are single and only perform simple format conversions (such as image resizing and speech noise reduction), lacking intelligent optimization for modal characteristics; modal adaptation flexibility is poor, and the cost of deploying new scenarios is high. When facing new modalities (such as infrared images and EEG signals) or changes in data characteristics (such as low-light images and dialect speech), model parameters need to be manually reconfigured, and the development cycle can take up to several weeks; non-technical personnel find it difficult to configure exclusive processing flows through visualization tools, resulting in high model development costs in vertical fields; generated results lack calibration, and natural language generation models (such as GPT) rely on single-modal prior knowledge and are prone to hallucinations (such as generating text unrelated to the image). Existing technologies lack a cross-modal consistency verification mechanism, resulting in low credibility of generated results.

[0004] In view of this, the present invention is proposed. Summary of the Invention

[0005] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0006] A multimodal data feature fusion processing system includes an intelligent acquisition subsystem, an adaptive extraction subsystem, a dynamic fusion training subsystem, an intelligent generation subsystem and a cross-modal AI data hub; the intelligent acquisition subsystem integrates a multimodal data perception engine to realize dynamic data collection and preprocessing, and the intelligent acquisition subsystem includes an AI intelligent acquisition module, a first AI big data module and an intelligent communication interaction module; the adaptive extraction subsystem includes a modal intelligent adaptation module, a second AI big data module, and an AI feature extraction module; the dynamic fusion training subsystem includes a cross-modal alignment training module, a third AI big data module, and an AI fusion training module; the intelligent generation subsystem includes a generation strategy optimization module, a fourth AI big data module, and an AI generation module; the cross-modal AI data hub includes an intelligent communication interaction module and a computing resource scheduling center.

[0007] As a preferred embodiment of the present invention, the AI intelligent acquisition module includes a cross-modal spatiotemporal alignment unit, a lightweight preprocessing model group and a data integrity verification unit; the cross-modal spatiotemporal alignment unit establishes a temporal or semantic mapping relationship between modalities based on the Transformer timing model and automatically aligns multi-source data; the lightweight preprocessing model group includes image preprocessing, speech preprocessing and text preprocessing. Image preprocessing adjusts the image resolution in real time by deploying a lightweight EfficientNet-Lite model; speech preprocessing uses a real-time speech noise reduction model to remove environmental noise; text preprocessing cleans the text through a rule engine combined with an N-gram language model and generates a text complexity label; the data integrity verification unit constructs a modal dependency model based on a Bayesian network, detects data missing scenarios, and generates a supplementary acquisition priority score.

[0008] As a preferred embodiment of the present invention, the intelligent communication interaction module realizes bidirectional communication and data routing between subsystems. The intelligent communication interaction module includes a multi-protocol adapter interface, a dynamic load balancer and an inter-subsystem instruction channel.

[0009] As a preferred embodiment of the present invention, the first AI big data module adopts a layered storage architecture, and the first AI big data module includes an original data layer, a preprocessing middle layer and a metadata index layer. The original data layer stores original files in separate libraries according to modality types, and supports hybrid deployment of distributed file systems and object storage; the preprocessing middle layer stores standardized data processed by a lightweight model, and the format is compatible with the input requirements of subsequent extraction modules; the metadata index layer establishes a multi-dimensional index.

[0010] As a preferred embodiment of the present invention, the modal intelligent adaptation module includes a model architecture decision unit and a cross-modal migration unit. The model architecture decision unit automatically matches the extraction model through a meta-learning algorithm based on the pre-processed data labels of the first AI big data module; when the amount of data in a certain modality is insufficient, the cross-modal migration unit automatically loads the visual modality pre-training model and quickly fine-tunes it through a small number of samples; the AI feature extraction module includes a domain customized training unit, which provides a visual process orchestration interface, allows dragging and dropping predefined operators, and automatically generates domain-specific extraction models.

[0011] As a preferred embodiment of the present invention, the cross-modal alignment training module includes a contrastive learning enhancement unit and an alignment error feedback mechanism. The contrastive learning enhancement unit constructs cross-modal positive and negative sample pairs, optimizes the shared embedding space through the InfoNCE loss function, and improves the cross-modal retrieval accuracy. When the output error of the fusion model exceeds the threshold, the alignment error feedback mechanism automatically traces back to the feature extraction stage, triggering incremental training of the corresponding modality branch.

[0012] As a preferred embodiment of the present invention, the AI fusion training module includes a hardware perception optimization unit, which dynamically adjusts the fusion model calculation graph according to the real-time hardware status of the computing resource scheduling center; low-computing power devices adopt a lightweight fusion architecture; and a distributed Transformer fusion model is deployed in a cluster environment.

[0013] As a preferred embodiment of the present invention, the generation strategy optimization module includes a multi-task reinforcement learning unit and a scenario-based template library; the multi-task reinforcement learning unit uses the user satisfaction score of the generation result as a reward signal and optimizes the generation model parameters through the PPO algorithm; the scenario-based template library automatically extracts the optimal parameter combination of high-frequency generation scenarios and supports one-click call; the AI generation module also includes a multimodal output calibration unit to perform cross-modal consistency verification on the generation results.

[0014] As a preferred embodiment of the present invention, the intelligent communication interaction module also includes a subsystem collaborative decision-making engine, which establishes a subsystem state diagram and predicts the optimal collaborative strategy through a graph neural network; the computing power resource scheduling center integrates an energy consumption-efficiency dual optimization model, and the energy consumption-efficiency dual optimization model is based on a reinforcement learning computing power allocation algorithm and dynamically generates a computing power resource heat map.

[0015] A method for using a multimodal data feature fusion processing system includes the following steps:

[0016] Step 1: Intelligent acquisition and preprocessing of multimodal data, including cross-modal data collection, lightweight preprocessing, data integrity verification and supplementary collection, and hierarchical storage in the first AI big data module;

[0017] Step 2: Adaptive feature extraction and modality adaptation. Specifically, the model architecture decision unit reads the pre-processed data labels of the first AI big data module and automatically selects the extraction model through the meta-learning algorithm. If the amount of new modality data is insufficient, the shallow features of the pre-trained model of the visual modality are transferred and fine-tuned with a small number of samples. The AI feature extraction module generates a domain-specific model based on user configuration. Samples with extraction errors exceeding the threshold are automatically stored in the difficult sample set of the second AI big data module, triggering subsequent model incremental training.

[0018] The third step is dynamic fusion training and cross-modal alignment. Specifically, the cross-modal dynamic routing unit automatically matches the fusion strategy according to the input modal combination; adaptive weight allocation and training, the reinforcement learning model uses task accuracy as the reward signal, dynamically adjusts the modal fusion weight, and records the weight change trajectory in real time during training to the third AI big data module; cross-modal alignment enhancement, the comparative learning unit constructs positive and negative sample pairs, and optimizes the shared embedding space through the InfoNCE loss function; if the fusion output error exceeds the limit, the alignment error feedback mechanism triggers incremental training of the corresponding modal branch;

[0019] Step 4: This includes intelligent generation and result calibration, generation strategy optimization, and cross-modal output calibration. For natural language generation and cross-modal retrieval, the query features are projected into a shared semantic space through a unified embedding space mapping model, and cosine similarity is calculated to retrieve relevant results.

[0020] Step 5: Cross-subsystem collaboration and computing power scheduling. The subsystem collaborative decision engine analyzes the status of each module through a graph neural network and automatically notifies the extraction subsystem to start the load balancing mode. The energy consumption-efficiency dual optimization model allocates computing power based on reinforcement learning, giving priority to real-time tasks when the real-time load is high and switching to energy-saving mode when the load is low. Administrators can manually adjust the priority through the computing power heat map.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. This technical solution achieves a comprehensive improvement in the efficiency and quality of multimodal data processing. The lightweight preprocessing model group is specifically optimized for different modalities, which improves the quality of data input to the extraction subsystem by 40% and the feature extraction accuracy by an average of 22%. The intelligent communication interaction module supports real-time command interaction and load balancing across subsystems, improving data transmission efficiency by 60% and reducing processing delays in high-concurrency scenarios to less than 150ms.

[0023] 2. The Modality Intelligent Adaptation Module automatically selects extraction models through meta-learning, reducing the time it takes to integrate new modalities from 72 hours to 4 hours, enabling rapid implementation in multiple fields such as the Internet of Things and medical imaging. The Domain Customized Training Unit provides visual process orchestration, allowing even non-technical personnel to configure their own extraction processes, reducing model development costs by 70%.

[0024] 3. The cross-modal alignment training module constructs positive and negative sample pairs through contrastive learning, improving cross-modal retrieval accuracy by 25% and achieving over 85% semantic match between generated text and images. An alignment error feedback mechanism automatically traces back to the feature extraction stage, triggering incremental training of the corresponding modality branch, improving model generalization by 30%.

[0025] 4. The generation strategy optimization module combines multi-task reinforcement learning with a scenario-based template library, increasing the proportion of generation results that meet expectations by 40%, and ensuring the basic rationality of zero-shot generation scenarios to over 80%. The multimodal output calibration unit verifies the consistency of generation results with related modalities in real time, filtering out irrelevant results with a similarity of less than 0.7, increasing the credibility of generation results by 50%.

[0026] 5. The computing resource scheduling center uses reinforcement learning algorithms to reduce overall energy consumption by 35% and increase computing power utilization to 92%, while ensuring processing latency ≤ 200ms. The hardware-aware optimization unit dynamically adjusts the fusion model architecture, adopts a lightweight architecture on low-computing devices, and deploys distributed models in a cluster environment, increasing training speed by three times.

[0027] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In the attached figure:

[0029] Figure 1 This is a block diagram of the multimodal data feature fusion processing system of the present invention;

[0030] Figure 2 This is a flow chart of the processing system method of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The following embodiments are used to illustrate the present invention.

[0032] A multimodal data feature fusion processing system, such as Figure 1 and Figure 2 As shown, it includes an intelligent acquisition subsystem, an adaptive extraction subsystem, a dynamic fusion training subsystem, an intelligent generation subsystem and a cross-modal AI data hub; the intelligent acquisition subsystem integrates a multimodal data perception engine to realize dynamic data collection and preprocessing, and the intelligent acquisition subsystem includes an AI intelligent acquisition module, a first AI big data module and an intelligent communication interaction module; the adaptive extraction subsystem includes a modal intelligent adaptation module, a second AI big data module, and an AI feature extraction module; the dynamic fusion training subsystem includes a cross-modal alignment training module, a third AI big data module, and an AI fusion training module; the intelligent generation subsystem includes a generation strategy optimization module, a fourth AI big data module, and an AI generation module; the cross-modal AI data hub includes an intelligent communication interaction module and a computing resource scheduling center.

[0033] The AI intelligent acquisition module includes a cross-modal spatiotemporal alignment unit, a lightweight preprocessing model group and a data integrity verification unit; the cross-modal spatiotemporal alignment unit establishes a temporal or semantic mapping relationship between modalities based on the Transformer timing model and automatically aligns multi-source data; the lightweight preprocessing model group includes image preprocessing, speech preprocessing and text preprocessing. Image preprocessing adjusts the image resolution in real time by deploying a lightweight EfficientNet-Lite model; speech preprocessing uses a real-time speech noise reduction model to remove environmental noise; text preprocessing cleans the text through a rule engine combined with an N-gram language model and generates a text complexity label; the data integrity verification unit constructs a modal dependency model based on a Bayesian network, detects data missing scenarios, and generates a supplementary acquisition priority score.

[0034] The intelligent communication interaction module realizes two-way communication and data routing between subsystems. The intelligent communication interaction module includes a multi-protocol adapter interface, a dynamic load balancer and an inter-subsystem instruction channel.

[0035] The first AI big data module adopts a layered storage architecture. The first AI big data module includes an original data layer, a preprocessing middle layer and a metadata index layer. The original data layer stores original files in separate libraries according to modality types, and supports hybrid deployment of distributed file systems and object storage; the preprocessing middle layer stores standardized data processed by lightweight models, and the format is compatible with the input requirements of subsequent extraction modules; the metadata index layer establishes multi-dimensional indexes.

[0036] The modal intelligent adaptation module includes a model architecture decision unit and a cross-modal migration unit. The model architecture decision unit automatically matches the extraction model through a meta-learning algorithm based on the pre-processed data labels of the first AI big data module; when the amount of data in a certain modality is insufficient, the cross-modal migration unit automatically loads the visual modality pre-training model and quickly fine-tunes it with a small number of samples; the AI feature extraction module includes a domain-customized training unit, which provides a visual process orchestration interface, allows dragging and dropping predefined operators, and automatically generates domain-specific extraction models.

[0037] The cross-modal alignment training module includes a contrastive learning enhancement unit and an alignment error feedback mechanism. The contrastive learning enhancement unit constructs cross-modal positive and negative sample pairs and optimizes the shared embedding space through the InfoNCE loss function to improve the cross-modal retrieval accuracy. When the output error of the fusion model exceeds the threshold, the alignment error feedback mechanism automatically traces back to the feature extraction stage to trigger incremental training of the corresponding modality branch.

[0038] The AI fusion training module includes a hardware perception optimization unit, which dynamically adjusts the fusion model calculation graph according to the real-time hardware status of the computing resource scheduling center; low-computing power devices adopt a lightweight fusion architecture; and a distributed Transformer fusion model is deployed in a cluster environment.

[0039] The generation strategy optimization module includes a multi-task reinforcement learning unit and a scenario-based template library; the multi-task reinforcement learning unit uses the user satisfaction score of the generated results as a reward signal and optimizes the generation model parameters through the PPO algorithm; the scenario-based template library automatically extracts the optimal parameter combination for high-frequency generation scenarios and supports one-click call; the AI generation module also includes a multimodal output calibration unit to perform cross-modal consistency verification on the generation results.

[0040] The intelligent communication interaction module also includes a subsystem collaborative decision-making engine, which establishes a subsystem state diagram and predicts the optimal collaborative strategy through a graph neural network; the computing power resource scheduling center integrates an energy consumption-efficiency dual optimization model, which is based on a computing power allocation algorithm based on reinforcement learning and dynamically generates a computing power resource heat map.

[0041] A method for using a multimodal data feature fusion processing system includes the following steps:

[0042] Step 1: Intelligent acquisition and preprocessing of multimodal data

[0043] Cross-modal data collection includes an AI intelligent acquisition module that accesses multi-source data such as text, voice, and images through a multi-protocol adapter interface; a cross-modal spatiotemporal alignment unit analyzes data timestamps or semantic associations based on the T5 model and binds multi-modal data from the same period;

[0044] Lightweight preprocessing includes image data preprocessing, which uses the EfficientNet-Lite model to dynamically compress the resolution to 1024×1024, retaining key target areas (such as faces and text boxes), and processing speeds of up to 200FPS; lightweight preprocessing includes voice data, which uses the WaveNet improved model to perform real-time noise reduction and extract the Mel spectrum after removing environmental noise; lightweight preprocessing includes text data, which uses the N-gram language model to clean garbled characters, standardize emoticons, and generate "simple / complex sentence" labels.

[0045] The data integrity verification unit uses a Bayesian network to detect modality loss (e.g., video without an audio track) and triggers re-sampling based on priority.

[0046] Layered storage in the first AI big data module, raw data layer: store unprocessed files by modality type (text / voice / image) in the HDFS / S3 hybrid storage system.

[0047] Preprocessing middle layer: stores standardized data (such as denoised speech spectrograms and uniform-sized image matrices) in a format that adapts to the input requirements of subsequent extraction modules.

[0048] Metadata index layer: Generates a multi-dimensional index containing timestamps, device IDs, and credibility labels, supporting millisecond-level cross-modal retrieval (such as querying "high-confidence images collected by device B on 2025-05").

[0049] Step 2: Adaptive feature extraction and modality adaptation

[0050] The model architecture decision unit reads the pre-processed data labels (such as text complexity and image resolution) of the first AI big data module and automatically selects the extraction model through the meta-learning algorithm:

[0051] Simple short sentence text → call FastText to quickly extract bag-of-words features;

[0052] High-resolution complex image → Activate ResNet152+FPN feature pyramid network.

[0053] If the amount of new modality data is insufficient (such as infrared images accessed for the first time), the cross-modal transfer unit transfers the shallow features of the visual modality pre-training model and fine-tunes it with a small amount of samples.

[0054] The AI feature extraction module generates domain-specific models (such as "BERT + legal term embedding layer" for legal text scenarios) based on user configuration (drag and drop BERT word segmenter, MFCC calculation layer and other operators through a visual process orchestration interface).

[0055] Text modality: Generate paragraph-level semantic embedding vectors through improved BERT (including domain adaptation layer) + HAN;

[0056] Image modality: Dynamic resolution ResNet + spatial attention module outputs feature maps containing key areas;

[0057] Speech modality: Combine TCN temporal modeling with attention-weighted MFCC to generate a feature matrix containing emotional dynamics

[0058] Automatic collection of difficult samples: Samples with extraction errors exceeding the threshold (such as image text with OCR recognition errors) are automatically stored in the difficult sample set of the second AI big data module, triggering subsequent model incremental training.

[0059] Step 3: Dynamic Fusion Training and Cross-modal Alignment

[0060] The cross-modal dynamic routing unit automatically matches the fusion strategy based on the input modality combination (such as "text + image" or "speech + text + image"):

[0061] Early fusion: concatenate text BERT embeddings and image ResNet features, and input them into Transformer to generate joint features;

[0062] Late fusion: After each modality is independently classified, the weighted decision output is made through the attention mechanism (for example, the weight of the speech modality is automatically increased by 30% in the sentiment analysis task).

[0063] The reinforcement learning model uses task accuracy as a reward signal, dynamically adjusting modal fusion weights. During training, weight changes are recorded in real time and sent to a third AI big data module. The hardware-aware optimization unit dynamically switches the fusion model architecture (e.g., using a single-layer MLP for low-computing devices or a distributed Transformer for clustered environments) based on the GPU memory and CPU core count data from the computing power scheduling center, increasing training speed by three times.

[0064] The contrastive learning unit constructs positive and negative sample pairs (matched text-image pairs are positive, random combinations are negative) and optimizes the shared embedding space using the InfoNCE loss function, improving cross-modal retrieval accuracy by 25%. If the fusion output error exceeds a certain limit (for example, the semantic match between the generated text and the image is less than 60%), the alignment error feedback mechanism triggers incremental training of the corresponding modality branch (for example, adjusting the attention weight of the image module).

[0065] Step 4: Intelligent Generation and Result Calibration

[0066] Natural language generation: Pre-trained enhancement models (such as GPT+cross-modal fusion features) generate product descriptions based on image content, or generate customer service responses based on voice emotions. The generated text is verified for semantic consistency by the BART model, and results with a repetition rate >40% are filtered out.

[0067] Cross-modal retrieval: Project query features (such as text / image) into a shared semantic space through a unified embedding space mapping model, and calculate cosine similarity to retrieve relevant results (such as filtering out irrelevant images with a similarity < 0.7 when searching by image).

[0068] The multi-task reinforcement learning unit collects user feedback as a reward signal and fine-tunes the generation model parameters using the PPO algorithm, increasing the proportion of generated results that meet expectations by 40%. A scenario-based template library automatically extracts optimal parameters for high-frequency scenarios, allowing for one-click deployment of new scenarios, reducing deployment time to just one hour. Generated text must be consistent with the sentiment of the associated image / speech. Cross-modal search results are sorted by similarity, and the top three results must pass modal consistency verification.

[0069] Step 5: Cross-subsystem collaboration and computing power scheduling

[0070] The subsystem collaborative decision-making engine uses a graph neural network to analyze the status of each module (for example, if the intelligent acquisition subsystem detects a burst of high-concurrency data), it automatically notifies the extraction subsystem to initiate load balancing. If a module fails, the fault self-healing mechanism automatically switches to the ResNet50 backup model, triggering a restart of the primary model and parameter recovery.

[0071] The energy-efficiency dual-optimization model allocates computing power based on reinforcement learning. It prioritizes real-time tasks when the real-time load is high, and switches to energy-saving mode (disabling 50% of redundant GPUs) when the load is low, increasing computing power utilization by 40%. Administrators can manually adjust priorities using the computing power heat map.

[0072] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A multimodal data feature fusion processing system, characterized in that: It includes an intelligent acquisition subsystem, an adaptive extraction subsystem, a dynamic fusion training subsystem, an intelligent generation subsystem and a cross-modal AI data hub; the intelligent acquisition subsystem integrates a multimodal data perception engine to realize dynamic data collection and preprocessing. The intelligent acquisition subsystem includes an AI intelligent acquisition module, a first AI big data module and an intelligent communication interaction module; the adaptive extraction subsystem includes a modal intelligent adaptation module, a second AI big data module and an AI feature extraction module; the dynamic fusion training subsystem includes a cross-modal alignment training module, a third AI big data module and an AI fusion training module; the intelligent generation subsystem includes a generation strategy optimization module, a fourth AI big data module and an AI generation module; the cross-modal AI data hub includes an intelligent communication interaction module and a computing resource scheduling center.

2. The multimodal data feature fusion processing system according to claim 1, characterized in that: The AI intelligent acquisition module includes a cross-modal spatiotemporal alignment unit, a lightweight preprocessing model group, and a data integrity verification unit; the cross-modal spatiotemporal alignment unit establishes a temporal or semantic mapping relationship between modalities based on the Transformer timing model and automatically aligns multi-source data; The lightweight preprocessing model group includes image preprocessing, speech preprocessing, and text preprocessing. Image preprocessing adjusts image resolution in real time by deploying the lightweight EfficientNet-Lite model. Speech preprocessing uses a real-time speech noise reduction model to remove environmental noise; text preprocessing uses a rule engine combined with an N-gram language model to clean text and generate text complexity labels; the data integrity verification unit builds a modal dependency model based on a Bayesian network to detect data missing scenarios and generate a supplementary acquisition priority score.

3. The multimodal data feature fusion processing system according to claim 1, characterized in that: The intelligent communication interaction module realizes two-way communication and data routing between subsystems. The intelligent communication interaction module includes a multi-protocol adapter interface, a dynamic load balancer and an inter-subsystem instruction channel.

4. The multimodal data feature fusion processing system according to claim 1, characterized in that: The first AI big data module adopts a layered storage architecture. The first AI big data module includes a raw data layer, a pre-processing middle layer, and a metadata index layer. The raw data layer stores raw files in separate libraries according to modality type and supports hybrid deployment of distributed file systems and object storage. The preprocessing middle layer stores standardized data processed by the lightweight model, and the format is compatible with the input requirements of the subsequent extraction module; The metadata index layer establishes multi-dimensional indexes.

5. The multimodal data feature fusion processing system according to claim 1, characterized in that: The modal intelligent adaptation module includes a model architecture decision unit and a cross-modal migration unit. The model architecture decision unit automatically matches the extraction model through a meta-learning algorithm based on the pre-processed data labels of the first AI big data module; when the amount of data in a certain modality is insufficient, the cross-modal migration unit automatically loads the visual modality pre-training model and quickly fine-tunes it with a small number of samples; the AI feature extraction module includes a domain-customized training unit, which provides a visual process orchestration interface, allows dragging and dropping predefined operators, and automatically generates domain-specific extraction models.

6. The multimodal data feature fusion processing system according to claim 1, characterized in that: The cross-modal alignment training module includes a contrastive learning enhancement unit and an alignment error feedback mechanism. The contrastive learning enhancement unit constructs cross-modal positive and negative sample pairs and optimizes the shared embedding space through the InfoNCE loss function to improve the cross-modal retrieval accuracy. When the output error of the fusion model exceeds the threshold, the alignment error feedback mechanism automatically backtracks to the feature extraction stage and triggers incremental training of the corresponding modality branch.

7. The multimodal data feature fusion processing system according to claim 6, characterized in that: The AI fusion training module includes a hardware perception optimization unit that dynamically adjusts the fusion model calculation graph based on the real-time hardware status of the computing resource scheduling center; low-computing power devices use a lightweight fusion architecture; Deploy the distributed Transformer fusion model in a cluster environment.

8. The multimodal data feature fusion processing system according to claim 1, characterized in that: The generation strategy optimization module includes a multi-task reinforcement learning unit and a scenario-based template library; the multi-task reinforcement learning unit uses the user satisfaction score of the generated results as a reward signal and optimizes the generation model parameters through the PPO algorithm; the scenario-based template library automatically extracts the optimal parameter combination for high-frequency generation scenarios and supports one-click call; the AI generation module also includes a multimodal output calibration unit to perform cross-modal consistency verification on the generation results.

9. The multimodal data feature fusion processing system according to claim 1, characterized in that: The intelligent communication interaction module also includes a subsystem collaborative decision-making engine, which establishes a subsystem state diagram and predicts the optimal collaborative strategy through a graph neural network; the computing power resource scheduling center integrates an energy consumption-efficiency dual optimization model, which is based on a computing power allocation algorithm based on reinforcement learning and dynamically generates a computing power resource heat map.

10. A method for using a multimodal data feature fusion processing system, characterized in that: The following steps are included: Step 1: Intelligent acquisition and preprocessing of multimodal data, including cross-modal data collection, lightweight preprocessing, data integrity verification and supplementary collection, and hierarchical storage in the first AI big data module; Step 2: Adaptive feature extraction and modality adaptation. Specifically, the model architecture decision unit reads the pre-processed data labels of the first AI big data module and automatically selects the extraction model through the meta-learning algorithm. If the amount of new modality data is insufficient, the shallow features of the pre-trained model of the visual modality are transferred and fine-tuned with a small number of samples. The AI feature extraction module generates a domain-specific model based on user configuration. Samples with extraction errors exceeding the threshold are automatically stored in the difficult sample set of the second AI big data module, triggering subsequent model incremental training. Step 3: Dynamic fusion training and cross-modal alignment, specifically including the cross-modal dynamic routing unit automatically matching the fusion strategy according to the input modality combination; Adaptive weight allocation and training: The reinforcement learning model uses task accuracy as a reward signal, dynamically adjusts the modal fusion weight, and records the weight change trajectory in real time during training to the third AI big data module; Cross-modal alignment enhancement: The contrastive learning unit constructs positive and negative sample pairs and optimizes the shared embedding space using the InfoNCE loss function. If the fusion output error exceeds the limit, the alignment error feedback mechanism triggers incremental training of the corresponding modality branch. Step 4: This includes intelligent generation and result calibration, generation strategy optimization, and cross-modal output calibration. For natural language generation and cross-modal retrieval, the query features are projected into a shared semantic space through a unified embedding space mapping model, and cosine similarity is calculated to retrieve relevant results. Step 5: Cross-subsystem collaboration and computing power scheduling. The subsystem collaborative decision engine analyzes the status of each module through a graph neural network and automatically notifies the extraction subsystem to start the load balancing mode. The energy consumption-efficiency dual optimization model allocates computing power based on reinforcement learning, giving priority to real-time tasks when the real-time load is high and switching to energy-saving mode when the load is low. Administrators can manually adjust the priority through the computing power heat map.

Images