Multi-modal data processing method and system, terminal and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a multi-modal data processing method and system, a terminal and a storage medium, and the method comprises the steps: capturing multi-source heterogeneous data in real time through a unified access layer, and dynamically analyzing semantics by using a large language model, and generating an abstract tag to update a unified directory; performing format normalization and cross-modal alignment based on comparative learning on the data, and extracting a standardized multi-modal feature vector through feature engineering; constructing a data consanguinity map, predicting a failure risk by using a map neural network, automatically identifying sensitive information in combination with a multi-modal identification model, and dynamically triggering a treatment strategy; a reinforcement learning optimization strategy is utilized, data drift is detected through a time sequence model, and synthetic data is generated by means of a diffusion model to correct distribution; and recording the full-link feature version and the data hash, and generating a traceable compliance report. According to the invention, real-time, intelligent and closed-loop treatment of the multi-modal data is realized, and the AI training data quality and treatment efficiency are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal data processing method, system, terminal, and computer-readable storage medium. Background Technology

[0002] With the rapid development of large-scale models and generative AI, AI systems are placing higher demands on the scale, quality, diversity, and real-time nature of data. Traditional data governance models are based on static rules, primarily targeting structured data, and lack the ability to process unstructured data. Furthermore, the governance process is passive and lagging, making it difficult to meet the dynamic needs of AI training and inference. In addition, multimodal data (such as text, images, audio, and video) suffers from heterogeneity, semantic inconsistency, and synchronization difficulties, resulting in high data fusion challenges and governance costs.

[0003] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0004] The main objective of this invention is to provide a multimodal data processing method, system, terminal, and computer-readable storage medium, aiming to solve the problems of poor real-time performance, weak adaptability, and severe data silos in traditional data governance applications for AI.

[0005] To achieve the above objectives, the present invention provides a multimodal data processing method, which includes the following steps: The system can access and capture multimodal raw data streams from multiple heterogeneous data sources in real time, perform dynamic semantic parsing on the raw data streams using a large language model, generate dynamic semantic context, summaries and tags for data objects, and update the unified data catalog. Based on the dynamic semantic context, the original data stream is normalized and cross-modal aligned to extract standardized multimodal feature vectors. Construct a dynamic data graph; based on the dynamic data graph and the multimodal feature vector, use a graph neural network to predict the risk of data failure in real time, and use a multimodal content recognition model to identify sensitive information. According to the predicted risk level and the identified sensitive content, dynamically trigger corresponding data quality repair, security level adjustment, desensitization, encryption or access control strategies. Collect policy execution effects and training feedback, continuously optimize governance policies using reinforcement learning models, monitor the distribution drift of the multimodal feature vectors based on a temporal anomaly detection model, and generate targeted synthetic data through a diffusion model to complete or correct the data distribution. For each set of standardized multimodal feature vectors and derived synthetic data generated, the model version, parameters, and input data hash values ​​used in the generation are recorded, and a fully auditable compliance report is generated.

[0006] Optionally, the multimodal data processing method, wherein the real-time access and capture of multimodal raw data streams from multiple heterogeneous data sources, the dynamic semantic parsing of the raw data streams using a large language model to generate dynamic semantic context, summaries, and tags for data objects, and the updating of a unified data catalog, specifically includes: Through a configured unified access layer, multiple data sources, including databases, message queues, object storage, and IoT devices, can be accessed via APIs, database connections, SaaS application interfaces, or IoT protocols. This allows for the real-time capture of multimodal raw data streams from multiple heterogeneous data sources, including both structured and unstructured data. The multimodal raw data stream is dynamically semantically parsed using a large language model to understand the data content, structure, and context, generate data summaries, business tags, and technical tags, and continuously optimize the metadata description based on user feedback on the summaries and tags. The data objects, dynamically updated data definitions, structures, semantic contexts, summaries, and tags are registered or updated to a unified data directory in real time.

[0007] Optionally, in the multimodal data processing method, the dynamic semantic context performs format normalization and cross-modal alignment processing on the original data stream to extract standardized multimodal feature vectors, specifically including: The format normalization engine converts files from different sources in the original data stream into standardized formats and performs unified encoding on the text data. Through the cross-modal alignment module, a contrastive learning method is used to align text-image pairs and audio-video streams at the semantic level to ensure that data from different modalities express the same topic or concept. The normalized and aligned modal data are processed using a feature engineering pipeline to obtain the features extracted from each modality. By fusing the features extracted from each modality, a set of standardized multimodal feature vectors with a unified dimension is generated.

[0008] Optionally, in the multimodal data processing method, the step of performing feature engineering pipeline processing on the normalized and aligned modal data to obtain the features extracted from each modality specifically includes: Generate word vectors, sentence vectors, and Chain-of-Thought enhanced features from text data; Visual embedding features based on ResNet or ViT are extracted from image data and fused with text information extracted by OCR and object detection labels; For audio and video data, extract text features, keyframe visual features, and voiceprint features from ASR transcription.

[0009] Optionally, in the multimodal data processing method, the construction of a dynamic data graph, based on the dynamic data graph and the multimodal feature vectors, utilizes a graph neural network to predict data failure risks in real time, and uses a multimodal content recognition model to identify sensitive information. According to the predicted risk level and the identified sensitive content, corresponding data quality repair, security level adjustment, desensitization, encryption, or access control strategies are dynamically triggered, specifically including: Construct an end-to-end data flow graph as a dynamic data map to record the entire lifecycle of data from collection, cleaning, parsing to feature generation; A graph neural network is used to analyze the dynamic data map, predict the risk of data failure, and locate the root cause. Based on a multimodal content recognition model, the multimodal feature vectors are scanned to automatically identify sensitive information, including faces, medical images, contract terms, and personal identification information. Based on the predicted data failure risk and the identified categories and levels of the sensitive information, the security level of the data object is dynamically adjusted, and corresponding data desensitization, encryption operations, or fine-grained access control policies are automatically triggered.

[0010] Optionally, the multimodal data processing method, wherein the collection strategy execution effect and training feedback are continuously optimized using a reinforcement learning model, the distribution drift of the multimodal feature vectors is monitored based on a temporal anomaly detection model, and targeted synthetic data is generated through a diffusion model to supplement or correct the data distribution, specifically includes: An intelligent governance decision engine is built using a reinforcement learning model. The reinforcement learning model takes the overall cost of data governance and the performance of downstream AI models as optimization objectives. Through interaction with the environment, it dynamically optimizes the data sampling frequency, the trigger threshold of cleaning rules, and the access control strategy. Based on the time-series anomaly detection model, the distribution of the multimodal feature vectors is continuously monitored to predict and identify data drift phenomena. By using a diffusion model, synthetic data that conforms to the target distribution is generated to address identified data imbalances or drift issues, and this data is then used to enhance AI model training.

[0011] Optionally, the multimodal data processing method, wherein the step of generating each set of standardized multimodal feature vectors and derived synthetic data, recording the model version, parameters, and input data hash values ​​used in the generation, and generating a fully auditable compliance report, specifically includes: Establish a version management mechanism to record the feature engineering model version used during generation, all processing parameters, and the cryptographic hash value of the input raw data for each set of standardized multimodal feature vectors and synthetic data generated. Based on the recorded feature versions, parameters, and hash information, a compliance audit report is generated that includes the full-chain lineage of data, features, models, and decisions.

[0012] Furthermore, to achieve the above objectives, the present invention also provides a multimodal data processing system, wherein the multimodal data processing system comprises: The data access module is used to access and capture multimodal raw data streams from multiple heterogeneous data sources in real time, perform dynamic semantic parsing of the raw data streams using a large language model, generate dynamic semantic context, summary and tags of data objects, and update the unified data catalog. The multimodal data fusion processing module is used to perform format normalization and cross-modal alignment processing on the original data stream based on the dynamic semantic context, and extract and form a standardized multimodal feature vector; The data lineage tracing module is used to construct a dynamic data graph; based on the dynamic data graph and the multimodal feature vector, it uses a graph neural network to predict the risk of data failure in real time, and uses a multimodal content recognition model to identify sensitive information. According to the predicted risk level and the identified sensitive content, it dynamically triggers corresponding data quality repair, security level adjustment, desensitization, encryption or access control strategies. The intelligent governance decision-making module is used to collect policy execution effects and training feedback, continuously optimize governance policies using reinforcement learning models, monitor the distribution drift of the multimodal feature vectors based on a time-series anomaly detection model, and generate targeted synthetic data through a diffusion model to complete or correct the data distribution. The report generation module is used to record the model version, parameters, and input data hash value used in the generation of each set of standardized multimodal feature vectors and derived synthetic data, and generate a fully auditable compliance report.

[0013] In addition, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multimodal data processing program stored in the memory and executable on the processor, wherein when the multimodal data processing program is executed by the processor, it implements the steps of the multimodal data processing method as described above.

[0014] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal data processing program, which, when executed by a processor, implements the steps of the multimodal data processing method as described above.

[0015] In this invention, multimodal raw data streams from multiple heterogeneous data sources are accessed and captured in real time. A large language model is used to dynamically semantically parse the raw data streams, generating dynamic semantic contexts, summaries, and tags for data objects, and updating a unified data catalog. Based on the dynamic semantic context, the raw data streams are normalized and cross-modal aligned to extract standardized multimodal feature vectors. A dynamic data graph is constructed. Based on the dynamic data graph and the multimodal feature vectors, a graph neural network is used to predict data failure risks in real time, and a multimodal content recognition model is used to identify sensitive information. According to the predicted risk level and the identified sensitive content, corresponding data quality repair, security level adjustment, desensitization, encryption, or access control strategies are dynamically triggered. The effectiveness of strategy execution and training feedback are collected, and a reinforcement learning model is used to continuously optimize the governance strategy. A time-series anomaly detection model is used to monitor the distribution drift of the multimodal feature vectors, and a diffusion model is used to generate targeted synthetic data to complete or correct the data distribution. For each set of standardized multimodal feature vectors and derived synthetic data generated, the model version, parameters, and input data hash value used in generation are recorded, generating a fully auditable compliance report. This invention employs a machine learning model to detect and predict data anomalies in real time and dynamically adjust governance strategies, thereby achieving real-time, intelligent, and closed-loop governance of multimodal data and significantly improving the quality of AI training data and governance efficiency. Attached Figure Description

[0016] Figure 1 This is a flowchart of a preferred embodiment of the multimodal data processing method of the present invention; Figure 2 This is a schematic diagram illustrating the entire process of fusing multimodal data in a preferred embodiment of the multimodal data processing method of the present invention; Figure 3 This is a structural diagram of a preferred embodiment of the multimodal data processing system of the present invention; Figure 4 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0018] The multimodal data processing method described in the preferred embodiment of the present invention, such as... Figure 1 and Figure 2 As shown, the multimodal data processing method includes the following steps: Step S10: Real-time access and capture of multimodal raw data streams from multiple heterogeneous data sources, use a large language model to perform dynamic semantic parsing of the raw data streams, generate dynamic semantic context, summary and tags of data objects, and update the unified data catalog.

[0019] Specifically, through a configured unified access layer, multiple data sources, including databases, message queues, object storage, and IoT devices, can be accessed via APIs, database connections, SaaS application interfaces, or IoT protocols. This allows for the real-time capture of multimodal raw data streams from multiple heterogeneous data sources, including structured and unstructured data.

[0020] The captured raw data stream (including file content, text fragments, metadata, etc.) is input into a pre-trained large language model. The large language model performs dynamic semantic parsing on the multimodal raw data stream (e.g., deep contextual understanding, entity recognition, and relation extraction for text data; for non-text data (such as images and audio), it performs joint analysis based on metadata such as filename, path, and surrounding text descriptions to infer latent semantics). This process understands the data content, structure, and context, generating data summaries, business tags, and technical tags. For example, based on the parsing results, the system automatically generates concise and accurate data summaries that summarize the core content of the data. Simultaneously, multi-dimensional tags are generated, including business tags reflecting business attributes (such as "sales contract" and "product fault log") and technical tags describing technical characteristics (such as "JPEG format" and "contains faces"). This greatly improves the interpretability and discoverability of the data, and continuously optimizes the metadata description based on user feedback on the summary and tags (i.e., users can manually modify the data summary and tags). The system allows users to verify, correct or supplement the automatically generated summary and tags. This user feedback is collected and used as training data or prompts to continuously optimize the subsequent semantic parsing and annotation process, so that the metadata description evolves and becomes more accurate with use.

[0021] The data objects, dynamically updated data definitions, structures, semantic contexts, summaries, and tags are registered or updated in real time to a unified data directory. In other words, the system maintains a unified data directory as a "living map" of enterprise data. Once a data object completes semantic parsing and annotation, its key information (including the data object's logical identifier, physical location, dynamically updated data definitions and structures, generated semantic context, summaries, and tags) is immediately registered (for new data) or updated (for existing data) to this directory. This dynamically updated directory is no longer a static asset list, but a knowledge base that reflects the full picture, connotation, and relationships of the data in real time. It enables downstream systems (such as data scientists and AI training pipelines) to quickly and accurately discover, understand, and acquire the high-value data they need based on rich semantic information.

[0022] Through a pluggable unified access layer, it flexibly supports almost all modern data sources, including databases, SaaS, and IoT, solving the "last mile" problem of data access and ensuring the timeliness of data supply. It endows data with machine-understandable semantics, automatically generating high-quality summaries and contextual tags using large language models, transforming "dark data" into "explicit data," fundamentally improving the usability of unstructured data and providing richer learning materials for AI models. It constructs a dynamic and intelligent data asset map, with a unified real-time synchronization mechanism for the data catalog, ensuring the accuracy and timeliness of the data asset view. This provides a reliable foundation for agile data discovery, lineage analysis, and governance decisions, significantly reducing data management costs and AI project startup time.

[0023] Step S20: Based on the dynamic semantic context, perform format normalization and cross-modal alignment processing on the original data stream to extract and form a standardized multimodal feature vector.

[0024] Specifically, a format normalization engine converts files from different sources in the original data stream into standardized formats and performs unified encoding on text data. For example, the system has a powerful built-in format normalization engine that calls corresponding decoding and transcoding libraries for standardization conversion of different types of unstructured files. For instance, it converts various versions of PDF documents into high-resolution JPG image sequences for visual analysis; unifies various encoded images into JPG format; decodes and re-encodes various audio files (such as MP3, AAC) into standard uncompressed WAV format; and transcodes various encapsulated and encoded video files (such as AVI, MOV) into standard MP4 format. Simultaneously, all text data (from documents, databases, or the network) undergoes unified character encoding (such as UTF-8) and basic cleaning to ensure the purity and consistency of the text input.

[0025] Through a cross-modal alignment module, employing contrastive learning, image-text pairs and audio / video streams are semantically aligned to ensure that data from different modalities express the same topic or concept. For example, the system implements this function through a deep learning-based cross-modal alignment module. The core of this module is trained using contrastive learning. The alignment process involves mapping the normalized representations of known related data pairs (e.g., an image and its title, a video and its audio narration) to a shared semantic vector space using a modality-specific encoder. Through contrastive learning objectives (such as InfoNCE loss), the module shortens the distance between related image-text pairs and audio / video streams in this space, while widening the distance between unrelated pairs. During the inference phase, this module ensures that image-text pairs and audio / video streams describing the same topic or concept are mapped to similar positions in this shared semantic space, thus achieving semantic alignment and laying the foundation for subsequent feature fusion.

[0026] A feature engineering pipeline is performed on the normalized and aligned modal data to obtain the features extracted for each modality. The aim is to extract deep, rich, and machine-understandable features from the aligned modal data, specifically including: Text feature extraction: For text data, in addition to generating basic word vectors (such as Word2Vec, GloVe) and sentence vectors (such as Sentence-BERT), Chain-of-Thought enhanced features are also introduced. That is, large language models are used to simulate the reasoning process and generate intermediate representations that describe the implicit logic, causal relationships or steps of the text, as a supplement to deep semantic features.

[0027] Image feature extraction: For image data, high-level visual embedding features are extracted using pre-trained deep convolutional networks (such as ResNet) or visual Transformer models (such as ViT). Simultaneously, optical character recognition is run in parallel to extract OCR text, and object detection models (such as YOLO) are run to obtain object detection labels in the image. These visual, textual, and symbolic features are fused to form a comprehensive understanding of the image content.

[0028] Audio and video feature extraction: For audio data, it is first transcribed into text using automatic speech recognition technology, and then text features are extracted. Simultaneously, voiceprint features are extracted to identify the speaker, and acoustic features such as the audio spectrogram are calculated. For video data, in addition to extracting audio features, keyframe extraction is performed, and the aforementioned image feature extraction process is executed on the keyframes to obtain a temporal visual feature sequence.

[0029] Features extracted from various modalities are fused to generate a standardized multimodal feature vector with a unified dimension. This involves aggregating all extracted heterogeneous features (vectors, sequences, and label embeddings of different dimensions). Deep fusion is then performed using feature concatenation, attention-weighted fusion, or a dedicated multimodal fusion network (such as a Transformer encoder). Finally, the fused high-dimensional features are mapped or pooled into a standardized multimodal feature vector with a unified dimension and fixed length. This vector integrates semantic information from all modalities, forming a compact, efficient, and semantically rich representation that can be directly used in downstream graph neural networks, risk assessment models, or as training input for AI models.

[0030] By automating the complex preprocessing and feature extraction processes through a format normalization engine and feature engineering pipeline, efficiency is significantly improved and human error is reduced. A cross-modal alignment module based on contrastive learning connects data from different modalities within a unified semantic space, enabling deep semantic understanding and fusion—a key breakthrough for achieving high-quality multimodal learning. Through deep fusion of visual embeddings, OCR text, ASR text, and logically enhanced features, standardized multimodal feature vectors are generated that combine surface information with deep semantics, exhibiting uniform dimensionality and high quality. This greatly simplifies the design complexity of downstream AI models and significantly improves model performance.

[0031] Step S30: Construct a dynamic data graph. Based on the dynamic data graph and the multimodal feature vector, use a graph neural network to predict data failure risk in real time, and use a multimodal content recognition model to identify sensitive information. According to the predicted risk level and the identified sensitive content, dynamically trigger corresponding data quality repair, security level adjustment, desensitization, encryption or access control strategies.

[0032] Specifically, an end-to-end data flow graph is constructed as a dynamic data graph, recording the entire lifecycle of data from acquisition, cleaning, parsing to feature generation. That is, the system automatically collects metadata at each stage of data processing (from acquisition and access, cleaning and parsing to feature generation), including data source, processing time, operating entity (such as processing model or task), input-output relationships, etc. This forms the basis of an end-to-end data flow graph, i.e., a dynamic data graph. In this dynamic data graph, data objects (such as an original image or a feature vector) serve as nodes, and data processing operations and the derivation and dependencies between data serve as edges, thus completely recording the entire lifecycle of data from acquisition, cleaning, parsing to feature generation.

[0033] A graph neural network (GNN) is employed to analyze the dynamic data graph, predicting data failure risks and identifying root causes. Specifically, the system uses a GNN for deep analysis of the dynamic data graph. The GNN aggregates information about nodes and their neighbors in the graph through a message passing mechanism, enabling it to learn patterns of data quality degradation and value loss. Through training, the GNN can predict the probability of a data node failing within a future time window (e.g., due to source contamination, processing logic errors, or expiration). More importantly, by analyzing the GNN's attention weights or performing subgraph analysis, the upstream nodes or processing stages contributing the greatest risk can be traced, achieving root cause identification and providing direction for precise governance.

[0034] Based on a multimodal content recognition model, the system scans the multimodal feature vectors to automatically identify sensitive information, including faces, medical images, contract terms, and personal identification information. For example, the system deploys pre-trained or continuously learned multimodal content recognition models that directly scan and analyze the generated standardized multimodal feature vectors, fully utilizing their fused semantic information. Furthermore, the model can identify various types of sensitive information, such as: recognizing facial regions and their identity attributes (if authorized) in visual features; identifying lesion areas or protected health information in medical image features; identifying key entities such as amounts, responsible parties, and confidential matters in contract terms by analyzing the combination of text and visual features; and detecting personal identification information such as ID numbers, phone numbers, and addresses in various features. The recognition results include not only the category but also the confidence level and sensitivity level.

[0035] Based on the predicted data failure risk and the identified categories and levels of sensitive information, the security level of data objects is dynamically adjusted, and corresponding data desensitization, encryption operations, or fine-grained access control policies are automatically triggered. For example, the system maintains a dynamic data security level policy engine, which dynamically adjusts the security level of data objects in real time and automatically based on the predicted data failure risk (e.g., high-risk data may mean unreliable quality and requires restricted use) and the identified categories and levels of sensitive information.

[0036] For example, a previously public audio recording, once identified as containing personally identifiable information, may immediately have its security classification upgraded to "internal restricted." Automated policy triggering: Based on the updated security classification and risk labels, the system automatically triggers preset or optimized governance policies. For example: for personal information containing faces, it triggers de-identification strategies (such as facial blurring). For document features involving trade secrets, it triggers encrypted storage and transmission strategies. For data with high risk of failure, it triggers isolation or quality remediation workflow notifications. For all data, fine-grained access control policies are implemented in real time according to its latest security classification, ensuring that only authorized entities can access the corresponding data in an appropriate manner.

[0037] By leveraging dynamic data mapping and Generative Neural Networks (GNNs), the system transforms passive response into proactive prediction, enabling rapid identification of root causes of risks and significantly improving the efficiency of handling data quality issues and the stability of downstream AI models. Identification based directly on deeply fused feature vectors is more accurate and comprehensive than traditional single-modal or rule-based methods, effectively reducing data leakage and compliance risks. A dynamic and precise proactive governance mechanism has been established: risk prediction, sensitive identification, and policy execution are linked in real time, enabling dynamic adaptive adjustment of data security levels and protection measures. This upgrades data governance from a static, "one-size-fits-all" compliance approach to intelligent, proactive protection tailored to specific data needs, optimizing data availability while ensuring security.

[0038] Step S40: Collect policy execution effects and training feedback, continuously optimize the governance policy using a reinforcement learning model, monitor the distribution drift of the multimodal feature vectors based on a time-series anomaly detection model, and generate targeted synthetic data through a diffusion model to complete or correct the data distribution.

[0039] Specifically, a reinforcement learning model is used to construct an intelligent governance decision engine. This model optimizes the overall cost of data governance and the performance of downstream AI models by interacting with the environment, dynamically optimizing data sampling frequency, cleaning rule trigger thresholds, and access control strategies. For example, the system constructs an intelligent governance decision engine whose core is a reinforcement learning model. This model formalizes the entire data governance environment (including data state, governance behavior, and downstream model feedback) as a Markov decision process. The optimization objective of the RL model is to maximize a composite reward signal, which is positively correlated with the performance improvement of downstream AI models (such as accuracy and F1 score) and negatively correlated with the overall cost of data governance (such as computational resource consumption, storage overhead, and manual review costs). The agent (i.e., the decision engine) learns through continuous interaction with the "environment": it observes the current state (such as data quality indicators, the proportion of sensitive data, and system load), dynamically optimizes, and outputs a series of governance actions. These actions directly regulate key parameters, such as adjusting the data sampling frequency, intensively sampling in high-value data areas, and reducing the frequency in stable areas to save resources. Adjust the trigger thresholds for data cleaning and quality restoration rules, automatically relaxing or tightening standards when data source quality fluctuates. Optimize the strictness of access control policies, promoting data flow when risks are manageable and strengthening control when risks escalate. Through continuous trial and error and reward feedback, the RL model learns to automatically find the optimal balance between cost and benefit in complex and dynamic environments, achieving continuous optimization of governance strategies.

[0040] Based on a temporal anomaly detection model, the distribution of the multimodal feature vectors is continuously monitored to predict and identify data drift phenomena. For example, the system deploys a temporal anomaly detection model (such as an LSTM-based autoencoder, Prophet model, etc.) to continuously monitor the statistical distribution of incoming standardized multimodal feature vectors. This model learns the normal patterns of historical data feature distributions and calculates the difference between new data features and historical baseline distributions in real time. When the difference exceeds a preset threshold, the system predicts and identifies that data drift (such as concept drift or covariate drift) has occurred.

[0041] By employing a diffusion model, synthetic data conforming to a target distribution is generated to address identified data imbalances or drift issues, and this data is then used to enhance AI model training. For example, once data drift or significant scarcity of data in a specific class (imbalanced sample distribution) is identified, the system initiates a generative augmentation process. Specifically, using the advanced generative AI technique of the diffusion model, the model is trained on a large amount of high-quality data, learning the intrinsic distribution of data features. When augmentation is needed, the diffusion model is driven to generate synthetic data (i.e., synthetic feature vectors) conforming to the target distribution (such as the distribution of minority class sample features to be supplemented, or the historical healthy distribution to be corrected back). The generated synthetic data is mixed with real data for AI model training augmentation. This not only effectively balances the sample distribution and alleviates class imbalance, but also "calibrates" the training set early on by injecting data conforming to a historically healthy distribution, delaying model performance degradation and buying time for proactive updates to the real dataset.

[0042] Through reinforcement learning, the system can automatically and continuously find optimal governance parameters, fundamentally solving the rigidity problem of static strategies. While ensuring data quality and security, it significantly reduces governance operation costs and improves overall efficiency. Real-time monitoring of feature distribution using a time-series model provides early warnings before data drift affects model performance. Combining diffusion models to generate high-quality synthetic data provides low-cost, high-efficiency data augmentation and distribution correction methods, enabling the AI ​​training dataset to dynamically maintain health and balance, greatly enhancing the robustness and sustainability of upstream data supply. Feeding governance effects (downstream model performance, cost) back to the decision engine forms an intelligent system capable of self-learning and self-improvement.

[0043] S50. For each set of standardized multimodal feature vectors and derived synthetic data generated, record the model version, parameters and input data hash value used in the generation, and generate a fully auditable compliance report.

[0044] Specifically, a version management mechanism is established to record the feature engineering model version used during generation, all processing parameters, and the cryptographic hash value of the input raw data for each set of standardized multimodal feature vectors and synthetic data generated. For example, the system establishes a strict feature version control mechanism, similar to a version control system for software code (such as Git), to manage each set of standardized multimodal feature vectors and generated synthetic data. Whenever a set of features or synthetic data is generated, the system automatically captures and persistently stores its complete generation context, forming a unique identifier for that data version. The recorded key metadata includes: Feature engineering model version: precisely recording the identifier, version number, and snapshot hash of each model used to generate the feature. All processing parameters: recording the precise values ​​of all configurable parameters in the data processing pipeline, including alignment thresholds, fusion weights, cleaning rule parameters, guiding prompts for synthetic data, and noise scheduling parameters. Cryptographic hash value of the input raw data: calculating the cryptographic hash value (e.g., SHA-256) of the original input data on which this feature vector depends. This hash value acts like a unique digital fingerprint of the data. Any slight change to the original data will cause a dramatic change in the hash value, thus establishing a strong correlation between the feature and the source data.

[0045] Based on recorded feature versions, parameters, and hash information, the system generates compliance audit reports that encompass the entire data, feature, model, and decision lineage. For example, based on recorded feature versions, parameters, and hash information, the system automatically constructs and dynamically maintains an extended, enhanced data lineage graph. This graph not only includes the processing flow lineage but also deeply integrates procedural metadata such as model versions and parameter configurations. Based on this enhanced graph, the system can automatically generate detailed compliance audit reports on demand (or periodically). The core of these reports is a clear display of the entire "data-feature-model-decision" lineage. This mechanism enables the system to support complete traceability from the final decision back to the original data. Auditors or developers can trace back layer by layer based on a model prediction result, a data bias case, or a security incident to accurately pinpoint the problematic stage (such as a specific model version, a failed cleansing rule, or a batch of raw data that was not properly anonymized), providing a solid technical foundation for problem diagnosis, model iteration, compliance verification, and liability determination.

[0046] By binding model versions, parameters, and data fingerprints, the "black box" of feature engineering is completely eliminated, enabling any feature to be accurately reproduced when needed, greatly improving the scientific rigor and collaborative efficiency of AI research and development. Audit reports generated based on cryptographic hashes and end-to-end lineage provide a machine-readable, human-understandable, and tamper-proof chain of evidence, significantly reducing compliance risks. By solidifying records of all preceding dynamic processing and intelligent decision-making, the entire system is not only intelligent during runtime but also completely transparent during post-event review. This achieves end-to-end trusted governance from data access to AI decision-making, extending data governance assurance capabilities from process control to outcome auditing.

[0047] This invention enables unified access, cleaning, parsing, feature engineering, version management, and compliance governance of multi-source heterogeneous data such as text, images, audio, and video. It employs machine learning models to detect and predict data anomalies in real time and dynamically adjust governance strategies, thereby solving the problems of poor real-time performance, weak adaptability, and severe data silos in traditional data governance applications for AI.

[0048] The technical effects that this invention can bring are as follows: (1) Realize real-time perception and intelligent understanding of full-domain data: Through a unified dynamic access layer and LLM-driven semantic parsing, the system can automatically access, understand and label massive heterogeneous multimodal data, solving the core bottleneck of rigid data access and lack of semantics in unstructured data in traditional governance.

[0049] (2) Constructing a high-quality, semantically aligned multimodal feature base: Through automated format normalization, cross-modal alignment based on contrastive learning, and deep feature extraction pipeline, the problem of "format fragmentation, semantic separation, and feature discreteness" of multimodal data is fundamentally solved, generating unified, high-quality, and fusionable standardized feature vectors, providing high-quality training input for upper-layer AI models.

[0050] (3) Establish a proactive governance mechanism with predictable risks and adaptive strategies: Utilize dynamic data lineage maps and graph neural networks to achieve early prediction and root cause localization of data failure risks; Combine with multimodal sensitive identification models to achieve a leap from static rule protection to dynamic and precise governance, which significantly improves governance efficiency and intelligence level while ensuring data security and quality.

[0051] (4) Forming a self-optimization and enhancement closed loop driven by effect feedback: The strategy parameters are dynamically optimized by the reinforcement learning model based on the downstream AI performance and governance costs, and the system actively combats data drift and sample imbalance by combining the time-series detection and diffusion model, so that the system has the ability to evolve and continuously optimize itself.

[0052] (5) Provide full-chain, tamper-proof traceability and compliance assurance: Through feature version control and cryptographic hash binding, it ensures that every link from raw data to AI decision-making is reproducible and auditable, generating a complete and credible evidence chain that runs through "data-feature-model-decision", providing key technical support for model compliance, responsibility definition and security audit.

[0053] Furthermore, such as Figure 3 As shown, based on the above-described multimodal data processing method, the present invention also provides a multimodal data processing system, wherein the multimodal data processing system includes: The data access module 51 is used to access and capture multimodal raw data streams from multiple heterogeneous data sources in real time, use a large language model to perform dynamic semantic parsing on the raw data streams, generate dynamic semantic context, summary and tags of data objects, and update the unified data catalog. The multimodal data fusion processing module 52 is used to perform format normalization and cross-modal alignment processing on the original data stream based on the dynamic semantic context, and extract and form a standardized multimodal feature vector; The data lineage tracking module 53 is used to construct a dynamic data map; based on the dynamic data map and the multimodal feature vector, it uses a graph neural network to predict the risk of data failure in real time, and uses a multimodal content recognition model to identify sensitive information. According to the predicted risk level and the identified sensitive content, it dynamically triggers corresponding data quality repair, security level adjustment, desensitization, encryption or access control strategies. The intelligent governance decision module 54 is used to collect policy execution effects and training feedback, continuously optimize governance policies using reinforcement learning models, monitor the distribution drift of the multimodal feature vectors based on the time-series anomaly detection model, and generate targeted synthetic data through a diffusion model to complete or correct the data distribution. The report generation module 55 is used to record the model version, parameters and input data hash value used in the generation of each set of standardized multimodal feature vectors and derived synthetic data, and generate a fully auditable compliance report.

[0054] Furthermore, such as Figure 4 As shown, based on the above-mentioned multimodal data processing method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 4 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0055] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a multimodal data processing program 40, which can be executed by the processor 10 to implement the multimodal data processing method of this application.

[0056] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the multimodal data processing method.

[0057] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The terminal's processor 10, memory 20, and display 30 communicate with each other via a system bus.

[0058] In one embodiment, when the processor 10 executes the multimodal data processing program 40 in the memory 20, it implements the steps of the multimodal data processing method as described above.

[0059] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal data processing program, which, when executed by a processor, implements the steps of the multimodal data processing method as described above.

[0060] In summary, this invention provides a multimodal data processing method, system, terminal, and computer-readable storage medium. The method includes: real-time access and capture of multimodal raw data streams from multiple heterogeneous data sources; dynamic semantic parsing of the raw data streams using a large language model to generate dynamic semantic context, summaries, and tags for data objects, and updating a unified data catalog; format normalization and cross-modal alignment of the raw data streams based on the dynamic semantic context to extract standardized multimodal feature vectors; construction of a dynamic data graph; and real-time prediction of data using a graph neural network based on the dynamic data graph and the multimodal feature vectors. This invention addresses the risk of data failure and utilizes a multimodal content recognition model to identify sensitive information. Based on the predicted risk level and the identified sensitive content, it dynamically triggers corresponding data quality repair, security level adjustment, desensitization, encryption, or access control strategies. It collects the effectiveness of strategy execution and training feedback, continuously optimizes governance strategies using a reinforcement learning model, monitors the distribution drift of the multimodal feature vectors based on a time-series anomaly detection model, and generates targeted synthetic data through a diffusion model to supplement or correct the data distribution. For each set of standardized multimodal feature vectors and derived synthetic data generated, it records the model version, parameters, and input data hash value used in generation, generating a fully auditable compliance report. This invention employs a machine learning model for real-time detection and prediction of data anomalies and dynamically adjusts governance strategies, achieving real-time, intelligent, and closed-loop governance of multimodal data, significantly improving the quality of AI training data and governance efficiency.

[0061] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0062] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0063] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multimodal data processing method, characterized in that, The multimodal data processing method includes: The system can access and capture multimodal raw data streams from multiple heterogeneous data sources in real time, perform dynamic semantic parsing on the raw data streams using a large language model, generate dynamic semantic context, summaries and tags for data objects, and update the unified data catalog. Based on the dynamic semantic context, the original data stream is normalized and cross-modal aligned to extract standardized multimodal feature vectors. A dynamic data graph is constructed. Based on the dynamic data graph and the multimodal feature vector, a graph neural network is used to predict the risk of data failure in real time. A multimodal content recognition model is used to identify sensitive information. According to the predicted risk level and the identified sensitive content, corresponding data quality repair, security level adjustment, desensitization, encryption or access control strategies are dynamically triggered. Collect policy execution effects and training feedback, continuously optimize governance policies using reinforcement learning models, monitor the distribution drift of the multimodal feature vectors based on a temporal anomaly detection model, and generate targeted synthetic data through a diffusion model to complete or correct the data distribution. For each set of standardized multimodal feature vectors and derived synthetic data generated, the model version, parameters, and input data hash values ​​used in the generation are recorded, and a fully auditable compliance report is generated.

2. The multimodal data processing method according to claim 1, characterized in that, The process involves real-time access and capture of multimodal raw data streams from multiple heterogeneous data sources, dynamic semantic parsing of the raw data streams using a large language model, generating dynamic semantic context, summaries, and tags for data objects, and updating the unified data catalog. Specifically, this includes: Through a configured unified access layer, multiple data sources, including databases, message queues, object storage, and IoT devices, can be accessed via APIs, database connections, SaaS application interfaces, or IoT protocols. This allows for the real-time capture of multimodal raw data streams from multiple heterogeneous data sources, including both structured and unstructured data. The multimodal raw data stream is dynamically semantically parsed using a large language model to understand the data content, structure, and context, generate data summaries, business tags, and technical tags, and continuously optimize the metadata description based on user feedback on the summaries and tags. The data objects, dynamically updated data definitions, structures, semantic contexts, summaries, and tags are registered or updated to a unified data directory in real time.

3. The multimodal data processing method according to claim 1, characterized in that, Based on the dynamic semantic context, the original data stream is subjected to format normalization and cross-modal alignment processing to extract standardized multimodal feature vectors, specifically including: The format normalization engine converts files from different sources in the original data stream into standardized formats and performs unified encoding on the text data. Through the cross-modal alignment module, a contrastive learning method is used to align text-image pairs and audio-video streams at the semantic level to ensure that data from different modalities express the same topic or concept. The normalized and aligned modal data are processed using a feature engineering pipeline to obtain the features extracted from each modality. By fusing the features extracted from each modality, a set of standardized multimodal feature vectors with a unified dimension is generated.

4. The multimodal data processing method according to claim 3, characterized in that, The feature engineering pipeline process is performed on the normalized and aligned modal data to obtain the features extracted from each modality, specifically including: Generate word vectors, sentence vectors, and Chain-of-Thought enhanced features from text data; Visual embedding features based on ResNet or ViT are extracted from image data and fused with text information extracted by OCR and object detection labels; For audio and video data, extract text features, keyframe visual features, and voiceprint features from ASR transcription.

5. The multimodal data processing method according to claim 1, characterized in that, The construction of a dynamic data graph involves using a graph neural network to predict data failure risks in real time based on the dynamic data graph and the multimodal feature vectors, and using a multimodal content recognition model to identify sensitive information. Based on the predicted risk level and the identified sensitive content, corresponding data quality repair, security level adjustment, desensitization, encryption, or access control strategies are dynamically triggered, specifically including: Construct an end-to-end data flow graph as a dynamic data map to record the entire lifecycle of data from collection, cleaning, parsing to feature generation; A graph neural network is used to analyze the dynamic data map, predict the risk of data failure, and locate the root cause. Based on a multimodal content recognition model, the multimodal feature vectors are scanned to automatically identify sensitive information, including faces, medical images, contract terms, and personal identification information. Based on the predicted data failure risk and the identified categories and levels of the sensitive information, the security level of the data object is dynamically adjusted, and corresponding data desensitization, encryption operations, or fine-grained access control policies are automatically triggered.

6. The multimodal data processing method according to claim 1, characterized in that, The collection strategy's execution effect and training feedback are used to continuously optimize the governance strategy using a reinforcement learning model. A temporal anomaly detection model is used to monitor the distribution drift of the multimodal feature vectors, and a diffusion model is used to generate targeted synthetic data to supplement or correct the data distribution. Specifically, this includes: An intelligent governance decision engine is built using a reinforcement learning model. The reinforcement learning model takes the overall cost of data governance and the performance of downstream AI models as optimization objectives. Through interaction with the environment, it dynamically optimizes the data sampling frequency, the trigger threshold of cleaning rules, and the access control strategy. Based on the time-series anomaly detection model, the distribution of the multimodal feature vectors is continuously monitored to predict and identify data drift phenomena. By using a diffusion model, synthetic data that conforms to the target distribution is generated to address identified data imbalances or drift issues, and this data is then used to enhance AI model training.

7. The multimodal data processing method according to claim 1, characterized in that, The process involves generating each set of standardized multimodal feature vectors and derived synthetic data, recording the model version, parameters, and input data hash values ​​used in the generation, and generating a fully auditable compliance report, specifically including: Establish a version management mechanism to record the feature engineering model version used during generation, all processing parameters, and the cryptographic hash value of the input raw data for each set of standardized multimodal feature vectors and synthetic data generated. Based on the recorded feature versions, parameters, and hash information, a compliance audit report is generated that includes the full-chain lineage of data, features, models, and decisions.

8. A multimodal data processing system, characterized in that, The multimodal data processing system includes: The data access module is used to access and capture multimodal raw data streams from multiple heterogeneous data sources in real time, perform dynamic semantic parsing of the raw data streams using a large language model, generate dynamic semantic context, summary and tags of data objects, and update the unified data catalog. The multimodal data fusion processing module is used to perform format normalization and cross-modal alignment processing on the original data stream based on the dynamic semantic context, and extract and form a standardized multimodal feature vector; The data lineage tracing module is used to construct a dynamic data graph; based on the dynamic data graph and the multimodal feature vector, it uses a graph neural network to predict the risk of data failure in real time, and uses a multimodal content recognition model to identify sensitive information. According to the predicted risk level and the identified sensitive content, it dynamically triggers corresponding data quality repair, security level adjustment, desensitization, encryption or access control strategies. The intelligent governance decision-making module is used to collect policy execution effects and training feedback, continuously optimize governance policies using reinforcement learning models, monitor the distribution drift of the multimodal feature vectors based on a time-series anomaly detection model, and generate targeted synthetic data through a diffusion model to complete or correct the data distribution. The report generation module is used to record the model version, parameters, and input data hash value used in the generation of each set of standardized multimodal feature vectors and derived synthetic data, and generate a fully auditable compliance report.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a multimodal data processing program stored in the memory and executable on the processor. When the multimodal data processing program is executed by the processor, it implements the steps of the multimodal data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multimodal data processing program, which, when executed by a processor, implements the steps of the multimodal data processing method as described in any one of claims 1-7.

Images