AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method

Abstract

This invention provides an AI-based multi-model collaborative intelligent screening and deep analysis method for multimodal data, comprising: differential preprocessing based on modal differences in multimodal samples, dynamically adjusting preprocessing parameters through sample metadata analysis; evaluating sample data quality using multi-dimensional indicators to obtain high-value data; using an importance and diversity coupled optimization strategy to screen high-value data to obtain effective samples; performing modal alignment on the screened samples to complete cross-modal feature fusion and generate a unified fused feature vector; performing multi-dimensional deep analysis on the fused feature vector to mine inter-modal correlation information, and combining domain transfer learning to achieve result generalization; evaluating the analysis results in real time based on preset indicators, and dynamically adjusting the core parameters of the entire process based on the evaluation results. The beneficial effects of this invention are: solving the problems of high screening cost, low accuracy, poor fusion effect, weak analytical capability, poor interpretability, and inflexible deployment in existing technologies.

Description

Technical Field

[0001] This invention belongs to the field of model selection and scheduling, and in particular relates to an AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method. Background Technology

[0002] With the rapid development of artificial intelligence and big data technologies, multimodal data (i.e., mixed data including text, images, audio, video, sensor data, etc.) has been widely used in many fields such as healthcare, autonomous driving, e-commerce, education, and security. For example, CT images and diagnostic text in the medical field, LiDAR point clouds and camera images in the autonomous driving field, product images and descriptive text in the e-commerce field, and video courses and lecture notes in the education field are all typical multimodal data scenarios.

[0003] Multimodal data contains rich information dimensions, and its value mining is of great significance for technological upgrades in various fields. However, it also faces many challenges: On the one hand, the scale of multimodal data is growing explosively. For example, multimodal datasets such as LLaVA-665K have reached hundreds of thousands of data points. Processing the entire dataset requires a large amount of computing and storage resources, resulting in low processing efficiency. On the other hand, multimodal data has problems such as strong heterogeneity, high noise levels, complex intermodal relationships, and inconsistent quality. For example, similar texts describe different images, audio data is noisy, and there is redundant numerical information in the text, which brings great difficulties to data screening and in-depth analysis.

[0004] Currently, intelligent screening and deep analysis of multimodal data has become a research hotspot in the field of artificial intelligence. Its core requirement is to rapidly filter high-value, high-quality data from massive and complex multimodal datasets, and to mine potential correlations and core information between modalities through deep analysis, providing reliable support for subsequent model training and decision support. However, existing technologies still have many shortcomings in terms of screening efficiency, screening accuracy, cross-modal alignment, deep analysis capabilities, interpretability, and deployment flexibility, making it difficult to meet the actual needs of various fields for efficient processing of multimodal data. Therefore, developing an efficient, accurate, and flexible intelligent screening and deep analysis method for multimodal input data has significant theoretical and practical application value. Summary of the Invention

[0005] In view of this, the present invention aims to propose an AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method, in order to solve at least one of the above-mentioned technical problems.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows: The first aspect of this invention proposes a method for intelligent screening and deep analysis of multimodal data based on AI multi-model collaborative processing, comprising: S1. Based on the modal differences of the samples, perform standardized preprocessing of the samples in a differentiated manner, and dynamically adjust the preprocessing parameters by analyzing the sample metadata; S2. Use multi-dimensional indicators to comprehensively evaluate the data quality of the sample to obtain high-value data; S3. Use an importance and diversity coupling optimization strategy to screen high-value data to obtain screened data; S4. Perform modal alignment on the samples after filtering the data, and then perform cross-modal fusion on the aligned samples to obtain the fused feature vector; S5. Perform multi-dimensional analysis on the fused feature vectors to obtain the correlation information between samples of each modality, and combine domain transfer learning to generalize the correlation information; S6. Evaluate the related information in real time based on preset indicators, and provide optimization instructions based on the evaluation results to adjust the parameters in steps S1 to S5.

[0007] Furthermore, the standardization preprocessing execution process includes: Extract the core metadata of each modality sample and perform differentiated preprocessing based on its inherent characteristics, then convert the preprocessed data into a standard format; The sample modalities include text, image, audio, video, and sensor time-series data; the differentiated operations include text deduplication and word segmentation, image denoising and normalization, audio filtering and silent segment removal, and sensor data drift removal and outlier removal; the preprocessing parameters include filter strength, deduplication threshold, normalization interval, and sampling rate parameter.

[0008] Furthermore, the multi-dimensional indicators include: Data importance is quantified by the semantic relevance of the data to the target task; Diversity is calculated using the dispersion and information entropy of the data feature distribution; Noise level, which is quantified by noise percentage and signal-to-noise ratio; Modal alignment is calculated by combining the cosine similarity of features between modalities with the semantic matching degree. Metadata integrity, which is calculated by the missing rate of metadata fields; Using the analytic hierarchy process and entropy weighting, weights are adaptively assigned to each indicator to quantify data quality. Based on the quantification results, the data is divided into three value levels: high, medium, and low. The process involves outputting high-value data in step S3, returning medium-value data to step S1 for secondary preprocessing, and deleting low-value data.

[0009] Furthermore, the working process of step S3 specifically includes: The global optimal solution is obtained through a combination of few-shot learning and gradient descent algorithm. Using reinforcement learning algorithms, the selection threshold is dynamically adjusted by combining the target task requirements, data distribution, and optimization instructions. Based on the comprehensive evaluation results of data quality in step S2 and the coupling optimization strategy, the high-value data output in step S2 is screened. During the screening process, secondary filtering of residual noise is performed simultaneously. The residual noise includes redundant text numbers, inter-modal mismatch information, and low-confidence feature fragments.

[0010] Furthermore, the working process of step S4 specifically includes: Extract the intermediate layer depth features of each modality screening data, and use a cross-attention mechanism and a fine-grained bipartite graph matching algorithm to align the intermediate layer depth features of each modality sample; Based on modal quality and task requirements, corresponding fusion weights are assigned to samples of each modality, and cross-modal intermediate layer deep feature fusion is performed based on the fusion architecture of graph neural network and Transformer.

[0011] Furthermore, during the cross-modal fusion process, a triple loss joint optimization strategy is used to simultaneously optimize the output layer classification loss, the intermediate layer fine-grained alignment loss, and the global consistency loss of the fused features.

[0012] Furthermore, the multidimensional analysis includes: Semantic dimension, which analyzes the deep semantic understanding of text, image, and audio modalities; Structural dimension, which analyzes the spatial structure and time series structure of the data; The association dimension analyzes the causal relationships and correlation mining between modalities and features.

[0013] Furthermore, the preset indicators include accuracy, recall, F1 score, processing latency, and resource consumption; The parameters adjusted according to the feedback optimization instructions include preprocessing parameters, evaluation index weights, screening thresholds, fusion weights, and deep analysis model parameters. Furthermore, during the execution of steps S1 to S6, key information of each step is recorded based on blockchain evidence storage technology to generate a processing log. The key information includes original input data information, processing parameters of each step, data change process, quality assessment results, screening decision basis, fusion strategy, analysis process, parameter optimization records, and result output information.

[0014] The second aspect of this invention proposes an intelligent screening and deep analysis system for multimodal data, used to implement the AI-based multi-model collaborative intelligent screening and deep analysis method for multimodal data described in the first aspect: The system includes a multimodal data input module, a standardized dynamic preprocessing module, a multidimensional quality assessment module, a coupled optimization intelligent screening module, a cross-modal high-precision fusion module, a multidimensional deep analysis module, a real-time evaluation and dynamic optimization module, and a result output and traceability module, which are connected sequentially through a forward data transmission link. The real-time evaluation and dynamic optimization module communicates with the standardized dynamic preprocessing module, the multi-dimensional quality evaluation module, the coupled optimization intelligent screening module, the cross-modal high-precision fusion module, and the multi-dimensional deep analysis module through a reverse feedback link. The multi-dimensional quality assessment module communicates with the standardized dynamic preprocessing module through a reverse feedback link to perform a dual closed-loop process of data diversion and parameter optimization.

[0015] Compared with existing technologies, the AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method described in this invention has the following advantages: 1. This invention achieves synergistic optimization of importance and diversity through a coupling optimization strategy, eliminating the need for full data to participate in model backpropagation, reducing screening costs by more than 60% compared to existing technologies. It introduces multi-dimensional quality assessment and secondary filtering methods such as text masking and fine-grained noise detection, resulting in higher quality data that is more suitable for the target task.

[0016] 2. This invention adopts an intermediate layer feature fusion and triple loss optimization strategy, combined with cross-attention mechanism and fine-grained alignment algorithm, which improves the modality alignment accuracy by more than 25% compared with the existing technology. It adopts a dynamic weight allocation algorithm to dynamically adjust the weights according to modality quality and task requirements, fully explore the potential correlation between modalities, and effectively avoid feature mismatch and information loss.

[0017] 3. This invention constructs a deep analysis system with semantic, structural, and relational dimensions, which can fully explore the core value of multimodal data. By combining domain transfer learning and lightweight model design, the generalization ability of the analysis model in different domains, small sample, and noisy data scenarios is significantly improved.

[0018] 4. Based on blockchain and data log technology, this invention enables full traceability of the analysis process, clearly presents the analysis process, feature correlation logic and decision basis, and facilitates problem location and accountability.

[0019] 5. This invention constructs a standardized and dynamically adjustable preprocessing system, adopts differentiated preprocessing strategies for different modal data, dynamically adjusts parameters according to data quality and noise type, and makes full use of metadata to improve preprocessing efficiency. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram illustrating the workflow of the AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method described in an embodiment of the present invention. Figure 2 This is a thermodynamic diagram comparing the performance of each model described in the embodiments of the present invention; Figure 3 This is a schematic diagram of the radar comparison of various models described in the embodiments of the present invention. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0022] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0023] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0024] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] like Figure 1 As shown, the AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method includes: S1. Based on the modal differences of the samples, perform standardized preprocessing of the samples in a differentiated manner, and dynamically adjust the preprocessing parameters by analyzing the sample metadata; S2. Use multi-dimensional indicators to comprehensively evaluate the data quality of the sample to obtain high-value data; S3. Use an importance and diversity coupling optimization strategy to screen high-value data to obtain screened data; S4. Perform modal alignment on the samples after filtering the data, and then perform cross-modal fusion on the aligned samples to obtain the fused feature vector; S5. Perform multi-dimensional analysis on the fused feature vectors to obtain the correlation information between samples of each modality, and combine domain transfer learning to generalize the correlation information; S6. Evaluate the related information in real time based on preset indicators, and provide optimization instructions based on the evaluation results to adjust the parameters in steps S1 to S5.

[0026] The execution process of the standardized preprocessing includes: Extract the core metadata of each modality sample and perform differentiated preprocessing based on its inherent characteristics, then convert the preprocessed data into a standard format; The sample modalities include text, image, audio, video, and sensor time-series data; the differentiated operations include text deduplication and word segmentation, image denoising and normalization, audio filtering and silent segment removal, and sensor data drift removal and outlier removal; the preprocessing parameters include filter strength, deduplication threshold, normalization interval, and sampling rate parameter.

[0027] The multi-dimensional indicators include: Data importance is quantified by the semantic relevance of the data to the target task; Diversity is calculated using the dispersion and information entropy of the data feature distribution; Noise level, which is quantified by noise percentage and signal-to-noise ratio; Modal alignment is calculated by combining the cosine similarity of features between modalities with the semantic matching degree. Metadata integrity, which is calculated by the missing rate of metadata fields; Using the analytic hierarchy process and entropy weighting, weights are adaptively assigned to each indicator to quantify data quality. Based on the quantification results, the data is divided into three value levels: high, medium, and low. The process involves outputting high-value data in step S3, returning medium-value data to step S1 for secondary preprocessing, and deleting low-value data.

[0028] The working process of step S3 specifically includes: The global optimal solution is obtained through a combination of few-shot learning and gradient descent algorithm. Using reinforcement learning algorithms, the selection threshold is dynamically adjusted by combining the target task requirements, data distribution, and optimization instructions. Based on the comprehensive evaluation results of data quality in step S2 and the coupling optimization strategy, the high-value data output in step S2 is screened. During the screening process, secondary filtering of residual noise is performed simultaneously. The residual noise includes redundant text numbers, inter-modal mismatch information, and low-confidence feature fragments.

[0029] In some embodiments, the coupling optimization strategy is specifically as follows: This approach treats data importance and data diversity as inseparable joint optimization objectives, iterating synchronously, constraining each other, and improving together within the same optimization framework, rather than processing them independently in stages. This avoids the decoupling that could lead to decreased screening accuracy, data redundancy, or loss of key information. In other words, the coupled optimization strategy includes two parts: synchronous constraints and joint iterative optimization. The synchronization constraints are as follows: If the importance is insufficient, it will be directly excluded; if the diversity is insufficient, duplicate samples will be forcibly removed; if the noise level exceeds the limit, it will be automatically filtered. The joint iterative optimization is as follows: The screening distribution is initialized by learning from a small sample, and the screening threshold is adjusted in real time through reinforcement learning. This allows high importance and high diversity to be achieved simultaneously in the same round of screening without having to traverse the entire dataset twice, thus achieving low computational overhead and high screening accuracy.

[0030] The above methods achieve a three-pronged screening effect of "importance-driven value preservation, diversity-driven overfitting prevention, and coupling constraints to prevent conflict," fundamentally solving the shortcomings of existing technologies such as "prioritizing importance over diversity, fragmented objectives, and ineffective trade-offs."

[0031] The coupling optimization strategy is the core screening mechanism proposed in this invention. It refers to an integrated optimization method that incorporates data importance and data diversity into a unified optimization framework, and performs joint constraints, synchronous iteration, and collaborative optimization to avoid target conflicts, trade-off failures, and accuracy reduction caused by processing them in stages.

[0032] This invention constructs an importance-diversity coupled objective function, initializes the distribution with small sample learning, and dynamically adjusts the screening threshold with reinforcement learning. Without backpropagation of the entire dataset, it achieves the synchronous output of high-value, highly diverse, and low-noise data, significantly reducing computational overhead and improving screening accuracy.

[0033] The specific working process of step S4 includes: Extract the intermediate layer depth features of each modality screening data, and use a cross-attention mechanism and a fine-grained bipartite graph matching algorithm to align the intermediate layer depth features of each modality sample; Based on modal quality and task requirements, corresponding fusion weights are assigned to samples of each modality, and cross-modal intermediate layer deep feature fusion is performed based on the fusion architecture of graph neural network and Transformer.

[0034] In some embodiments, modal quality refers to the overall usability of a modal data (text / image / audio / video) in terms of information integrity, noise ratio, clarity, semantic validity, intramodal consistency, etc., and is used to determine whether the modality is suitable for subsequent fusion and analysis.

[0035] The quantification dimensions of modal quality include: Text quality: redundancy, misspelling rate, semantic completeness, and percentage of numerical noise; Image quality: sharpness, blur, signal-to-noise ratio, effective area ratio; Audio quality: signal-to-noise ratio, noise level, and sentence segmentation completeness; Modal alignment quality: the degree of matching with other modalities (text-image matching degree, audio-visual synchronization degree).

[0036] In the cross-modal high-precision fusion module, higher-quality modalities are assigned higher feature weights, while low-quality modalities are automatically weighted down or partially masked, thereby improving the effectiveness and robustness of the overall fused features.

[0037] Modal quality is used to characterize the usability of single-modal data, including information completeness, clarity, noise ratio, semantic validity, intramodal consistency, and intermodal matching.

[0038] Text quality is reflected in redundancy, semantic integrity, and the proportion of digital noise; image quality is reflected in sharpness, signal-to-noise ratio, and the proportion of effective area; audio quality is reflected in signal-to-noise ratio and sentence segmentation integrity.

[0039] Modal quality directly determines the weight allocation ratio in the cross-modal fusion stage. Higher quality modalities are assigned higher feature weights, thereby improving the robustness of fused features.

[0040] In some embodiments, the task requirements refer to the specific application goals that the current system is targeting, including three categories: high-precision tasks, high-recall tasks, and real-time tasks.

[0041] High-precision missions prioritize importance and low noise. High recall missions emphasize diversity and coverage; Real-time tasks emphasize lightweight design and low latency.

[0042] The system adaptively adjusts the preprocessing intensity, filtering threshold, fusion weight, and analysis model complexity according to task requirements to achieve optimal matching between strategy and task.

[0043] During cross-modal fusion, a triple loss joint optimization strategy is used to simultaneously optimize the output layer classification loss, the intermediate layer fine-grained alignment loss, and the global consistency loss of the fused features.

[0044] The multidimensional analysis includes: Semantic dimension, which analyzes the deep semantic understanding of text, image, and audio modalities; Structural dimension, which analyzes the spatial structure and time series structure of the data; The association dimension analyzes the causal relationships and correlation mining between modalities and features.

[0045] The preset metrics include accuracy, recall, F1 score, processing latency, and resource consumption. The parameters adjusted according to the feedback optimization instructions include preprocessing parameters, evaluation index weights, screening thresholds, fusion weights, and deep analysis model parameters. During the execution of steps S1 to S6, key information of each step is recorded based on blockchain evidence storage technology to generate a processing log. The key information includes original input data information, processing parameters of each step, data change process, quality assessment results, screening decision basis, fusion strategy, analysis process, parameter optimization record, and result output information.

[0046] In this embodiment, the present invention proposes an intelligent screening and deep analysis system for multimodal data, used to realize the above-mentioned AI-based multi-model collaborative intelligent screening and deep analysis method for multimodal data: The system includes a multimodal data input module, a standardized dynamic preprocessing module, a multidimensional quality assessment module, a coupled optimization intelligent screening module, a cross-modal high-precision fusion module, a multidimensional deep analysis module, a real-time evaluation and dynamic optimization module, and a result output and traceability module, which are connected sequentially through a forward data transmission link. The real-time evaluation and dynamic optimization module communicates with the standardized dynamic preprocessing module, the multi-dimensional quality evaluation module, the coupled optimization intelligent screening module, the cross-modal high-precision fusion module, and the multi-dimensional deep analysis module through a reverse feedback link. The multi-dimensional quality assessment module communicates with the standardized dynamic preprocessing module through a reverse feedback link to perform a dual closed-loop process of data diversion and parameter optimization.

[0047] In some embodiments, the intelligent screening and deep analysis system for multimodal data described above consists of eight core functional modules. Each module independently performs a specific function, while they are interconnected through data links and feedback links to form a complete technical system. The specific structure of each module is as follows: Multimodal data input module: As a data entry point, it is used to collect various multimodal input data, supporting the input of multiple modal data such as text, images, audio, video, and sensor data. It includes three sub-units: a data acquisition unit, a data format recognition unit, and a data temporary storage unit. The data acquisition unit supports multi-source data acquisition (local files, network interfaces, sensors, etc.); the data format recognition unit automatically identifies the modality type and format of the input data and generates modality identifiers and format information; the data temporary storage unit adopts a distributed storage method to temporarily store the input data to avoid data loss, while also supporting fast data retrieval.

[0048] Standardized dynamic preprocessing module: It is used to perform targeted preprocessing on input multimodal data, remove noise and redundant information, unify data format, and lay the foundation for subsequent screening and analysis. It includes four sub-units: metadata extraction unit, modality-targeted preprocessing unit, standardization unit, and dynamic adjustment unit. The metadata extraction unit extracts the core metadata of each modality (such as image resolution, text keywords, audio sampling rate, etc.); the modality-specific preprocessing unit adopts differentiated preprocessing strategies for different modalities; the standardization unit unifies the preprocessed modalities into a preset format; and the dynamic adjustment unit dynamically adjusts the preprocessing parameters based on data quality and noise type.

[0049] Multi-dimensional quality assessment module: It is used to conduct a comprehensive quality assessment of preprocessed multimodal data and provide a basis for intelligent screening. It includes three sub-units: assessment index construction unit, index quantification calculation unit, and quality level classification unit. The evaluation index construction unit constructs a multi-dimensional evaluation index system that includes importance, diversity, noise level, modal alignment, and metadata integrity. The index quantification calculation unit quantifies each evaluation index to obtain specific evaluation values. The quality level classification unit classifies the data into three levels: high value, medium value, and low value based on the evaluation values. High-value data enters the subsequent screening stage, medium-value data undergoes secondary preprocessing, and low-value data is directly removed.

[0050] Coupling Optimization Intelligent Filtering Module: As the core filtering module, it is used to filter out high-value data from preprocessed multimodal data. It includes four sub-units: coupling optimization unit, noise secondary filtering unit, dynamic adjustment of filtering threshold unit, and data filtering unit. The coupling optimization unit optimizes the coupling of data importance and diversity, avoiding decoupling between the two; the noise secondary filtering unit performs secondary filtering on residual noise (such as redundant text numbers and intermodal mismatch information) found during the screening process; the screening threshold dynamic adjustment unit dynamically adjusts the screening threshold according to task requirements and data distribution; and the data screening unit selects high-value data based on the quality assessment results and the coupling optimization strategy.

[0051] Cross-modal high-precision fusion module: It is used to achieve accurate alignment and efficient fusion of high-value multimodal data and to explore potential correlations between modalities. It includes four sub-units: intermediate layer feature extraction unit, cross-modal alignment unit, dynamic weight allocation unit, and feature fusion unit. The intermediate layer feature extraction unit extracts intermediate layer features from each modality data to improve feature robustness; the cross-modal alignment unit uses a cross-attention mechanism and a fine-grained matching algorithm to achieve accurate alignment between modalities; the dynamic weight allocation unit dynamically allocates the weights of each modality according to modality quality and task requirements; and the feature fusion unit uses a combination of graph neural networks and Transformers to achieve efficient feature fusion and generate a fused feature vector.

[0052] Multi-dimensional in-depth analysis module: It is used to perform in-depth, multi-dimensional analysis on the fused feature vectors to extract core information. It includes four sub-units: feature mining unit, multi-dimensional analysis unit, domain adaptation unit, and preliminary result generation unit. The feature mining unit uses deep learning algorithms to mine potential associations and core features in the fused features; the multi-dimensional analysis unit conducts a comprehensive analysis from three dimensions: semantics, structure, and association; the domain adaptation unit adjusts the analysis strategy and parameters according to the specific application domain; and the preliminary result generation unit generates preliminary analysis results, including feature reports and decision suggestions.

[0053] Real-time evaluation and dynamic optimization module: It is used to evaluate the analysis results in real time and dynamically optimize the parameters of each preceding module to form a closed-loop optimization. It includes three sub-units: evaluation unit, feedback unit, and parameter optimization unit. The evaluation unit uses metrics such as accuracy, recall, and efficiency to evaluate the analysis results in real time; the feedback unit converts the evaluation results into optimization instructions and feeds them back to the preceding modules; the parameter optimization unit dynamically adjusts preprocessing parameters, evaluation metric weights, screening thresholds, fusion weights, and analysis parameters according to the optimization instructions.

[0054] Results output and traceability module: It is used to output analysis results and provide traceability of the analysis process, and includes three sub-units: result processing unit, result output unit, and process traceability unit. The results processing unit organizes the preliminary analysis results into a standardized report; the results output unit supports multiple formats of results output (text, tables, visualization charts, etc.); and the process traceability unit records all parameters, data changes, and decision-making basis throughout the entire processing flow, enabling traceability of the analysis process.

[0055] In other embodiments, the core working principle of the above eight core functional modules is based on theories such as multimodal data processing, deep learning, coupling optimization, and cross-modal fusion. By addressing the shortcomings of existing technologies, it achieves multi-dimensional, efficient, and accurate intelligent screening and deep analysis. The principles of each core component are as follows: Based on the heterogeneity of multimodal data, this paper adopts the preprocessing principle of "modal differentiation + dynamic adjustment". Differentiated preprocessing algorithms are designed for the characteristics of different modal data (such as text redundancy, image noise characteristics, and audio interference characteristics). At the same time, data quality and noise type are analyzed through metadata, and preprocessing parameters (such as filter strength and threshold size) are dynamically adjusted to ensure the relevance and effectiveness of the preprocessing effect. In addition, based on ontology modeling, a unified metadata architecture is constructed to achieve the integration of structured and unstructured metadata, providing support for subsequent screening and analysis.

[0056] Based on the value characteristics of multimodal data, a multi-dimensional evaluation system is constructed, comprising "importance, diversity, noise level, modal alignment, and metadata integrity." Importance is quantified by the relevance of data to the target task; diversity is calculated by the dispersion of data distribution; noise level is quantified by the noise ratio and signal strength ratio; modal alignment is calculated by the similarity of features between modalities (such as cosine similarity); and metadata integrity is calculated by the metadata missing rate. The Analytic Hierarchy Process (AHP) is used to determine the weight of each evaluation indicator, thereby achieving a comprehensive and objective evaluation of data quality.

[0057] Based on coupled optimization theory, this approach breaks the decoupling dilemma between importance and diversity by treating them as a whole optimization objective. A coupled optimization function is constructed, and the optimal solution for the optimization objective is found through few-shot learning and gradient descent algorithms, achieving synergistic optimization of importance and diversity. Simultaneously, algorithms such as text masking and noise detection are combined to perform secondary filtering on residual noise, further improving the screening accuracy. Through reinforcement learning algorithms, the screening threshold is dynamically adjusted according to data distribution and task requirements to ensure the adaptability of the screening results.

[0058] Based on the intermediate layer feature fusion theory, intermediate layer features (rather than output layer features) of each modality data are extracted to avoid the problem of output layer features being susceptible to noise interference and improve feature robustness. A cross-attention mechanism is adopted to achieve accurate matching of intermediate layer features of different modalities. Combined with fine-grained alignment algorithms (such as bipartite graph matching algorithms), the modality alignment accuracy is improved. Based on graph neural network theory, potential correlations between modalities are mined to construct a modality correlation graph. Combined with a dynamic weight allocation algorithm (dynamically adjusting weights according to modality quality and task requirements), efficient feature fusion is achieved to generate a fused feature vector with high correlation and high effectiveness. At the same time, triple loss optimization (output layer loss, intermediate layer fine-grained loss, and fused feature loss) is adopted to further improve the fusion effect.

[0059] Based on deep learning and feature mining theory, this study employs algorithms such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Transformer to perform in-depth feature mining on fused feature vectors, extracting potential correlations and core information between modalities. A comprehensive analysis is conducted from three dimensions: semantics (e.g., text semantics, image semantics), structure (e.g., spatial structure, temporal structure of data), and correlation (e.g., causal and correlational relationships between modalities), ensuring the comprehensiveness of the analysis results. Furthermore, by combining domain transfer learning theory and fine-tuning within specific domains, the study enhances the adaptability of the analysis model across different domains, addressing the generalization problem in small sample scenarios.

[0060] Based on closed-loop control theory, a closed-loop mechanism of "evaluation-feedback-optimization" is constructed. The analysis results are evaluated in real time using indicators such as accuracy, recall, processing efficiency, and resource consumption. The evaluation results are compared with preset thresholds. If the preset standards are not met, optimization instructions are generated and fed back to the preceding modules to dynamically adjust relevant parameters (such as the filtering strength of preprocessing, the weight of evaluation indicators, the screening threshold, and the fusion weight). A reinforcement learning algorithm is used to continuously optimize the parameter configuration, ensuring that the model performance continues to improve and achieving adaptive optimization of the entire system.

[0061] Based on data logs and blockchain technology, all key information in the entire processing flow is recorded, including the original information of the input data, the processing parameters of each module, the data change process, the quality assessment results, the screening criteria, the fusion strategy, the analysis process, the optimization records, etc., forming an immutable processing log, realizing the traceability of the analysis process, and facilitating problem location and accountability.

[0062] In other embodiments, the action relationships between the above eight core functional modules follow the principle of "sequential execution + closed-loop feedback." The actions of each module coordinate with each other to ensure the smooth operation of the entire technical solution. The specific action relationships are as follows: The multimodal data input module first performs data acquisition, collecting multi-source multimodal data through the data acquisition unit. The data format recognition unit identifies the modality type and format of the data and generates modality identifiers and format information. The data temporary storage unit temporarily stores the input data. Subsequently, the data input module performs data transmission, transmitting the acquired multimodal data to the standardized dynamic preprocessing module.

[0063] After receiving the data, the standardized dynamic preprocessing module first performs metadata extraction to extract the core metadata of each modality. Then, the modality-specific preprocessing unit performs differentiated preprocessing actions (such as text deduplication, image denoising, and audio filtering) according to the modality type. The standardization unit performs format standardization to unify the preprocessed modality data into a preset format. The dynamic adjustment unit adjusts the preprocessing parameters according to the data quality and noise type to ensure the preprocessing effect. After preprocessing is completed, the data transmission action is performed to transmit the preprocessed data to the multi-dimensional quality assessment module.

[0064] After receiving the preprocessed data, the multi-dimensional quality assessment module performs the following actions: the assessment index construction unit initializes the assessment indexes to determine the multi-dimensional assessment indicators and their weights; the index quantification calculation unit calculates the indicators to obtain specific assessment values; the quality level classification unit classifies the data into high, medium, and low value levels based on the assessment values; after the assessment is completed, the data is distributed to the coupled optimization intelligent filtering module, medium-value data is returned to the standardized dynamic preprocessing module for secondary preprocessing, and low-value data is removed.

[0065] After receiving high-value data, the coupling optimization intelligent screening module performs coupling optimization actions, constructs a coupling optimization function, and solves for the optimal solution for importance and diversity. The noise secondary filtering unit performs noise filtering actions to remove residual noise. The screening threshold dynamic adjustment unit performs threshold adjustment actions to adjust the screening threshold according to task requirements and data distribution. The data screening unit performs screening actions to screen high-value data based on quality assessment results and coupling optimization strategies. After screening is completed, the data transmission action is performed to transmit the screened high-value data to the cross-modal high-precision fusion module.

[0066] After receiving the filtered high-value data, the cross-modal high-precision fusion module performs the following steps: the intermediate layer feature extraction unit extracts intermediate layer features from each modality; the cross-modal alignment unit performs alignment using a cross-attention mechanism and fine-grained matching algorithm to achieve accurate alignment between modalities; the dynamic weight allocation unit dynamically allocates weights to each modality based on modality quality and task requirements; the feature fusion unit performs fusion using a combination of graph neural networks and Transformers to generate fused feature vectors; and after fusion, data transmission is performed to transfer the fused feature vectors to the multi-dimensional deep analysis module.

[0067] After receiving the fused feature vector, the multi-dimensional deep analysis module performs feature mining to extract core features and potential associations. The multi-dimensional analysis unit performs multi-dimensional analysis to conduct a comprehensive analysis from three dimensions: semantics, structure, and association. The domain adaptation unit performs domain adaptation to adjust the analysis strategy and parameters according to the application domain. The preliminary result generation unit performs result generation to generate preliminary analysis results and decision suggestions. After the analysis is completed, the data transmission unit performs data transmission to transmit the preliminary analysis results to the real-time evaluation and dynamic optimization module.

[0068] After receiving the preliminary analysis results, the real-time evaluation and dynamic optimization module performs an evaluation action, using preset indicators to evaluate the analysis results in real time. The evaluation results are compared with preset thresholds. If the preset standards are met, a result transmission action is performed, transmitting the preliminary analysis results to the result output and traceability module. If the preset standards are not met, the feedback unit performs a feedback action, feeding back optimization instructions to the preceding modules (standardized dynamic preprocessing module, multi-dimensional quality evaluation module, coupled optimization intelligent screening module, cross-modal high-precision fusion module, and multi-dimensional deep analysis module). The parameter optimization unit performs parameter optimization actions, and each preceding module adjusts its own parameters according to the optimization instructions. After adjustment, the corresponding actions are re-executed until the analysis results reach the preset standards.

[0069] After receiving qualified analysis results, the results output and traceability module performs the results processing action to organize the analysis results into a standardized report; the results output unit performs the results output action to output the analysis results in multiple formats; the process traceability unit performs the traceability recording action to generate a processing log to achieve traceability of the analysis process; after the entire action flow is completed, the termination action is performed. If new multimodal data is input, the above action flow is repeated.

[0070] The actions of each module are coordinated and mutually restrictive. The quality of the standardized dynamic preprocessing module directly affects the effectiveness of quality assessment and screening; the results of the quality assessment module determine the screening strategy of the screening module; the results of the screening module determine the quality of the basic data for fusion and analysis; the effect of the fusion module directly affects the accuracy of deep analysis; the results of deep analysis determine the optimization direction of the optimization module; and the actions of the optimization module, in turn, improve the effects of the preceding modules, forming a complete collaborative closed loop to ensure the efficient and stable operation of the entire technical solution.

[0071] Current existing technologies include at least: a multimodal data analysis method (CN121682720A) and the COIDO multimodal data screening framework. The method described in this invention has stronger deep analysis capabilities and better generalization. Although the multimodal data analysis method achieves multimodal analysis, it lacks multi-dimensional and deep feature mining and has poor generalization. The COIDO multimodal data screening framework only focuses on screening and does not involve deep analysis.

[0072] While existing technologies like the COIDO framework achieve low-cost screening, they lack the integration of multi-dimensional quality assessment and secondary noise filtering, resulting in insufficient screening accuracy. Zhuowang Digital's solution fails to address the issues of high screening costs and target decoupling. This invention, through a coupling optimization strategy, achieves synergistic optimization of importance and diversity, eliminating the need for full data participation in model backpropagation and reducing screening costs by over 60% compared to existing technologies. Simultaneously, by introducing multi-dimensional quality assessment and secondary filtering methods such as text masking and fine-grained noise detection, screening accuracy is improved by over 30% compared to existing technologies, resulting in higher-quality data with better suitability for the target task.

[0073] Existing technologies mostly rely on output layer features for modality alignment, which is susceptible to noise interference and lacks flexibility in weight allocation, resulting in insufficient mining of potential correlations between modalities. This invention employs an intermediate layer feature fusion and triple loss optimization strategy, combined with a cross-attention mechanism and a fine-grained alignment algorithm, improving modality alignment accuracy by more than 25% compared to existing technologies. Simultaneously, a dynamic weight allocation algorithm is used to dynamically adjust weights based on modality quality and task requirements, fully mining potential correlations between modalities. The effectiveness of feature fusion is improved by more than 40% compared to existing technologies, effectively avoiding feature mismatch and information loss.

[0074] This invention constructs a deep analysis system encompassing semantics, structure, and association, which can fully explore the core value of multimodal data. At the same time, it combines domain transfer learning with lightweight model design. like Figure 2As shown, by comparing the performance of different data selection methods on various multimodal tasks, the model of the present invention has significantly improved the generalization ability in different fields, small sample and noisy data scenarios. The analysis accuracy on the small-scale PETS dataset is improved by more than 15% compared with the existing technology, and the R@1 is improved by more than 6% in the image and text retrieval task of the MS-COCO dataset, adapting to more complex scenarios.

[0075] Existing technologies are mostly "black box" analyses, lacking traceability of the analysis process and dynamic optimization mechanisms, making it difficult to continuously improve model performance. This invention, based on blockchain and data log technology, achieves full traceability of the analysis process, clearly presenting the analysis process, feature correlation logic, and decision-making basis, facilitating problem identification and accountability. Simultaneously, it constructs a closed-loop mechanism of "evaluation-feedback-optimization," dynamically optimizing the parameters of each preceding module based on the analysis results. Model performance continuously improves with data accumulation, and compared to existing static models, the analysis accuracy can be improved by more than 20% after long-term operation.

[0076] Existing preprocessing methods are fixed, lack standardization, cannot adapt to different types and qualities of multimodal data, and do not fully utilize metadata. This invention constructs a standardized, dynamically adjustable preprocessing system that employs differentiated preprocessing strategies for different modalities, dynamically adjusts parameters based on data quality and noise type, and fully utilizes metadata to improve preprocessing efficiency. The noise removal rate of the preprocessed data is improved by more than 35% compared to existing technologies, and the data standardization level reaches over 95%, providing a higher quality data foundation for subsequent screening and analysis.

[0077] The method described in this invention offers greater deployment flexibility and lower resource consumption. Existing technologies often rely on large computing clusters, which cannot adapt to resource-constrained scenarios such as terminals and edge devices, resulting in excessive resource consumption.

[0078] like Figure 3 As shown, a comparison of five key methods (FullData, Random, ICONS, COINCIDE, and COIDO) across five core tasks clearly demonstrates that this invention offers comprehensive lightweight optimization of the model structure and computational process, supporting deployment across multiple scenarios including cloud, terminal, and edge devices, significantly improving deployment flexibility. Simultaneously, it reduces computational resource consumption by over 50% compared to existing technologies and increases inference speed by over 45%, meeting real-time processing requirements and expanding the practical application scope of the technology. It is adaptable to resource-constrained scenarios such as medical terminals and autonomous driving edge devices.

[0079] Existing technologies for quality assessment often rely on single indicators, failing to comprehensively reflect data quality. This invention constructs a multi-dimensional quality assessment system covering five core dimensions: importance, diversity, noise level, modal alignment, and metadata integrity. This system can comprehensively and objectively assess the quality of multimodal data, providing a more scientific basis for intelligent screening. The targeting of screening is significantly improved compared to existing technologies, effectively avoiding the omission of high-value data and the redundancy of low-value data.

[0080] Existing technologies often focus on a single step (screening or analysis), failing to form a complete "screening-analysis-optimization" closed loop, thus limiting their practicality. This invention constructs a complete intelligent screening and deep analysis system for multimodal input data, covering the entire process of data input, preprocessing, screening, fusion, analysis, optimization, output, and traceability, forming a closed-loop operation. It can be directly applied to multiple fields such as healthcare, autonomous driving, e-commerce, and education, without requiring additional technical solutions, making it more practical, easier to implement, and possessing higher industrial application value.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent screening and deep analysis of multimodal data based on AI multi-model collaboration, characterized in that: include: S1. Based on the modal differences of the samples, perform standardized preprocessing of the samples in a differentiated manner, and dynamically adjust the preprocessing parameters by analyzing the sample metadata; S2. Use multi-dimensional indicators to comprehensively evaluate the data quality of the sample to obtain high-value data; S3. Use an importance and diversity coupling optimization strategy to screen high-value data to obtain screened data; S4. Perform modal alignment on the samples after filtering the data, and then perform cross-modal fusion on the aligned samples to obtain the fused feature vector; S5. Perform multi-dimensional analysis on the fused feature vectors to obtain the correlation information between samples of each modality, and combine domain transfer learning to generalize the correlation information; S6. Evaluate the related information in real time based on preset indicators, and provide optimization instructions based on the evaluation results to adjust the parameters in steps S1 to S5.

2. The AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method according to claim 1, characterized in that, The execution process of the standardized preprocessing includes: Extract the core metadata of each modality sample and perform differentiated preprocessing based on its inherent characteristics, then convert the preprocessed data into a standard format; The sample modality includes at least one of text, image, audio, video, or sensor time-series data; the differentially performed operations include at least one of text deduplication and word segmentation, image denoising and normalization, audio filtering and silent segment removal, and sensor data drift removal and outlier removal; the preprocessing parameters include at least one of filter strength, deduplication threshold, normalization interval, and sampling rate parameter.

3. The AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method according to claim 1, characterized in that, The multi-dimensional indicators include: Data importance is quantified by the semantic relevance of the data to the target task; Diversity is calculated using the dispersion and information entropy of the data feature distribution; Noise level, which is quantified by noise percentage and signal-to-noise ratio; Modal alignment is calculated by combining the cosine similarity of features between modalities with the semantic matching degree. Metadata integrity, which is calculated by the missing rate of metadata fields; Using the analytic hierarchy process and entropy weighting, weights are adaptively assigned to each indicator to quantify data quality. Based on the quantification results, the data is divided into three value levels: high, medium, and low. The process involves outputting high-value data in step S3, returning medium-value data to step S1 for secondary preprocessing, and deleting low-value data.

4. The AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method according to claim 1, characterized in that, The working process of step S3 specifically includes: The global optimal solution is obtained through a combination of few-shot learning and gradient descent algorithm. Using reinforcement learning algorithms, the selection threshold is dynamically adjusted by combining the target task requirements, data distribution, and optimization instructions. Based on the comprehensive evaluation results of data quality in step S2 and the coupling optimization strategy, the high-value data output in step S2 is screened. During the screening process, secondary filtering of residual noise is performed simultaneously. The residual noise includes redundant text numbers, inter-modal mismatch information, and low-confidence feature fragments.

5. The AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method according to claim 1, characterized in that, The specific working process of step S4 includes: Extract the intermediate layer depth features of each modality screening data, and use a cross-attention mechanism and a fine-grained bipartite graph matching algorithm to align the intermediate layer depth features of each modality sample; Based on modal quality and task requirements, corresponding fusion weights are assigned to samples of each modality, and cross-modal intermediate layer deep feature fusion is performed based on the fusion architecture of graph neural network and Transformer.

6. The AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method according to claim 5, characterized in that: During cross-modal fusion, a triple loss joint optimization strategy is used to simultaneously optimize the output layer classification loss, the intermediate layer fine-grained alignment loss, and the global consistency loss of the fused features.

7. The AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method according to claim 1, characterized in that, The multidimensional analysis includes: Semantic dimension, which analyzes the deep semantic understanding of text, image, and audio modalities; Structural dimension, which analyzes the spatial structure and time series structure of the data; The association dimension analyzes the causal relationships and correlation mining between modalities and features.

8. The AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method according to claim 1, characterized in that: The preset metrics include accuracy, recall, F1 score, processing latency, and resource consumption. The parameters adjusted according to the feedback optimization instructions include preprocessing parameters, evaluation index weights, screening thresholds, fusion weights, and deep analysis model parameters.

9. The AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method according to claim 1, characterized in that: During the execution of steps S1 to S6, key information of each step is recorded based on blockchain evidence storage technology to generate a processing log. The key information includes original input data information, processing parameters of each step, data change process, quality assessment results, screening decision basis, fusion strategy, analysis process, parameter optimization record, and result output information.

10. An AI-based multi-model collaborative multimodal data intelligent screening and deep analysis system, used to implement the AI-based multi-model collaborative multimodal data intelligent screening and deep analysis method as described in any one of claims 1-9, characterized in that: The system includes a multimodal data input module, a standardized dynamic preprocessing module, a multidimensional quality assessment module, a coupled optimization intelligent screening module, a cross-modal high-precision fusion module, a multidimensional deep analysis module, a real-time evaluation and dynamic optimization module, and a result output and traceability module, which are connected sequentially through a forward data transmission link. The real-time evaluation and dynamic optimization module communicates with the standardized dynamic preprocessing module, the multi-dimensional quality evaluation module, the coupled optimization intelligent screening module, the cross-modal high-precision fusion module, and the multi-dimensional deep analysis module through a reverse feedback link. The multi-dimensional quality assessment module communicates with the standardized dynamic preprocessing module through a reverse feedback link to perform a dual closed-loop process of data diversion and parameter optimization.

Images