Intelligent driving multi-modal data fusion method for data middle platform

Abstract

The application relates to an intelligent driving multi-modal data fusion method for a data middle station, and belongs to the technical field of data middle stations, multi-modal data processing, artificial intelligence resource scheduling and semantic fusion. The application comprises the following steps: multi-modal data standardization preprocessing; intelligent synchronous scheduling based on hierarchical reinforcement learning (HRL); explicit + implicit intelligent correlation matching; joint missing identification and cross-modal consistency completion; decoupling unified semantic space double-layer fusion driven by representation learning; verification output and full-link feedback optimization based on causal inference; and optimization parameters are fed back through causal inference to form a full-link closed-loop cooperation. The application constructs a "prediction scheduling-decoupling fusion-consistency completion-causal feedback" closed loop, wherein HRL is adaptively synchronized to avoid strategy shock, decoupling representation extraction shares semantics, joint missing completion guarantees cross-modal consistency, causal inference locates a root cause and realizes accurate optimization, and full-link intelligent cooperation is realized.

Description

Technical Field

[0001] This invention belongs to the fields of data middle platform, multimodal data processing, artificial intelligence resource scheduling and semantic fusion technology, and specifically relates to an intelligent-driven multimodal data fusion method for data middle platform. Background Technology

[0002] Enterprise data platforms typically integrate multimodal data, leading to technical challenges such as heterogeneous storage structures, rigid synchronization strategies, wasted computing resources, inconsistent timing, incorrect associations, inaccurate completion, and semantic misalignment. Traditional solutions rely on fixed-frequency synchronization, rule matching, default padding, and shallow feature concatenation, which struggle to achieve unified cross-modal computation, resulting in low resource utilization, high manual intervention requirements, and difficulty in supporting large-scale real-time business operations.

[0003] To address the aforementioned issues, existing patents and industry solutions have proposed methods from different perspectives, but they still have shortcomings in areas such as intelligent synchronization scheduling, joint missing completion, decoupled semantic fusion, and closed-loop causal feedback, which are the focus of this invention. Specifically:

[0004] Patent CN120781307A (Minimally Invasive Software) Dynamic heterogeneous encoder + cross-modal adversarial alignment + ontology network concept mapping The data synchronization and scheduling issues were not addressed; the fusion method did not distinguish between shared and unique features; and a closed-loop feedback mechanism was not provided. Patent CN120688537A (Suzhou Laplace) Agent role assignment and data flow rate control based on reinforcement learning (single-layer RL) Using a single-layer RL approach can easily lead to policy oscillations under burst traffic; it lacks predictive states, resulting in reactive scheduling; and it fails to address the multi-timescale optimization problem of synchronization strategies. Patent CN120724376A (University of Electronic Science and Technology of China) Search enhancement + diffusion generation achieve missing modality completion The "joint missing" scenario is not defined; each modality is generated independently, which may lead to cross-modal semantic contradictions; cross-modal consistency constraints are lacking. Patent CN121462608A (China Metallurgical Huatian) DRL models enable task-level resource scheduling between the edge and cloud. The scheduling granularity is at the task level rather than the data synchronization level; a single-layer DRL is used; decoupling and fusion, and causal feedback are not involved. Patent CN121542982A (Qingdao Xueshan Network) A two-stage fusion algorithm combining DQN and attention, based on LSTM for urgency prediction and resource reservation. Resource scheduling is at the task level (not the data synchronization level); the fusion method does not decouple shared and unique features; and the joint missing completion problem is not addressed. Patent CN121509246A (Zhejiang Laboratory) Meta-reinforcement learning enables adaptive bandwidth allocation in multimodal networks. The application scenario is network bandwidth allocation, and its technical solution is not directly applicable to the multimodal data synchronization and scheduling of the data platform; it does not involve data synchronization strategies, decoupled representation fusion, and joint missing data completion. Patent CN121350435A (Zhongshu Information) Feature distribution ellipse correction and missing data completion for global context retrieval The system failed to differentiate between scenarios with joint missing parts; the completion method primarily relied on retrieval and lacked generative consistency constraints. Patent CN121524948A (Henan University) Data augmentation methods for random missing modalities include generating hints, reverse hints, and optimal delivery (OT) guided hints. The scenario of missing joint functions is not defined; no cross-modal consistency determination mechanism is provided. Patent CN121350651A (Guizhou University) Cross-modal semantic alignment method based on fine-grained semantic decoupling Decoupling targets fine-grained semantics, rather than the separation of modality sharing and unique features; it is not coordinated with hierarchical scheduling and causal feedback. Industry-standard solutions (typical data processing solutions based on the Hadoop ecosystem) Shallow feature stitching + fixed frequency synchronization Unable to bridge the semantic gap; significant waste of computing power; lack of self-optimization capabilities.

[0005] In summary, the existing technologies and industry solutions mentioned above share the following shortcomings in addressing the core issues addressed by this invention:

[0006] 1. In terms of synchronization scheduling: synchronization strategies are either fixed (unable to adapt to traffic changes) or use single-layer RL for reactive adjustment (prone to strategy oscillation and lack of forward-looking prediction), making it difficult to cope with sudden traffic and maintain stability.

[0007] 2. Missing data completion: The special scenario of "joint missing data" was not distinguished. Each modality was generated independently or only single-modality completion was performed, resulting in semantic contradictions between the completed cross-modal data.

[0008] 3. In terms of semantic fusion: the shared features and unique features of the modalities were not explicitly decoupled, resulting in poor interpretability of the fusion results and the shared semantic learning being easily interfered with by unique information.

[0009] 4. Regarding closed-loop feedback: The lack of a closed-loop feedback mechanism, or the mere simple retries and fine-tuning of global parameters, makes it impossible to accurately pinpoint the root cause of fusion quality issues, resulting in low optimization efficiency.

[0010] It should be noted that even combining the aforementioned existing technologies (e.g., combining the single-layer RL scheduling of CN120688537A, the adversarial alignment fusion of CN120781307A, the generative completion of CN120724376A, and the parameter acquisition optimization of CN121542982A) still cannot solve the core problem of this invention. The reasons are: ① The combination of single-layer RL and LSTM prediction cannot handle multi-timescale conflicts in synchronous scheduling (millisecond-level bursts and minute-level planning cannot be simultaneously addressed); ② There is a lack of consistency constraints between independent generative completion and adversarial alignment fusion, resulting in a disconnect between the completion result and the fusion process; ③ Parameter acquisition optimization can only perform global parameter tuning and cannot distinguish the causal contributions of synchronization delay, correlation errors, and completion failures to the fusion quality.

[0011] To address the four aforementioned shortcomings, this invention proposes the following targeted improvements: ① hierarchical reinforcement learning predictive scheduling (solving synchronization policy oscillations); ② joint missing information identification and cross-modal consistency completion (solving semantic contradictions); ③ decoupled representation learning fusion (solving the lack of separation between shared and unique features); ④ closed-loop feedback for causal inference (solving root cause localization deficiencies). This invention achieves end-to-end intelligent self-optimization from data synchronization to semantic fusion through a three-layer synergy of hierarchical decision structure, joint latent space generation, and causal root cause localization. This synergistic effect cannot be achieved through the simple superposition of existing technologies. Summary of the Invention

[0012] (a) Technical problems to be solved

[0013] The technical problem to be solved by this invention is how to provide an intelligent-driven multimodal data fusion method for data platforms, so as to solve the problems of resource waste and timing inconsistency in multimodal data synchronization under high-frequency burst scenarios, as well as the cross-modal semantic completion distortion problem under joint missing scenarios.

[0014] (II) Technical Solution

[0015] To address the aforementioned technical problems, this invention proposes an intelligent-driven multimodal data fusion method oriented towards a data platform, which includes the following steps:

[0016] Step S1: Multimodal data standardization preprocessing

[0017] The multimodal data is formatted, denoised, and its basic features are extracted, and predictive state input is provided for the HRL high-level strategy in step S2.

[0018] Step S2: Intelligent Synchronization Scheduling Based on Hierarchical Reinforcement Learning (HRL)

[0019] A two-layer decision-making architecture is adopted: the high-level strategy determines the mode, and the low-level strategy outputs parameters. Predictive states and mode switching penalties are introduced to dynamically adapt the synchronization strategy, avoid strategy oscillation, ensure timing consistency, and reduce CPU and bandwidth consumption. Specifically, the high-level DQN decides the macro mode every 10 seconds, including: stable, bursty, and catch-up. The state includes the volume prediction value for the next minute provided in step S1, and the reward includes the mode switching penalty. The low-level PPO outputs the synchronization method, synchronization interval, and resource weights every second, based on the mode output by the high-level strategy. The two are trained together through an Option-Critic architecture to output a synchronized and aligned multimodal data stream.

[0020] Step S3: Intelligent Association Matching

[0021] The synchronized and aligned multimodal data stream output in step S2 achieves multimodal data association through a combination of explicit ID matching and implicit semantic matching.

[0022] Step S4: Joint missing item identification and cross-modal consistent completion

[0023] Define the "joint missing" scenario and achieve completion through joint latent space generation and consistency discriminator to ensure cross-modal semantic consistency and support both generative and retrieval completion strategies;

[0024] Step S5: Decoupling representation learning-driven two-layer fusion of unified semantic space

[0025] A unified semantic space is constructed, and shared features and unique features of different modalities are explicitly separated. Only the shared parts are fused to improve the fusion accuracy and interpretability. Each modality extracts 512-dimensional features through an independent encoder, which are then decomposed into shared semantic vectors and modality-specific vectors by a decoupling module. Shared and unique features are separated by minimizing mutual information, and shared semantics are aligned through cross-modal comparative learning.

[0026] Step S6: Verify the output and optimize the end-to-end feedback based on causal inference.

[0027] By accurately locating the root cause of fusion quality problems through causal graphs, a synergistic improvement of the entire chain of synchronization, correlation, completion and fusion is achieved. When the fusion confidence is lower than the threshold, the average causal effect ACE of each preceding node is estimated by dual machine learning, the node with the largest ACE is selected as the root cause, the parameters are adjusted in a targeted manner and written back to the corresponding module to form a closed loop.

[0028] The present invention also provides an intelligent-driven multimodal data fusion system for data middleware, which includes 8 layers, and the data flow and control flow interact between the layers through message queues and RPC calls;

[0029] Layer 1: Data source layer; stores multimodal raw data, pushes data to the preprocessing layer via Kafka, and receives processing confirmation signals;

[0030] Layer 2: Preprocessing and predictive feature extraction layer; performs format unification, noise reduction, and missing data labeling; deploys the LSTM prediction module to output the predicted data volume for the next minute; the standardized data and predicted values ​​are then passed to subsequent layers.

[0031] Layer 3: Intelligent Synchronization Scheduling Layer; dynamically adjusts synchronization interval and resource allocation weights based on hierarchical reinforcement learning (HRL); receives prediction values ​​from Layer 2 as input to higher-level states, outputs synchronized data streams to the association layer, and reports the synchronization status to the feedback optimization layer.

[0032] Layer 4: Association Matching Layer; performs explicit ID matching and implicit semantic matching; outputs association results and 512-dimensional semantic feature vectors of each modality to the missing feature completion layer;

[0033] Layer 5: Missing Completion Layer; Identifies joint missing data and uses C-VAE + consistency discriminator for generative or fallback retrieval-based completion; Outputs the completed data and optional latent space vectors to the semantic fusion layer;

[0034] Layer 6: Semantic fusion layer; performs decoupled representation learning, decomposes the features of each modality into shared semantic vectors and unique vectors, and outputs a unified semantic vector through two-layer fusion; receives the completed data and outputs the fusion result to the layer;

[0035] Layer 7: Verification output and feedback optimization layer; real-time calculation of fusion confidence, association error rate, and completion failure rate; when the fusion confidence is lower than the threshold, the causal inference module is triggered: using the most recent 1000 samples to estimate the average causal effect ACE of each preceding node, locate the root cause and generate a targeted adjustment signal; the adjustment signal is written back to layers 3-6; qualified fusion results are output to the service interface and storage; unqualified results can trigger a retry;

[0036] Layer 8: Monitoring and maintenance layer; collects resource usage, abnormal logs, and historical fusion effects of each layer in real time; provides data to Layer 7 for cause-effect graph updates; issues critical alarms and supports link switching.

[0037] (III) Beneficial Effects

[0038] This invention proposes an intelligent-driven multimodal data fusion method for data platforms. Addressing the challenges of wasted synchronization resources and joint missing data distortion in multimodal fusion within data platforms, this invention constructs a closed loop of "predictive scheduling - decoupled fusion - consistent completion - causal feedback." Specifically, HRL adaptive synchronization avoids policy oscillations, decoupled representations extract shared semantics, joint missing data completion ensures cross-modal consistency, and causal inference locates root causes and enables precise optimization, achieving intelligent collaboration across the entire chain. Attached Figure Description

[0039] Figure 1 This is the overall flowchart of the present invention. Detailed Implementation

[0040] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0041] This invention belongs to the fields of data middleware, multimodal data processing, artificial intelligence resource scheduling, and semantic fusion technology, specifically involving an intelligent fusion method and system. It aims to solve the problems of resource waste and temporal inconsistency in multimodal data synchronization under high-frequency, sudden data burst scenarios, as well as the challenge of cross-modal semantic completion distortion in scenarios with joint missing data.

[0042] (Problem scenario overview: A data platform needs to process different types of data simultaneously, such as tables, text, images, and audio. However, traditional methods either synchronize at a fixed frequency, wasting resources and easily causing chaos; or they fill in the missing data separately when multiple data are missing at the same time, resulting in contradictory results; and once the fusion quality is poor, it is difficult to pinpoint whether the problem is slow synchronization, incorrect association, or a problem with the completion. This invention aims to solve these problems of chaotic synchronization, contradictory completion, and difficulty in troubleshooting.)

[0043] Basic Concepts

[0044] 1. Data Platform: A unified technical infrastructure for enterprise-wide data access, storage, governance, integration, and services, used to eliminate data silos, unify data structures, and improve resource utilization efficiency.

[0045] 2. Multimodal data: includes heterogeneous data such as structured, semi-structured, text, image, and audio, with significant differences between the modalities in feature space, update frequency, and storage format.

[0046] 3. Multimodal data fusion: Transform heterogeneous multimodal data from different sources and with different features into data objects with unified representation, semantic alignment, and joint computation capability.

[0047] 4. Intelligent Driven: In this invention, it refers to a technical path that achieves fully automated scheduling and self-optimization through hierarchical reinforcement learning (HRL), multimodal decoupled representation learning, and adaptive feedback mechanisms based on causal inference as core technical means. Specifically, HRL is responsible for predictive resource scheduling and policy anti-oscillation; decoupled representation learning decomposes the features of each modality into shared semantics and unique features, supporting accurate semantic fusion; causal feedback quantifies the signal based on the fusion result, infers the root cause, and adjusts preceding parameters. These three elements work together to achieve intelligent self-optimization across the entire process.

[0048] 5. Joint missing data: refers to scenarios where multiple modalities have missing data at the same time, and there is a clear semantic relationship between the missing data (such as the simultaneous absence of the image and description of the target). These scenarios cannot be completed by a single modality and require joint completion by combining multimodal semantic features.

[0049] This invention addresses the challenges of wasted synchronization resources and distortion due to joint missing data in multimodal fusion of data platforms by constructing a closed loop of "predictive scheduling - decoupled fusion - consistent completion - causal feedback". Specifically, HRL adaptive synchronization avoids policy oscillations, decoupled representations extract shared semantics, joint missing data completion ensures cross-modal consistency, and causal inference locates root causes and enables precise optimization, achieving intelligent collaboration across the entire chain.

[0050] The present invention performs closed-loop processing in the following 6 steps:

[0051] Step S1: Multimodal data standardization preprocessing;

[0052] Step S2: Intelligent synchronization scheduling based on hierarchical reinforcement learning (HRL);

[0053] Step S3: Explicit + Implicit Intelligent Association Matching;

[0054] Step S4: Joint missing item identification and cross-modal consistent completion;

[0055] Step S5: Decouple the two-layer fusion of the unified semantic space driven by representation learning;

[0056] Step S6: Verify the output and optimize the end-to-end feedback based on causal inference;

[0057] In step S6, the optimization parameters are fed back to steps S2-S5 through causal inference, forming a closed-loop collaboration across the entire link.

[0058] Step S1: Multimodal data standardization preprocessing (including temporal prediction feature extraction)

[0059] The multimodal data undergoes format unification, denoising, and basic feature extraction, providing predictive state input for the HRL high-level strategy in step S2. The specific implementation method is as follows:

[0060] 1. Structured data: Perform the following preprocessing:

[0061] (1) Anomaly detection, unified coding, and numerical standardization.

[0062] (2) Missing value handling: only mark the missing positions (without filling), and hand them over to step S4 for unified processing.

[0063] (3) Extract numerical volatility (the average absolute value of numerical changes at adjacent time points) as a time series feature.

[0064] (4) Time Series Prediction Module: For each data source, a lightweight LSTM prediction model is deployed. The specific design is as follows:

[0065] Input / output definition: The sampling interval is 1 second. The model input is the actual data volume (number of records / second, calculated in real time by the system) of the most recent 60 time points, and the output is the predicted average data volume for the next minute (i.e., from the 61st second to the 120th second).

[0066] Offline pre-training: Using 7 days of historical second-level data, the task is to predict the average value of the next 60 seconds based on the sequence of the past 60 seconds, and training is performed using MSE loss.

[0067] Online incremental updates: After deployment, an online SGD update (learning rate 0.001) will be performed every 5 minutes. During the update, the current time frame will be collected. The previous 60 seconds of data points were used as input. and wait The actual average data volume will be obtained as the target value after seconds. This forms a set of supervised samples, and a gradient descent step is performed. The entire model is not reconstructed.

[0068] Cold start: Before a new data source accumulates 60 historical time points, the LSTM cannot output valid prediction values. During this period, the status input field "Predicted volume value for the next minute" of the HRL high-level strategy in step S2 is filled with the current real-time volume (i.e., the instantaneous value of the most recent second).

[0069] Inference call: Whenever a prediction is needed (e.g., before HRL senior management makes a decision every 10 seconds), forward inference is performed using the most recent 60 seconds of data as input, with a single call taking less than 10ms. Online updates of model parameters run in parallel with the inference service, without affecting prediction availability.

[0070] Output purpose: The predicted value serves as the state input for the HRL high-level policy in step S2, and is used for forward-looking pattern decision-making.

[0071] 2. Semi-structured data: Perform the following preprocessing:

[0072] (1) Automatic parsing, redundancy removal, unified format as JSON, and retention of update timestamp.

[0073] (2) Extract the sequence of adjacent update time intervals and input it into the above time series prediction module to capture the change in update frequency.

[0074] 3. Text data: Perform the following preprocessing:

[0075] (1) Noise removal, word segmentation, and removal of stop words.

[0076] (2) Extract keywords and vector representations (for use in step S3 for association matching).

[0077] (3) Statistical analysis of text length and update frequency is used as an auxiliary feature input to the time series prediction module.

[0078] 4. Image data: Perform the following preprocessing:

[0079] (1) Noise reduction, size uniformity (e.g., 224×224), and pixel standardization (mean is 0, standard deviation is 1).

[0080] (2) Extract basic visual features (such as the intermediate layer output of ResNet50, for use in step S5 decoupling and fusion).

[0081] (3) Calculate the image histogram fluctuation feature (the mean of the difference between the histograms of adjacent frames) as a representation of the update frequency and input it into the time series prediction module.

[0082] 5. Audio data: Perform the following preprocessing:

[0083] (1) Noise reduction, unified sampling rate (e.g., 16kHz), and spectrum standardization.

[0084] (2) Extract spectral features (such as Mel spectrum, for use in step S5 decoupling and fusion).

[0085] (3) Simultaneously extract timing features such as audio energy, zero-crossing rate, and spectral centroid, and input them into the timing prediction module.

[0086] Step S2: Intelligent Synchronization Scheduling Based on Hierarchical Reinforcement Learning (HRL)

[0087] This step employs a two-layer decision-making architecture: a high-level strategy determines the model, while a low-level strategy outputs parameters. Predictive states and mode-switching penalties are introduced to dynamically adapt the synchronization strategy, avoiding strategy oscillations, ensuring timing consistency, and reducing CPU and bandwidth usage. Specifically, the high-level DQN decides on the macro-level model every 10 seconds, including: stable, bursty, and catch-up. The state includes the predicted volume for the next minute provided in step S1, and the reward includes the mode-switching penalty. The low-level PPO outputs the synchronization method, synchronization interval, and resource weights (CPU and bandwidth) every second, conditioned on the model output by the high-level layer. Both layers are trained collaboratively through an Option-Critic architecture, outputting a synchronized, aligned multimodal data stream.

[0088] 1. High-level strategy (slow strategy, executed every 10 seconds):

[0089] State space: =[Current data volume, volume change slope, predicted volume value for the next 1 minute, current synchronization delay, delay change trend, anomaly rate]. The "predicted volume value for the next 1 minute" is provided by the LSTM prediction module deployed in step S1.

[0090] Action space: ;

[0091] Reward function: .in To mitigate the switching penalty factor, set it to 0.2 (range 0.1~0.3) to force higher layers to avoid frequent switching. The weighting coefficient for low-level cumulative rewards is set to 0.9 (range 0.8~0.95), used to balance the current mode switching penalty and future low-level strategy benefits. The cumulative reward for the lower-level strategy at the next moment is defined as the sum of the discounted rewards obtained by the lower-level strategy within 10 lower-level time steps over the next 10 seconds starting from the current higher-level decision moment. Discount factor , For the lower level The immediate reward for each step. This cumulative reward is calculated after the lower-level policy is executed and is used to evaluate the long-term effect of the pattern selected by the higher-level policy.

[0092] 2. Low-level strategy (fast strategy, executed once every 1 second):

[0093] State space: =[Update frequency, data volume, CPU utilization, bandwidth utilization, synchronization delay, anomaly rate, macroscopic pattern given by higher level]. All the above state variables are normalized to the [0,1] interval. Update frequency and data volume are normalized using maximum-minimum normalization; CPU utilization and bandwidth utilization are obtained by dividing by 100 as a percentage; synchronization delay is normalized using a preset maximum delay threshold (e.g., 500ms); and the anomaly rate is a probability value between 0 and 1. The macroscopic pattern given by higher level is one-hot encoded (3-dimensional vector) and concatenated with the other 6 continuous variables to form a 9-dimensional state vector. The update frequency is provided by the update frequency prediction value output by the time series prediction module in step S1, or calculated by the system in real-time statistics of the number of data arrivals within the most recent second.

[0094] Action space: =[Synchronization Method, Synchronization Interval, Resource Allocation Weight]; where, synchronization method ∈ {stream synchronization, batch synchronization}, synchronization interval ∈ {100ms, 1min, 5min, 10min, 30min}, and resource allocation weight is the ratio of CPU and bandwidth allocation (weight). , ∈[0,1] and sum to 1). GPU resources are uniformly scheduled by step S5 and are not allocated in the synchronization layer to avoid feedback signal contamination. The specific actions output by the lower-level strategy.

[0095] Reward function:

[0096]

[0097] The "threshold" represents the maximum allowed synchronization latency for this data source, which can be set according to the business SLA (500ms for normal scenarios and 100ms for low-latency scenarios). Resource utilization rate is... This means that the weights of the action outputs are directly used to perform a weighted summation of CPU and bandwidth utilization, enabling the strategy to learn a reasonable resource allocation. This is the anomaly rate coefficient, with a default value of 0.1. Weight , Dynamic adjustments based on macroeconomic models: In a stable mode , , ; in the sudden mode , , (Prioritize resource allocation); In pursuit mode , , (Prioritizing delay). The baseline weights for each mode can be adaptively determined through Bayesian optimization, but this does not constitute a limitation.

[0098] Handling burst mode: In burst mode, the adjustment of the reward function weights mentioned above will naturally guide the lower-level strategy to choose a shorter synchronization interval and a higher bandwidth weight, rather than forcing the overlay strategy output.

[0099] 3. Algorithm Implementation and Collaborative Training Mechanism:

[0100] The high-level strategy employs DQN (Deep Q-Network), where the Q function is defined as follows: The lower-level strategy employs Proximal Policy Optimization (PPO), a conditional strategy. . , , Specifically, there are high-level state space, low-level state space, and low-level action space. Indicates a macroscopic model.

[0101] The two achieve collaborative training through the following mechanism:

[0102] (1) The top leadership, based on the current situation Output macroscopic mode (mode).

[0103] (2) The lower layer executes multiple time steps (usually 10 steps, corresponding to 10 seconds) in this mode to collect experience. , The corresponding rewards are calculated, and the cumulative rewards at lower levels are also calculated. (Discounts and). Among them To perform the action The lower-level state in the next moment.

[0104] (3) According to the high-level Update Q network, reward .

[0105] (4) The gradient is propagated back from the lower layer to the higher layer through the option value function.

[0106] In this process, the implementation of experience playback and hybrid action space is as follows:

[0107] Experience replay: To stabilize the training process and break the temporal correlation between data, separate experience replay buffers are maintained for high-level and low-level data. The high-level buffer stores the complete experience tuple for each high-level decision. The high-level decision-making state includes the current data volume, the slope of volume change, the predicted volume value for the next minute, the current synchronization delay, the delay change trend, and the anomaly rate; the macro-mode selected by the high-level decision-making (stable / bursting / chasing); the reward obtained by the high-level decision-making after executing the mode; and the next high-level state after executing the mode (after 10 seconds of low-level policy interaction). The high-level buffer uses random sampling, and a small batch of samples is drawn from it during each training session to update the Q-network of DQN.

[0108] The low-level buffer stores the empirical tuple for each low-level time step (per second). These are the states during low-level decision-making (including update frequency, data volume, CPU utilization, bandwidth utilization, synchronization latency, anomaly rate, and the macroscopic pattern given by the high-level layer), the actions performed at the low level (synchronization method, synchronization interval, resource allocation weight), the immediate reward obtained after performing the action (the calculation formula is shown in the low-level reward function), the next low-level state after performing the action, and the current high-level macroscopic pattern (as a condition for the low-level policy). The low-level buffer also uses random sampling for updating the policy network and value network of the PPO algorithm.

[0109] High-level and low-level buffers are maintained independently to avoid mutual interference. High-level decision cycles are longer (10 seconds), and the number of experience samples is smaller; therefore, the high-level buffer capacity can be set to a smaller value (e.g., 500 samples). Low-level decision cycles are shorter (1 second), and experience samples are generated quickly; therefore, the buffer capacity needs to be set to a larger value (e.g., 10,000 samples). During sampling, the low-level buffer only extracts experience tuples corresponding to the current high-level pattern, ensuring that the training data distribution matches the current policy.

[0110] Hybrid Action Space Implementation: The discrete and continuous actions output from the parameterized action space described above are stored together with the reward signal in a low-level buffer for joint optimization of the policy network. Using the parameterized action space method, discrete actions (synchronization mode, synchronization interval) are output through a softmax distribution, while continuous actions (resource weights) are output through a Gaussian distribution. Both discrete and continuous actions share the underlying network and are jointly optimized through separately calculated losses. Specifically, the policy network outputs the logits of discrete actions and the mean and variance of continuous actions. The total loss is the sum of the cross-entropy loss of discrete actions and the negative log-likelihood loss of continuous actions. Shared network parameters are jointly updated through backpropagation.

[0111] 4. Synchronous Execution: Real-time data uses streaming synchronization, and batch data uses adaptive batch synchronization, with a coordinator ensuring timing consistency. In burst mode, due to the guidance of the reward function weights, the synchronization interval of the policy output is usually short (can be reduced to 100ms), and the bandwidth weight is high; in pursuit mode, the policy prioritizes latency, and CPU resources can be temporarily increased (reflected through resource allocation weights).

[0112] 5. Anomaly Handling: An autoencoder is used for real-time anomaly identification. The autoencoder takes timing characteristics such as synchronization delay, data volume, and anomaly rate as input (window size 60). If the reconstruction error exceeds a threshold, it is considered an anomaly. Anomaly classification and handling:

[0113] (1) Minor anomalies: Automatically remove anomalies without affecting the strategy.

[0114] (2) Moderate anomaly: Dynamically adjust the low-level reward function (Abnormality rate coefficient), temporarily increase the penalty.

[0115] (3) Serious anomaly: triggers link switching and alarm.

[0116] The anomaly rate of the autoencoder output has been used as a low-level state. The "abnormality rate" variable is input to form a closed loop.

[0117] (The following is for understanding: This step introduces hierarchical reinforcement learning to solve the policy oscillation problem in data synchronization, and adds predictive state (future volume prediction value) and mode switching penalty, which is different from the single-layer RL (such as CN120688537A, CN121462608A), the scheduling method based on LSTM reserved resources (CN121542982A) or fixed policy in existing technologies.)

[0118] Technical Difficulty Analysis of the Strategy Oscillation Problem (Simplified):

[0119] (1) Environmental non-stationarity: Under sudden flow, the state changes drastically, and a single-layer RL is difficult to adapt.

[0120] (2) Time scale conflict: Synchronous scheduling needs to take into account both millisecond-level response and minute-level planning, and single-layer decision frequency cannot be taken into account.

[0121] (3) Reward sparsity and delay: Synchronization effect needs to be evaluated in subsequent fusion steps, and single-layer RL is difficult to attribute.

[0122] The solution mechanism of this invention:

[0123] (1) Time abstraction: the high-level layer sets the pattern in 10 seconds and the low-level layer takes action in 1 second to smooth out short-term noise and ensure timely response.

[0124] (2) Mode switching penalty: introduced in high-level rewards Forced mode is stable.

[0125] (3) Predictive state space: includes future volume prediction to achieve forward scheduling.

[0126] Unlike HRL applications in general task scheduling, this invention addresses the unique temporal scale conflict inherent in multimodal data synchronization. In general task scheduling, each task is independent, and the scheduler only needs to optimize the completion time of the current task. However, data synchronization requires ensuring temporal consistency between different modalities (e.g., images and text must be aligned with the same timestamp), and adjustments to the synchronization interval directly affect the input quality of subsequent association matching and the confidence of the fusion results. This chain dependency does not exist in general resource scheduling. Therefore, the HRL in this invention specifically designs a high-level mode decision based on "future volume prediction value" and a cross-layer collaborative mechanism that feeds back "low-level cumulative rewards" to the high-level reward function, representing a customized innovation for data synchronization scenarios.

[0127] It is important to note that the combination of LSTM prediction and reinforcement learning in existing technologies is fundamentally different from the predictive state space of this invention. For example, CN121542982A uses an LSTM predictor independent of RL decision-making, solely for reserving computing resources, which is a task-level passive prediction; its prediction results do not participate in the mode decision of the RL policy, and the RL policy is still a reactive adjustment to the current state. In contrast, this invention directly encodes the predicted volume for the next minute into the state space of the HRL high-level policy, enabling the high-level DQN to proactively switch modes based on the prediction of future traffic (e.g., switching from a stable mode to a burst mode in advance), achieving policy-level forward scheduling. This deep integration mechanism of prediction and decision-making is the key feature that distinguishes this invention from existing LSTM+RL schemes.

[0128] Step S3: Intelligent Association Matching

[0129] The synchronized and aligned multimodal data stream output in step S2 achieves multimodal data association by combining explicit ID matching and implicit semantic matching, thereby improving the accuracy of multimodal data association and providing a basis for subsequent missing data completion.

[0130] 1. Explicit association matching

[0131] (1) Unified association ID generation: For each data source, extract the fields in its metadata that can uniquely identify the data entity (such as database primary key, file name, business ID). If a data source lacks an explicit ID, perform MD5 hashing on its key fields (such as text title, image MD5 value) to generate a 128-bit fixed-length ID (or obtain the first 128 bits of SHA-256).

[0132] (2) Hash Index Construction: Store all extracted association IDs in a hash table, with the ID value as the key and the data source identifier and data location pointer as the value. When new data arrives, calculate its ID and search for it in the hash table. If a match is found, establish the association directly.

[0133] (3) Confidence setting: Explicit association is based on exact matching, and the confidence is set to 1.0 (or 0.9999 to account for the very low probability of hash collisions).

[0134] 2. Implicit semantic association matching

[0135] (1) Unified semantic feature extraction: For different modalities, corresponding pre-trained encoders are used, and the outputs are mapped to the same dimensional space (512 dimensions):

[0136] Text: Using the BERT-base model, the 768-dimensional output of the [CLS] token is taken and reduced to 512 dimensions through a fully connected layer.

[0137] Image: Visual encoder (ViT-B / 32) using CLIP, outputting 512-dimensional features.

[0138] Audio: Using the Wav2Vec2.0 model, attention pooling (or average pooling, suitable for short audio clips <10 seconds) is applied to the audio clips, outputting 768 dimensions and then reducing the dimensions to 512 dimensions through a fully connected layer.

[0139] Structured data: Numerical fields are normalized and concatenated, categorical fields are encoded using an embedding layer, and then encoded to 512 dimensions through two fully connected networks (256→512).

[0140] (2) Vector Index Library Construction: For the historical data that has been associated, its 512-dimensional semantic feature vectors are used to construct an IVF index (inverted file index) using the FAISS library. The index parameters are set to the number of cluster centers nlist=100 and the number of probe clusters nprobe=10, which can be dynamically adjusted according to the amount of historical data (e.g., nlist=sqrt(N)). Note: In the FAISS (Facebook AISimilarity Search) library, nlist represents the number of cluster centers, the number of inverted lists, and the number of buckets; nprobe represents the number of probe clusters and the number of clusters accessed during the search.

[0141] (3) Implicit association retrieval: For new data to be associated, its 512-dimensional semantic features are extracted as query vectors. The top-5 most similar historical records are retrieved from the FAISS index, and the cosine similarity of each record is calculated as the implicit association confidence score. No lower limit is set, and all top-5 results are returned.

[0142] 3. Fusion of Association Results and Determination of Confidence

[0143] (1) Candidate set construction:

[0144] If an explicit association is found, a candidate (ID_explicit, confidence = 1.0) is generated.

[0145] All Top-5 results (ID_implicit_i, confidence = sim_i) returned by implicit retrieval are added to the candidate set.

[0146] (2) Decision-making rules:

[0147] No candidates: If the explicit association fails and the implicit search yields no results (the historical database is empty), it is marked as "association failed" and proceeds to the missing data completion process in step S4 (considered as missing data).

[0148] Single candidate: If there is only one candidate, the result is adopted directly.

[0149] Multiple candidates: Select the candidate with the highest confidence level. If there is a tie for the highest confidence level, the explicit association result (if any) shall be used first.

[0150] (3) Manual review and processing:

[0151] In this step, two confidence thresholds are defined: the automatic adoption threshold (0.95 in this example) and the manual review lower bound (0.85 in this example). These two thresholds will be referred to as the association result confidence thresholds in the following text.

[0152] Automatic adoption: If the confidence level of the selected candidate is ≥0.95, the association result will be automatically adopted.

[0153] Manual review: If 0.85 ≤ confidence level < 0.95, the result is transferred to the manual review queue. The manual review result is used as a feedback signal input to the cause-effect graph.

[0154] Trigger completion: If the confidence level of the selected candidate is <0.85, it is marked as "association questionable" and the missing completion process in step S4 is triggered (to enhance or correct the association).

[0155] (4) Result storage and traceability:

[0156] The final association result is stored in a relation table, with fields including: source data identifier, target association ID, association method (explicit / implicit / fusion), confidence level, and timestamp.

[0157] Establish a complete traceability chain from raw data to related results. Key intermediate results (such as selected feature vectors) can be stored optionally to balance storage overhead and traceability.

[0158] Finally, the association results (including successfully associated ID pairs, confidence levels, and failure / questionable flags) are passed to step S4 (missing quantification and intelligent completion layer) via a message queue. Step S4 will then determine whether completion needs to be performed and the priority of completion based on these results.

[0159] Step S4: Joint missing item identification and cross-modal consistent completion

[0160] This step defines the "joint missing" scenario and achieves completion through joint latent space generation and consistency discriminator, ensuring cross-modal semantic consistency and supporting both generative and retrieval-based completion strategies.

[0161] Joint missing data determination method: A pre-trained language model (such as BERT) is used to encode the metadata description of each missing field (including field name, field business meaning description text, and context labels of the table / document where the field is located) into a vector, and the cosine similarity is calculated. If the similarity exceeds a preset threshold (set to 0.7 in this scheme), it is determined to be joint missing. This threshold is determined based on the statistical distribution of similarity between positive and negative samples in historical data (e.g., taking the point with the maximum Youden index) and can be adaptively adjusted according to the business scenario.

[0162] Connection with steps S3 and S5:

[0163] (1) Input: Receive association results (including successfully associated ID pairs, confidence level, association failure / doubtful markers) and semantic feature vectors of each modality from step S3.

[0164] (2) Output: Pass the completed data (including completion markers and confidence) and the latent space vector generated during the completion process (optional) to step S5 (unified semantic fusion layer).

[0165] Special cases: If step S3 is marked as "association failed" (no candidate ID), the missing identification in this step will be based on the original data; if it is marked as "association questionable" (low confidence), then completion will be attempted first to strengthen the association.

[0166] 1. Missing data identification: Rules + autoencoder identify missing data types, which are divided into the following types.

[0167] (1) Rule recognition: Check whether each modal field is NULL, empty string, NaN, empty file, etc., and output the missing type:

[0168] Local missing: Partial data of a single modality is missing.

[0169] Completely missing: All data for a certain modality is missing.

[0170] (2) Autoencoder recognition (optional, used to capture implicit anomalies): Train an autoencoder (structure: input layer → 256 → 128 → 256 → output layer, ReLU activation) with complete data, input the test data, and calculate the reconstruction error. If the error exceeds a preset threshold, it is judged as an anomaly. The threshold is set to the 95th percentile of the reconstruction error of the training set, and can be dynamically adjusted according to the false positive rate during online operation (e.g., updated weekly). The adjustment method is as follows: if the false positive rate (the proportion of normal samples that are misclassified as anomalies) exceeds 5%, the threshold is increased to the 98th percentile; if the false negative rate exceeds 5%, it is decreased to the 90th percentile.

[0171] (3) Joint missing data determination: If the rule identifies multiple modalities as missing, and the BERT cosine similarity of the metadata of each missing field is >0.7, then it is marked as joint missing data.

[0172] 2. Quantification of the impact of missing values: To balance complexity and practicality, a combination of AHP and entropy weighting is used.

[0173] (1) Use AHP (analytic hierarchy process) to determine the weight of each business indicator (such as shared impact factor, user experience factor, etc.) by business experts.

[0174] (2) Use the entropy weight method to determine the weight of data features based on the statistical characteristics (missing rate, field importance, etc.) of historical missing data.

[0175] (3) Weighted calculation of the impact of missing values ​​on the score, mapped to a four-level priority:

[0176] Level 1 (High): Impacts core business metrics; prioritize completion, and allocate more system resources accordingly.

[0177] Level 2 (Medium-High): Affects secondary indicators.

[0178] Level 3 (Medium): Does not affect core business but affects data integrity.

[0179] Level 4 (Low): No actual business impact; completion can be delayed or only marked.

[0180] 3. Completion method selection strategy:

[0181] Generative-first approach: Suitable for business scenarios that allow for a certain degree of fictitiousness, such as content recommendation and data augmentation. It prioritizes generation with cross-modal consistency constraints (see 4.4). The degree of freedom of generation can be controlled by adjusting the threshold of the consistency discriminator (e.g., from 0.95 to 0.90).

[0182] Search-based priority: For scenarios with strict requirements on data authenticity, a search is forced from the standard library. If the data cannot be found, it is marked as missing and no data is generated.

[0183] Hybrid strategy (default): For general business scenarios, first try generative completion. If the consistency discriminator still fails to pass the threshold (e.g., 0.95) after multiple samplings (e.g., 3 times), it will automatically fall back to retrieval completion.

[0184] 4. Cross-modal consistency constraint generator: This generator aims to address the semantic inconsistencies caused by independent generation of each modality in scenarios with joint missing features. It encodes available multimodal features into the same joint latent space, samples from it, and decodes the feature representation of the missing modality. Simultaneously, a discriminator is introduced to force high semantic consistency (≥0.95) between the generated missing modality and existing modalities. This "joint generation + consistency constraint" mechanism ensures that the completion results are coordinated and consistent across modalities.

[0185] (1) Joint latent space variational autoencoder (C-VAE)

[0186] Encoder: Inputs available multimodal features (such as non-missing parts of images or text), outputs latent space. The mean and variance of.

[0187] Example structure: Fully connected layer [input dimension] → 256 → 128, output mean and logarithmic variance , Dimensions = 128.

[0188] Decoder: From Reconstructing the feature representation of the missing modality. Explanation: The latent variable z obtained by the joint latent space variational autoencoder (C-VAE) is used by the decoder network to generate (reconstruct) the feature vector of the originally missing modality, thereby completing the missing data.

[0189] Example structure: Fully connected layer 128→256→[output dimension].

[0190] Training data: Use complete, semantically aligned multimodal pairs (e.g., target image-description pairs). If complete samples are scarce, use semi-supervised learning: transfer parameters from pre-trained models in similar domains (e.g., general image-text models) and then fine-tune them with a small number of complete samples.

[0191] (2) Consistency Discriminator

[0192] Structure: Binary classifier (MLP: input is the concatenation of features of generated cross-modal pairs, output is the consistency probability).

[0193] Training data: Positive examples are real multimodal pairs (from the complete training set). Negative examples are automatically constructed contradictory pairs, which are constructed by randomly selecting images and text from different samples in the training set and combining them; or by using an independently trained modality generator to generate each modality separately and then randomly pairing them.

[0194] Usage: After C-VAE generates candidate missing modalities, the discriminator evaluates their consistency with existing modalities. Only results with a discriminant confidence score > 0.95 are retained. If multiple sampling attempts (e.g., 3 attempts) fail, it reverts to retrieval-based completion.

[0195] Feature reuse with step S3: To reduce redundant computation, this generator reuses the semantic features (512-dimensional vectors) of each modality extracted in step S3, without the need for re-encoding. For missing modalities, the generator outputs their semantic feature vectors, which are then processed by the decoupling and fusion module in step S5.

[0196] 5. Strong Consistency Validation (Post-Filtering): After completion, a pre-trained model (CLIP or BERT similarity model) is used to calculate cross-modal semantic similarity. If the similarity is <0.95, the completion result is rejected; if a hybrid strategy is used, it reverts to retrieval-based completion; if retrieval-based completion also fails, it is marked as "Completion Failed" and submitted for manual review. The similarity calculation here can directly use the features extracted in step S3 to avoid redundant encoding.

[0197] (Creative Description: This invention explicitly defines the concept of "joint missing" and designs a specialized cross-modal consistency constraint generator (C-VAE + discriminator) to enforce cross-modal semantic consistency of the generated results, which differs from the independent generation or retrieval completion methods in existing technologies (such as the retrieval + diffusion generation in CN120724376A, the retrieval-based completion in CN121350435A, and the data augmentation method in CN121524948A).)

[0198] Step S5: Decoupling representation learning-driven two-layer fusion of unified semantic space

[0199] This step constructs a unified semantic space, explicitly separating modality-shared features from modality-specific features, and fusing only the shared parts to improve fusion accuracy and interpretability. Each modality extracts 512-dimensional features via an independent encoder, which are then decomposed into shared semantic vectors and modality-specific vectors by a decoupling module. Shared and unique features are separated by minimizing mutual information, and shared semantics are aligned through cross-modal comparative learning. The first layer uses attention-weighted summation, and the second layer uses Transformer decoding to output a unified semantic vector.

[0200] This step receives the complete data from step S4. If step S4 has already extracted the 512-dimensional semantic features of each modality, it can be reused directly to avoid repeated encoding; otherwise, the encoder in this step is used to extract them again.

[0201] 1. Model architecture (taking image-text-audio trimodal as an example, but not limited to this)

[0202] (1) Encoder: Each modality uses an independent pre-trained encoder, with a uniform output dimension of 512:

[0203] Text: BERT-base, taking the 768-dimensional output of the [CLS] token, and reducing the dimensionality to 512 dimensions through a fully connected layer.

[0204] Image: ResNet50, with the classification head removed, 2048-dimensional global pooled features extracted, and then reduced to 512 dimensions via a fully connected layer.

[0205] Audio: WaveNet, taking the last layer's time dimension average pooling, and then mapping it to 512 dimensions through a fully connected layer.

[0206] (2) Decoupling module: The 512-dimensional encoded output of each modality is passed through a decomposition network to output a shared semantic vector. (256-dimensional) and mode-specific vectors (256 dimensions). The decomposition network structure consists of two fully connected layers. First layer: 512→1024 (ReLU); Second layer: 1024→512 (linear). The 512-dimensional output is decomposed into... (256 dimensions) and (256 dimensions).

[0207] Note: This scheme does not output a noise vector to avoid redundancy. If regularization is required, it can be achieved by... Add Dropout implementation to the above.

[0208] (3) Decoder (for reconstruction): from (512-dimensional) The original encoded features (512-dimensional) are reconstructed through a decoder, using MSE loss constraints. Decoder structure: two fully connected layers 512→1024→512, with ReLU used in the intermediate layer.

[0209] 2. Loss Function and Training Strategy

[0210] (1) Decompose constraint loss

[0211] Minimize Mutual Information (MINE): Minimize and Mutual information between them ensures the separation of shared and unique information. The mutual information neural estimator (MINE) is used to estimate mutual information. And minimize it.

[0212] Reconstruction Loss (MSE): From The original features are reconstructed by the decoder, and the loss is ,in This is the original feature vector (512-dimensional) output by the encoder. The feature vector reconstructed by the decoder.

[0213] Cross-modal contrastive learning loss (InfoNCE): The loss between different modalities... They should have high similarity. Positive example: different modalities of the same data sample. Negative examples: different samples The loss is recorded as .

[0214] Masked Modeling Loss (MLM): Randomly masks the input features of a particular modality, requiring the model to extract features from other modalities. The system predicts blocked content to enhance the robustness of cross-modal semantic understanding. The loss is denoted as... .

[0215] (2) Multi-target loss balancing and progressive training: The total loss is:

[0216]

[0217] The progressive training strategy (which resolves the conflict between minimizing mutual information and cross-modal consistency) is as follows:

[0218] Phase 1 (Warm-up): The first 50 epochs, using only... Without minimizing mutual information, each modality... Initial alignment.

[0219] Phase 2 (Joint Optimization): Starting from the 51st epoch, a mutual information minimization term is added, with an initial weight of 0.1. Simultaneously, the mutual information is dynamically adjusted. The weights are decreased by 0.05 every 10 epochs from 0.4 until they reach 0.2; the mutual information weights are increased by 0.05 every 10 epochs from 0.1 until they reach 0.3. Training continues until the loss on the validation set converges.

[0220] 3. Fusion Layer (Dual-Layer Fusion)

[0221] First layer (feature-level weighted summation): Calculates each mode Attention weights. Calculate the first... Modal attention score:

[0222]

[0223] Among them, the weight matrix bias Attention vector , We hide the dimension for attention (e.g., 128). Then we compute the normalized weights using the softmax function:

[0224]

[0225] The final weighted summation yields the first layer of fused features:

[0226]

[0227] The second layer (semantic-level Transformer decoding): will... As a query, each modality As keys and values, these are input to a Transformer decoder (2 layers, 4-head attention, 512 feedforward dimensions), outputting a final unified semantic vector. Output dimension configuration:

[0228] (1) Lightweight deployment (number of modalities ≤ 3): 128 dimensions;

[0229] (2) General middleware scenario (number of modalities ≥ 4): 256 dimensions;

[0230] (3) High precision requirement: 512 dimensions.

[0231] 4. Training Data and Initialization: Each modality encoder is initialized using publicly available pre-trained weights (BERT, ResNet50, WaveNet). The decoupling module, decoder, and fusion layer are randomly initialized using Xavier. End-to-end fine-tuning is performed using real-world business data (the same complete multimodal pairs as in step S4) to optimize the training. .

[0232] Creative Description: This invention introduces decoupled representation learning into multimodal fusion, explicitly separating modality-shared features from unique features, and differs from existing technologies by minimizing mutual information constraints.

[0233] (1) Comparison with patent CN120781307A (adversarial alignment): no separation of shared / unique features was performed.

[0234] (2) Comparison with patent CN121350651A (fine-grained semantic decoupling): its decoupling is aimed at attribute-level fine-grained semantics, while the present invention separates shared and unique information at the modal level, with different goals and application scenarios.

[0235] (3) Compared with the Hadoop ecosystem (shallow splicing): it cannot bridge the semantic gap and has no interpretability.

[0236] (This invention improves the robustness and interpretability of fusion by progressively training a balance between minimizing mutual information and cross-modal consistency. HRL provides high-quality temporally aligned data for fusion, and decoupled fusion provides interpretable confidence metrics for causal feedback. Causal feedback, in turn, optimizes the parameters of HRL and completion. This positive cycle cannot be achieved through simple superposition.)

[0237] Step S6: Verify the output and optimize the end-to-end feedback based on causal inference.

[0238] This step uses a cause-effect graph to accurately pinpoint the root cause of fusion quality issues, rather than relying on fuzzy parameter tuning, achieving a synergistic improvement across the entire synchronization-association-completion-fusion chain. When the fusion confidence level falls below a threshold, dual machine learning is used to estimate the average causal effect (ACE) of each preceding node. The node with the highest ACE is selected as the root cause, and parameters are adjusted accordingly and written back to the corresponding module to form a closed loop.

[0239] This step receives the unified semantic vector (and each intermediate feature vector) output from step S5, and obtains information such as synchronization status, association results, and completion records from steps S2 to S4 for quality verification and causal inference.

[0240] 1. Fusion quality verification: Calculate the following three metrics:

[0241] (1) Fusion confidence during offline evaluation: defined as the cosine similarity between the output unified semantic vector and the standard semantic vector in the validation set. Quality metrics during online runtime: if real-time annotations are not available, the model's embedded confidence (such as the maximum probability output by the classifier softmax) is used as a surrogate metric. This surrogate metric is positively correlated with the offline cosine similarity (verified offline). The threshold for triggering causal feedback (such as 0.85) is for the online surrogate metric.

[0242] (2) Association Error Rate: Periodically (e.g., every 1000 association operations) samples are manually reviewed to calculate the deviation ratio between the association results and the manually labeled results. The mean of the association confidence score (from step S3) is used as a proxy indicator in the online feedback. As a node variable (link_error) in the causal graph, it participates in the average causal effect (ACE) estimation to determine whether the association error has a significant causal impact on the fusion confidence score.

[0243] (3) Completion failure rate: The proportion of completion results rejected by consistency check in step S4 out of all completion requests. As an intermediate node (completion_fail_rate) in the cause-effect graph, it connects completion parameters (such as temperature coefficient, consistency threshold) and fusion confidence to help locate the root cause of the completion process.

[0244] The calculation results of the three indicators are all written to the system log for offline analysis and cause-effect graph updates.

[0245] 2. Causal Inference Feedback Module:

[0246] (1) Construction of cause-effect graph

[0247] The causal graph nodes include: synchronization delay (sync_delay), synchronization anomaly rate (sync_anomaly), link error rate (link_error, using a surrogate metric), completion generation parameters (such as the C-VAE temperature coefficient temp), completion consistency threshold (consistency_th), completion failure rate (completion_fail_rate), and fusion confidence (fusion_conf, the target variable).

[0248] The edges of the cause-effect graph represent the causal direction. The initial structure is predefined based on domain expert knowledge, for example: sync_delay→fusion_conf; sync_anomaly→fusion_conf; link_error→fusion_conf; temp→completion_fail_rate→fusion_conf; consistency_th→completion_fail_rate→fusion_conf.

[0249] After accumulating 1000 new fused samples (or every 24 hours, triggered if either condition is met), the current causal graph is first used for root cause localization (see below). Then, the PC algorithm (the full name of the PC algorithm is the Peter-Clark algorithm, which is a causal structure learning algorithm based on conditional independence tests, used to infer causal relationship graphs (directed acyclic graphs) between variables from observed data) is used to learn and correct the causal graph structure, ensuring that the causal relationships are continuously updated as the system runs. If the sample accumulation rate is too fast, the execution frequency of the PC algorithm will be suppressed by time interval limits to control the system load. The PC algorithm uses the Fisher Z test with a significance level set to 0.01, and continuous variables are first subjected to a Gaussian kernel transformation. The causal graph is updated using the most recent 2000 samples (including new samples and some historical samples).

[0250] (2) Root cause localization (average causal effect ACE estimation). When the fusion confidence is below the threshold, the average causal effect (ACE) of each preceding node on the fusion confidence is estimated using the most recent 1000 fusion samples (or all samples if accumulation is insufficient). The Double / Debiased Machine Learning method is adopted, and the specific steps are as follows:

[0251] Step 1 (Data Preparation): Collect recent data One fusion sample ( Record each candidate root cause node. The value and fusion confidence Candidate root cause nodes include: synchronization delay, synchronization anomaly rate, association error rate (proxy), C-VAE temperature coefficient, and completion consistency threshold.

[0252] Step 2 (Residual Calculation): For each candidate root cause node Fit using machine learning models (such as XGBoost or Random Forest):

[0253] Use all nodes except T as covariates. )predict .

[0254] Using the same covariates predict .

[0255] in In the cause-and-effect diagram, except and All nodes outside the set (i.e., the minimum fully adjusted set). Calculate the residuals:

[0256] ,

[0257] Step 3 (ACE estimation): Linear regression ACE estimate is obtained .in This is known as the average causal effect (ACE).

[0258] Step 4 (Significance Validation): Calculate using the bootstrap method (resample 1000 times). The 95% confidence interval is used. If the confidence interval does not contain 0 and the p-value after Bonferroni correction is <0.05 (i.e., the original p-value <0.05 / number of nodes), then the node is considered to have a significant causal effect on the fusion confidence.

[0259] (3) Parameter adjustment. Select the node with the largest and statistically significant ACE as the root cause, and then adjust the parameters for that node:

[0260] The root cause is synchronization delay: increase the reward function of the lower-level policy. (Delay weighting), and adjust the high-level reward function by adding a delay-related penalty to guide the high-level policy into "chasing mode" rather than forcibly switching modes. For example, modify the reward to... ,in The default penalty coefficient is set (e.g., 0.1). The higher-level reward function can be expanded with the penalty term without affecting the original training mechanism.

[0261] The root cause is the synchronization anomaly rate: increase the sensitivity of autoencoder anomaly detection (lower the reconstruction error threshold), or trigger link switching.

[0262] The root cause is the association error rate: automatically adjusting the lower bound of manual review in the association result fusion decision (e.g., reducing it from 0.85 to 0.80) to allow more low-confidence associations to enter the completion process, rather than increasing the proportion of manual review.

[0263] Root cause to complete the generation parameters (C-VAE temperature coefficient): reduce the temperature coefficient (reduce randomness), or tighten the consistency check threshold (e.g., increase from 0.95 to 0.96, but not exceeding 0.98).

[0264] Root cause: Completion consistency threshold: The threshold is dynamically adjusted (within the range of 0.90~0.98) to balance completion pass rate and accuracy. The specific adjustment step size is 0.01.

[0265] Write all adjusted parameters back to the corresponding modules (steps S2, 3, 4, 5) and record the adjustment log.

[0266] 3. Feedback Signal Quantization, Transmission, and Loop Closure. Adjusted parameters are written back to each module via the configuration center or direct API calls. The time, root cause, adjustment content, and fusion confidence level before and after each feedback are recorded for subsequent causal graph updates and effect evaluation. Every 1000 feedbacks (or every 24 hours), the PC algorithm is re-executed to update the causal graph structure, and the new graph is used for subsequent ACE estimation.

[0267] (Creative Description: This invention introduces causal inference into the closed-loop feedback of the system, achieving root cause localization and precise parameter adjustment, which differs from existing technologies.)

[0268] (1) Simple threshold feedback (such as retry): cannot distinguish the contribution of each link.

[0269] (2) Only feed back the execution results to RL (e.g., CN120688537A): Only adjust the RL parameters, without involving modules such as association and completion.

[0270] (3) Optimization methods that collect feedback data but do not locate the root cause (such as CN121542982A): can only adjust parameters globally, but cannot locate specific links and parameters.

[0271] By using ACE estimation and significance testing, this invention can accurately identify the root causes of the decline in fusion quality and automatically perform targeted optimizations to achieve end-to-end self-healing.

[0272] Compared to conventional feedback (such as "periodically collecting feedback data and optimizing system parameters" in CN121542982A), this invention has a fundamental difference: conventional feedback can only obtain the correlation between parameters and fusion quality (e.g., the greater the synchronization delay, the lower the fusion confidence), but it cannot distinguish the causal direction—the synchronization delay may cause the fusion to deteriorate, or the fusion module itself may have a problem that makes it more dependent on synchronization data. This invention, through causal graphs and dual machine learning estimation of average causal effect (ACE), can eliminate confounding factors (such as the anomaly rate affecting both synchronization and fusion) and identify the true causality. For example, experiments showed that when the fusion confidence decreased, conventional feedback would incorrectly increase synchronization resources, while this invention could correctly locate the root cause as "the C-VAE temperature coefficient is too high," a causal root cause localization capability that existing feedback mechanisms lack.

[0273] Example 1: System Architecture

[0274] This system adopts a layered architecture, consisting of eight layers. Data and control flows between these layers interact via message queues (Kafka) and RPC calls. The overall architecture is as follows: Figure 1 As shown.

[0275] Layer 1: Data Source Layer. Stores multimodal raw data (structured, semi-structured, text, images, audio), pushes data to the preprocessing layer via Kafka, and receives processing confirmation signals.

[0276] Layer 2: Preprocessing and Predictive Feature Extraction Layer (corresponding to step S1). Perform format unification, noise reduction, and missing data labeling (without padding). Deploy the LSTM prediction module to output the predicted data volume for the next minute. The standardized data and predicted values ​​are then passed to subsequent layers.

[0277] Layer 3: Intelligent Synchronization Scheduling Layer (corresponding to step S2). Based on hierarchical reinforcement learning (HRL, high-level DQN + low-level PPO), it dynamically adjusts the synchronization interval and resource allocation weights (CPU, bandwidth). It receives the predicted values ​​from Layer 2 as high-level state inputs, outputs the synchronized data stream to the association layer, and reports the synchronization status (latency, anomaly rate, number of mode switching) to the feedback optimization layer.

[0278] Layer 4: Association Matching Layer (corresponding to step S3). Perform explicit ID matching (hash table) and implicit semantic matching (FAISS index). Output the association results (association ID, confidence, failure / doubtful marker) and the 512-dimensional semantic feature vector of each modality to the missing feature completion layer.

[0279] Layer 5: Missing Data Completion Layer (corresponding to step S4). Joint missing data is identified (based on BERT metadata similarity), and generative or fallback retrieval-based completion is performed using a C-VAE+consistency discriminator. The completed data (including completion tags and confidence scores) and optional latent space vectors are output to the semantic fusion layer.

[0280] Layer 6: Semantic Fusion Layer (corresponding to step S5). Performs decoupled representation learning, decomposing features from each modality into shared semantic vectors and unique vectors. Outputs a unified semantic vector through two-layer fusion (attention-weighted summation + Transformer decoding). Receives the completed data and outputs the fusion result to the verification feedback layer.

[0281] Layer 7: Verification Output and Feedback Optimization Layer (corresponding to step S6). Real-time calculation of fusion confidence, association error rate (surrogate metric), and completion failure rate. When the fusion confidence is below the threshold (0.85), the causal inference module is triggered: it estimates the average causal effect (ACE) of each preceding node using the most recent 1000 samples, locates the root cause, and generates targeted adjustment signals (such as delay weights, association thresholds, C-VAE temperature coefficients, etc.). The adjustment signals are written back to layers 3-6; qualified fusion results are output to the service interface and storage; unqualified results can trigger retries (up to 3 times).

[0282] Layer 8: Monitoring and Operations Layer. Real-time collection of resource usage (CPU / memory / bandwidth), anomaly logs, and historical fusion effects from each layer. Provides data to Layer 7 for cause-effect graph updates (PC algorithm structure learning is performed after every 1000 feedbacks); issues critical alarms (e.g., three consecutive ineffective adjustments) and supports link switching.

[0283] Example 2:

[0284] A method for intelligent synchronization and fusion of multimodal data for data platforms includes:

[0285] Multimodal data is acquired, and intelligent synchronization scheduling is performed based on a hierarchical reinforcement learning model. The hierarchical reinforcement learning model includes a high-level policy and a low-level policy. The state space of the high-level policy contains the predicted data volume, which directly participates in the pattern decision of the high-level policy as an integral part of the state space. The action space is a macro-level pattern, which includes at least one of a stable pattern, a burst pattern, and a number-chasing pattern. The reward function includes a pattern switching penalty term. The low-level policy outputs the synchronization method, synchronization interval, and resource allocation weights (including CPU weight and bandwidth weight) according to the macro-level pattern.

[0286] The synchronized multimodal data is input into the decoupled representation learning module, which decomposes each modality feature into a shared semantic vector and a modality-specific vector, and maps only the shared semantic vector to a unified semantic space, outputting a unified semantic vector of a preset dimension.

[0287] When joint missing data is detected, a cross-modal consistency constraint generator is used for completion. Joint missing data refers to the simultaneous presence of missing data in multiple modalities and the semantic relationship between the missing data. The generator includes a joint latent space encoder and a consistency discriminator, which forces the cross-modal semantic similarity of the completion result to be no less than a threshold.

[0288] Based on the quantitative signal of the fusion result, the average causal effect (ACE) of each preceding step on the fusion quality is calculated through causal inference, the root cause is located and the corresponding parameters are adaptively adjusted to form a closed-loop collaboration across the entire chain.

[0289] Furthermore, the high-level policy adopts DQN, and the low-level policy adopts PPO, and the two are trained collaboratively through an option-commentator architecture; wherein the Q function of the high-level policy is defined as Q(s_high, mode), the conditional policy of the low-level policy is π(a_low | s_low, mode), and the gradient is backpropagated through the option value function.

[0290] Furthermore, the state space of the high-level strategy also includes at least one of the following: current data volume, volume change slope, synchronization delay, delay change trend, and anomaly rate; the volume prediction value for the next 1 minute is provided by the LSTM time series prediction module deployed in the preprocessing stage.

[0291] Furthermore, the action space of the low-level strategy includes synchronization method, synchronization interval, and resource allocation weight; the synchronization method includes stream synchronization or batch synchronization, the synchronization interval is selected from at least one of 100ms, 1min, 5min, 10min, and 30min, and the resource allocation weight is the allocation ratio of CPU and bandwidth; the reward function of the low-level strategy is a weighted combination, and the weight is dynamically adjusted according to the macro mode.

[0292] Furthermore, the decoupled representation learning module achieves feature separation by minimizing the mutual information between shared semantic vectors and modality-specific vectors. The training loss includes mutual information minimization loss, cross-modal contrastive learning loss, and reconstruction loss. The training adopts a progressive strategy: first, the cross-modal contrastive learning loss is pre-trained, and then the mutual information minimization loss is introduced for fine-tuning.

[0293] Furthermore, the cross-modal consistency constraint generator is a joint latent space variational autoencoder (C-VAE), and the consistency discriminator is a binary classifier. During training, positive examples are real multimodal pairs, and negative examples are automatically constructed contradictory multimodal pairs. During generation, the C-VAE generates candidate missing modalities, and the discriminator filters the completion results with a judgment confidence exceeding the threshold; otherwise, it falls back to retrieval-based completion.

[0294] Furthermore, the determination method for joint missing data is as follows: based on a pre-trained language model, the metadata description of the missing field is encoded into a vector, and the cosine similarity is calculated. If the similarity exceeds a preset threshold, it is determined to be joint missing data.

[0295] Furthermore, the causal inference employs dual machine learning to estimate the average causal effect (ACE), and the adjusted parameters include at least one of the following: synchronization delay weight, association result confidence threshold, completion consistency check threshold, and C-VAE temperature coefficient; when the fusion confidence is lower than the threshold, the node with the largest and statistically significant ACE is selected as the root cause for parameter adjustment.

[0296] Beneficial effects:

[0297] This invention systematically solves the multimodal fusion problem of data middleware from four dimensions: resource scheduling, heterogeneous storage, semantic unification, and closed-loop self-optimization. It can be widely applied in retail, finance, government and enterprise, media, manufacturing and other fields, supporting high-value businesses such as cross-modal retrieval, intelligent recommendation, content generation, and intelligent decision-making. It has significant technological progress and industrial value, and can be quickly implemented and promoted.

[0298] Novelty analysis of the present invention

[0299] The claims of this invention are novel compared to existing technologies (including CN120781307A, CN120688537A, CN120724376A, CN121462608A, CN121542982A, CN121509246A, CN121350435A, CN121524948A, CN121350651A, and the Hadoop ecosystem solution), specifically in the following ways:

[0300] 1. Apply hierarchical reinforcement learning (HRL) to the multimodal data synchronization scheduling of the data platform, explicitly including high-level mode decision-making (stable / bursting / chasing) and low-level parameter output, and introduce mode switching penalty and predictive state (future volume prediction value).

[0301] 2. The concept of “joint missing” is proposed and a cross-modal consistency constraint generator (C-VAE+consistency discriminator) is designed, which is different from existing single-modal generation or retrieval completion (such as retrieval + diffusion generation in CN120724376A and retrieval-based completion in CN121350435A).

[0302] 3. Decoupled representation learning is introduced into multimodal fusion to explicitly separate shared semantics from modality-specific features and adopts mutual information minimization constraints, which is different from existing adversarial alignment (CN120781307A), fine-grained semantic decoupling (CN121350651A) or shallow splicing.

[0303] 4. Applying causal inference to the closed-loop feedback of a multimodal fusion system, the root cause is accurately located and parameters are adjusted by calculating the average causal effect (ACE), which is different from existing optimization methods that rely on simple retries, result feedback RL (CN120688537A), or collecting feedback data but without causal root cause location (CN121542982A).

[0304] The aforementioned technical features are not fully disclosed in existing patents and literature.

[0305] Inventiveness Analysis of the Invention

[0306] Non-obviousness argument:

[0307] (1) HRL scheduling is not a simple variant of single-layer RL.

[0308] When faced with sudden surges in traffic, those skilled in the art typically resort to scaling up, caching, or single-layer RL parameter tuning. Introducing hierarchical decision-making (higher layers determine the "mode," lower layers execute the "parameters") and adding mode-switching penalties aims to address the oscillation problem of RL strategies in non-stationary environments—a deep-seated challenge in real-world systems. Migrating HRL from robot control and game AI to synchronous scheduling in a data platform, and designing a predictive state space (future volume predictions), requires creative effort. Existing patents (CN120688537A, CN121462608A) only use single-layer RL; while CN121542982A uses LSTM to predict reserved resources, its scheduling is task-level rather than data-synchronization-level, and it does not involve a hierarchical decision-making structure.

[0309] Distinguishing features: It adopts a two-level structure of high-level mode decision-making + low-level parameter output, and introduces mode switching penalty in the reward function.

[0310] Non-obviousness argument: When faced with policy oscillation problems, those skilled in the art typically approach the issue by increasing the exploration rate, using experience replay, or adjusting the learning rate, rather than introducing a hierarchical decision-making structure. Migrating HRL from the domain of robot control to the domain of data synchronization and scheduling in a data platform, and specifically designing mode-switching penalties to address policy oscillations, requires overcoming technical biases.

[0311] Furthermore, existing predictions (such as the LSTM prediction in CN121542982A) are only used for reserving resources and belong to task-level scheduling. This invention is the first to directly encode fine-grained time-series prediction features (the volume of the next 1 minute) into the state space of the HRL high-level policy, making the scheduling decision change from "reactive" to "look-ahead". This is a qualitative change from task-level prediction to policy-level prediction, which further enhances non-obviousness.

[0312] (2) Specific applications of decoupled representation learning in multimodal fusion

[0313] Existing mainstream multimodal fusion methods (such as adversarial alignment in CN120781307A) do not perform explicit decoupling. While CN121350651A involves decoupling, it focuses on fine-grained semantics rather than the separation of modal sharing and unique features. This invention recognizes that "modal-unique features can interfere with shared semantic learning" and uses mutual information minimization for separation, representing a non-obvious approach to specifically address this problem.

[0314] Distinguishing features: Each modality's encoded features are decomposed into shared semantic vectors and modality-specific vectors. Only the shared semantic parts are fused, and mutual information minimization constraints are applied.

[0315] Technical problem: Traditional multimodal fusion methods do not distinguish between shared information and unique information, which leads to unique features interfering with the learning of shared semantics, resulting in poor interpretability of the fusion results.

[0316] Non-obviousness argument: Those skilled in the art generally believe that multimodal fusion should utilize all available information as much as possible. This invention takes the opposite approach, actively separating and discarding modality-specific features, and only fusing shared semantic parts. This requires overcoming the technical bias in the field ("more information is necessarily better").

[0317] (3) Causal inference feedback far exceeds conventional closed loop.

[0318] Existing technologies often provide feedback such as "low confidence → retry" or "adjust global learning rate," or, as in CN121542982A, only "periodically collect feedback data and optimize system parameters" without addressing root cause localization. This invention introduces a causal graph and Average Causality Effect (ACE), requiring intervention inference to distinguish between "synchronization delay" and "association error" as the true root cause. This involves deep expertise in causal inference and is not a conventional design.

[0319] Distinguishing features: Construct a cause-effect graph, locate the root cause by estimating the average causal effect (ACE), and adjust the parameters of the corresponding modules accordingly.

[0320] Technical problem: Traditional closed-loop feedback cannot distinguish the true causal contribution of each preceding step to the fusion quality, resulting in blind parameter tuning and limited effectiveness.

[0321] Non-obviousness argument: In multi-module collaborative systems, there are complex coupling relationships between parameters of each component. Simple statistical correlation cannot distinguish between "correlation" and "causation," which can easily lead to incorrect root cause judgments. Solving this problem using causal inference methods requires cross-domain technical integration capabilities, which goes beyond the scope of conventional system design.

[0322] (4) Consistency constraints for joint missing completion

[0323] Existing generative completion methods (such as CN120724376A) mostly rely on retrieval and diffusion generation, or, as in CN121350435A, primarily rely on retrieval completion. While CN121524948A involves augmentation of missing modal data, it does not define joint missing data and lacks a consistency discrimination mechanism. This invention proposes a joint latent space generation + consistency discriminator to address joint missing data, forcing cross-modal semantic similarity. This approach is technically complex but performs better than independent generation.

[0324] Distinguishing features: Clearly define joint missing scenarios, construct a C-VAE+ consistency discriminator, and enforce cross-modal semantic consistency of generated results.

[0325] Technical problem: Existing missing completion methods generate each modality independently when multiple modalities are missing at the same time, resulting in semantic contradictions between the generated cross-modal pairs.

[0326] Non-obviousness argument: Existing technologies mostly focus on single-modal missing data completion or data augmentation. This invention is the first to identify the special scenario of "joint missing data" and design a dedicated joint completion scheme, which requires overcoming the technical inertia of "treating multimodal missing data as independent problems and handling them separately".

[0327] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A smart-driven multimodal data fusion method for data platforms, characterized in that, The method includes the following steps: Step S1: Multimodal data standardization preprocessing The multimodal data is formatted, denoised, and its basic features are extracted, and predictive state input is provided for the HRL high-level strategy in step S2. Step S2: Intelligent Synchronization Scheduling Based on Hierarchical Reinforcement Learning (HRL) A two-layer decision-making architecture is adopted: the high-level strategy determines the mode, and the low-level strategy outputs parameters. Predictive states and mode switching penalties are introduced to dynamically adapt the synchronization strategy, avoid strategy oscillation, ensure timing consistency, and reduce CPU and bandwidth consumption. Specifically, the high-level DQN decides the macro mode every 10 seconds, including: stable, bursty, and catch-up. The state includes the volume prediction value for the next minute provided in step S1, and the reward includes the mode switching penalty. The low-level PPO outputs the synchronization method, synchronization interval, and resource weights every second, based on the mode output by the high-level strategy. The two are trained together through an Option-Critic architecture to output a synchronized and aligned multimodal data stream. Step S3: Intelligent Association Matching The synchronized and aligned multimodal data stream output in step S2 achieves multimodal data association through a combination of explicit ID matching and implicit semantic matching. Step S4: Joint missing item identification and cross-modal consistent completion Define the "joint missing" scenario and achieve completion through joint latent space generation and consistency discriminator to ensure cross-modal semantic consistency and support both generative and retrieval completion strategies; Step S5: Decoupling representation learning-driven two-layer fusion of unified semantic space A unified semantic space is constructed, and shared features and unique features of different modalities are explicitly separated. Only the shared parts are fused to improve the fusion accuracy and interpretability. Each modality extracts 512-dimensional features through an independent encoder, which are then decomposed into shared semantic vectors and modality-specific vectors by a decoupling module. Shared and unique features are separated by minimizing mutual information, and shared semantics are aligned through cross-modal comparative learning. Step S6: Verify the output and optimize the end-to-end feedback based on causal inference. By accurately locating the root cause of fusion quality problems through causal graphs, a synergistic improvement of the entire chain of synchronization, correlation, completion and fusion is achieved. When the fusion confidence is lower than the threshold, the average causal effect ACE of each preceding node is estimated by dual machine learning, the node with the largest ACE is selected as the root cause, the parameters are adjusted in a targeted manner and written back to the corresponding module to form a closed loop.

2. The intelligent-driven multimodal data fusion method for data middleware as described in claim 1, characterized in that, In S1, For structured data: perform the following preprocessing: Anomaly detection, unified coding, and numerical standardization; Missing value handling: Only mark the missing locations; Numerical volatility is extracted as a time series feature; Time series prediction module: For each data source, a lightweight LSTM prediction model is deployed; the specific design is as follows: Input / output definition: The sampling interval is 1 second; the model input is the actual data volume of the last 60 time points, and the output is the predicted average data volume for the next minute; Offline pre-training: Using 7 days of historical second-level data, the task is to predict the average value of the next 60 seconds based on the sequence of the past 60 seconds, and training is performed using MSE loss; Online Incremental Update: After going live, an online SGD update will be performed every 5 minutes; during the update, the current time will be collected. The previous 60 seconds of data points were used as input. and wait The actual average data volume will be obtained as the target value after seconds. This forms a set of supervised samples, and a gradient descent step is performed without reconstructing the entire model. Cold start: Before the new data source accumulates 60 historical time points, the LSTM cannot output a valid prediction value; during this period, the status input field "predicted volume value in the next minute" of the HRL high-level strategy in step S2 is filled with the current real-time volume; Inference call: Each time a prediction is needed, forward inference is performed using the data volume of the most recent 60 seconds as input. The online update of model parameters runs in parallel with the inference service, without affecting the availability of predictions. Output purpose: The predicted value serves as the state input for the HRL high-level policy in step S2, and is used for forward-looking pattern decision-making; For semi-structured data: perform the following preprocessing: Automatic parsing, redundancy removal, uniform JSON format, and preservation of update timestamps; Extract the sequence of adjacent update time intervals and input it into the time series prediction module to capture changes in update frequency; For text data, the following preprocessing is performed: Denoising, word segmentation, and stop word removal; Extract keywords and their vector representations; The text length and update frequency are statistically analyzed and used as auxiliary features input into the time series prediction module. For image data, the following preprocessing is performed: Noise reduction, size uniformity, and pixel standardization; Extract basic visual features; Calculate the histogram fluctuation features of the image as a representation of the update frequency, and input them into the time series prediction module; For audio data, the following preprocessing is performed: Denoising, unified sampling rate, and spectrum standardization; Extract spectral features; Simultaneously extract audio energy, zero-crossing rate, and spectral centroid timing features, and input them into the timing prediction module.

3. The intelligent-driven multimodal data fusion method for data middleware as described in claim 1, characterized in that, In S2, the high-level strategy includes: State space: =[Current data volume, volume change slope, predicted volume value in the next minute, current synchronization delay, delay change trend, anomaly rate]; where, "predicted volume value in the next minute" is provided by the LSTM prediction module deployed in step S1; Action space: ; Reward function: ;in The switching penalty coefficient, ranging from 0.1 to 0.3, forces higher layers to avoid frequent switching. This is the weighting coefficient for the accumulated rewards at lower levels, with a value ranging from 0.8 to 0.

95. It is used to balance the current mode switching penalty with the future benefits of lower-level strategies. The cumulative reward for the lower-level strategy at the next moment is defined as the sum of the discounted rewards obtained by the lower-level strategy within 10 lower-level time steps over the next 10 seconds starting from the current higher-level decision moment. Discount factor , For the lower level The immediate reward for each step; the cumulative reward is calculated after the lower-level policy is executed and is used to evaluate the long-term effect of the pattern selected by the higher-level policy.

4. The intelligent-driven multimodal data fusion method for data middleware as described in claim 3, characterized in that, In S3, the low-level strategies include: State space: =[Update frequency, data volume, CPU utilization, bandwidth utilization, synchronization delay, anomaly rate, macroscopic pattern given by higher level]; All the above state variables are normalized to the range [0,1]. Among them, update frequency and data volume are normalized by maximum-minimum, CPU utilization and bandwidth utilization are obtained by dividing by 100 as a percentage, synchronization delay is normalized by a preset maximum delay threshold, and anomaly rate is a probability value between 0 and 1; the macroscopic pattern given by higher level is one-hot encoded and then concatenated with the other 6-dimensional continuous variables to form a 9-dimensional state vector; the update frequency is provided by the update frequency prediction value output by the time series prediction module in step S1, or it is calculated by the system in real time by counting the number of data arrivals in the last second; Action space: =[Synchronization Method, Synchronization Interval, Resource Allocation Weight]; where, synchronization method ∈ {stream synchronization, batch synchronization}, synchronization interval ∈ {100ms, 1min, 5min, 10min, 30min}, and resource allocation weight is the ratio of CPU and bandwidth allocation (weight). , ∈[0,1] and sum to 1); let The specific actions output by the lower-level strategy; Reward function: The "threshold" refers to the maximum allowed synchronization latency of the data source, set to 500ms for normal scenarios and 100ms for low-latency scenarios, based on the business SLA. Resource utilization rate is... That is, the weights of the action outputs are directly used to perform a weighted summation of CPU and bandwidth utilization, so that the strategy can learn a reasonable resource allocation. This is the anomaly rate coefficient, with a default value of 0.1; weight. , Dynamic adjustments based on macroeconomic models: In a stable mode , , ; in the sudden mode , , In pursuit mode , , ; Handling burst mode: In burst mode, the adjustment of the reward function weights mentioned above will naturally guide the lower-level strategy to choose a shorter synchronization interval and a higher bandwidth weight, rather than forcing the overlay strategy output.

5. The intelligent-driven multimodal data fusion method for data middleware as described in claim 4, characterized in that, In S2, the two are trained collaboratively through an Option-Critic architecture, and the output synchronized and aligned multimodal data stream includes: The high-level strategy uses DQN, and the Q function is defined as follows: The lower-level strategy uses PPO, a conditional strategy. ; , , Specifically, there are high-level state space, low-level state space, and low-level action space. Indicates a macroeconomic model; The two achieve collaborative training through the following mechanism: (1) The top leadership, based on the current situation Output macroscopic mode; (2) The lower layer executes multiple time steps in this mode to collect experience. , The corresponding rewards are calculated, and the cumulative rewards at lower levels are also calculated. ;in To perform the action The lower-level state in the next instant; (3) According to the high-level Update Q network, reward ; (4) The gradient is propagated back from the lower layer to the higher layer through the option value function; In this process, the implementation of experience playback and hybrid action space is as follows: Experience replay: To stabilize the training process and break the temporal correlation between data, separate experience replay buffers are maintained for high-level and low-level layers; the high-level buffer stores the complete experience tuple for each high-level decision. These are the state during high-level decision-making, the macro-mode chosen by the high-level, the reward obtained by the high-level after executing the mode, and the next high-level state after executing the mode; the high-level buffer uses random sampling, and a small batch of samples is drawn from it each time training is performed to update the Q network of DQN; The lower-level buffer stores the experience tuple for each lower-level time step. These are the state during low-level decision-making, the action performed at the low level, and the immediate reward obtained after performing the action. The calculation formulas are found in the low-level reward function, the next low-level state after performing the action, and the current high-level macroscopic pattern. The low-level buffer also uses random sampling and is used for updating the policy network and value network of the PPO algorithm. The high-level and low-level buffers are maintained independently. The high-level decision cycle is 10 seconds, and its number of experience samples is relatively small, so the capacity of the high-level buffer is set to 500. The low-level decision cycle is 1 second, and experience samples are generated quickly, so the buffer capacity is set to 10,000. During sampling, the low-level buffer only extracts experience tuples corresponding to the current high-level pattern, so that the training data distribution matches the current policy. Hybrid Action Space Implementation: The discrete and continuous actions output by the parameterized action space are stored in a low-level buffer along with the reward signal for joint optimization of the policy network. Using the parameterized action space method, the synchronization mode and synchronization interval of discrete actions are output through softmax, while the resource weights of continuous actions are output through Gaussian distribution. Discrete and continuous actions share the underlying network and are jointly optimized through separately calculated losses. Specifically, the policy network outputs the logits of discrete actions and the mean and variance of continuous actions. The total loss is the sum of the cross-entropy loss of discrete actions and the negative log-likelihood loss of continuous actions. The shared network parameters are jointly updated through backpropagation. Synchronous execution: Real-time data uses streaming synchronization, and batch data uses adaptive batch synchronization, with the coordinator ensuring time consistency; in burst mode, due to the guidance of the reward function weight, the synchronization interval of the policy output is usually short, and the bandwidth weight is high; in pursuit mode, the policy prioritizes delay. Anomaly handling: An autoencoder is used to identify anomalies in real time. The autoencoder takes synchronization delay, data volume, and anomaly rate as temporal characteristics as inputs, and an anomaly is determined when the reconstruction error exceeds a threshold. The anomaly rate output by the autoencoder is used as a low-level state. The "abnormality rate" variable is input to form a closed loop.

6. The intelligent-driven multimodal data fusion method for data middleware as described in claim 1, characterized in that, S3 includes: Explicit association matching: Unified association ID generation: For each data source, extract the field in its metadata that can uniquely identify the data entity; if a data source lacks an explicit ID, perform MD5 hashing on its key fields to generate a 128-bit fixed-length ID. Hash index construction: Store all extracted association IDs in a hash table, with the ID value as the key and the data source identifier and data location pointer as the value; when new data arrives, calculate its ID and look it up in the hash table. If a match is found, establish the association directly. Confidence setting: Explicit association is based on exact match, with a confidence level of 1.0; Implicit semantic association matching Unified semantic feature extraction: Different pre-trained encoders are used for different modalities, and the outputs are mapped to the same dimensional space. Text: Using the BERT-base model, the 768-dimensional output of the [CLS] token is taken and reduced to 512 dimensions through a fully connected layer; Image: Visual encoder using CLIP, outputting 512-dimensional features; Audio: The Wav2Vec2.0 model is used, and attention pooling is applied to the audio segments to output 768 dimensions, which are then reduced to 512 dimensions by a fully connected layer; Structured data: Numerical fields are normalized and concatenated, categorical fields are encoded using an embedding layer, and then encoded to 512 dimensions through two fully connected networks; Vector index library construction: For historical data that has been associated, use the FAISS library to build an IVF index for its 512-dimensional semantic feature vectors. The index parameters are set to the number of cluster centers nlist=100 and the number of probe clusters nprobe=10, and are dynamically adjusted according to the amount of historical data. Implicit association retrieval: For new data to be associated, extract its 512-dimensional semantic features as a query vector, retrieve the top-5 most similar historical records in the FAISS index, and calculate the cosine similarity of each record as the implicit association confidence score. Association Result Fusion and Confidence Determination (1) Candidate set construction: If an explicit association is found, generate a candidate (ID_explicit, confidence = 1.0); Add all the Top-5 results (ID_implicit_i, confidence = sim_i) returned by implicit retrieval to the candidate set; (2) Decision-making rules: No candidates: If the explicit association fails and the implicit search yields no results, it is marked as "association failed" and proceeds to the missing completion process in step S4. Single candidate: If there is only one candidate, the result is adopted directly; Multiple candidates: Select the candidate with the highest confidence; if the highest confidence is tied, the explicit association result is preferred. (3) Manual review and processing: Two confidence thresholds are defined: an automatically adopted threshold and a manually reviewed lower bound. These two thresholds are collectively referred to as the confidence thresholds for association results. Automatic adoption: If the confidence level of the selected candidate is ≥0.95, the association result is automatically adopted; Manual review: If 0.85 ≤ confidence level < 0.95, the result is transferred to the manual review queue; the result of the manual review is used as a feedback signal to input the cause-effect graph. Trigger completion: If the confidence level of the selected candidate is <0.85, it is marked as "association questionable" and the missing completion process in step S4 is triggered; Result storage and traceability: The final association results are stored in a relation table, with fields including: source data identifier, target association ID, association method, confidence level, and timestamp. Establish a complete traceability chain from raw data to related results, with optional storage of key intermediate results; Finally, the association result is passed to step S4 in the form of a message queue; step S4 will determine whether completion needs to be performed and the priority of completion based on this result.

7. The intelligent-driven multimodal data fusion method for data middleware as described in claim 1, characterized in that, S4 includes: Joint missing data determination method: The metadata description of each missing field is encoded into a vector using a pre-trained language model, and the cosine similarity is calculated; if the similarity exceeds a preset threshold, it is determined to be joint missing data; this threshold is determined based on the similarity distribution statistics of positive and negative samples in historical data, and is adaptively adjusted according to the business scenario; Missing data identification: Rules + autoencoder identify missing data types, which are categorized as follows; (1) Rule recognition: Check whether each modal field is NULL, empty string, NaN, or empty file, and output the missing type: Partial missing data: Some data for a single modality is missing; Completely missing: All data for a certain modality is missing; (2) Autoencoder recognition: Train an autoencoder with complete data, input the test data, and calculate the reconstruction error; if the error exceeds the preset threshold, it is judged as abnormal missing; the threshold is set to the 95th percentile of the reconstruction error of the training set, and is dynamically adjusted according to the false alarm rate during online operation; the adjustment method is: if the false alarm rate exceeds 5%, the threshold is increased to the 98th percentile; if the false alarm rate exceeds 5%, it is decreased to the 90th percentile. (3) Joint missing data determination: If the rule identifies multiple modalities as missing, and the BERT cosine similarity of the metadata of each missing field is >0.7, then it is marked as joint missing data; The impact of missing data is quantified using a combination of AHP and entropy weighting. (1) Use AHP to have business experts determine the weights of each business indicator; (2) Use the entropy weight method to determine the weights of data features based on the statistical characteristics of historical missing data; (3) Weighted calculation of the impact of missing values ​​on the score, mapped to a four-level priority: Level 1: Those affecting core business metrics should be prioritized for completion, and the system should allocate more resources to them. Level 2: Affects secondary indicators; Level 3: Does not affect core business but affects data integrity; Level 4: No actual business impact; completion can be delayed or only marked. Completion method selection strategy: Generative-first approach: Suitable for business scenarios that allow for a certain degree of fictitiousness, it prioritizes generation with cross-modal consistency constraints and controls the degree of freedom of generation by adjusting the threshold of the consistency discriminator; Search-based priority: For scenarios with strict requirements on data authenticity, a search is forced from the standard library. If the data cannot be found, it is marked as missing and no data is generated. Hybrid strategy: For other business scenarios, first try generative completion. If the consistency discriminator still fails to pass the threshold after multiple samplings, it will automatically fall back to retrieval completion. Cross-modal consistency constraint generator: Encodes available multimodal features into the same joint latent space, samples from it, and decodes the feature representation of the missing modality; At the same time, a discriminator is introduced to force the generated missing modality to maintain high semantic consistency with the existing modality; Through this mechanism of "joint generation + consistency constraint", the completion result is ensured to be coordinated and unified at the cross-modal level; (1) Joint latent space variational autoencoder C-VAE Encoder: Inputs available multimodal features, outputs latent space The mean and variance; Decoder: The latent variable z obtained by sampling from the joint latent space variational autoencoder C-VAE is passed through the decoder network to generate the feature vector of the missing mode, thereby completing the missing data; Training data: Use complete, semantically aligned multimodal pairs; if complete samples are scarce, use semi-supervised learning: transfer parameters from pre-trained models in similar domains, and then fine-tune with a small number of complete samples; (2) Consistency Discriminator Structure: Binary classifier MLP: Input is the concatenation of features from generated cross-modal pairs, output is the consistency probability; Training data: Positive examples are real multimodal pairs; negative examples are automatically constructed contradictory pairs, which are constructed by randomly selecting images and texts from different samples in the training set and combining them; or by using an independently trained modality generator to generate each modality separately and then randomly pairing them. Usage: After C-VAE generates candidate missing modalities, the discriminator evaluates their consistency with existing modalities; only results with a discriminant confidence score > 0.95 are retained; if multiple sampling attempts fail, it reverts to retrieval-based completion. Strong consistency check: After completion, use a pre-trained model to calculate cross-modal semantic similarity; if the similarity is <0.95, reject the completion result; if a hybrid strategy is used, fall back to retrieval-based completion; if retrieval-based completion also fails, mark it as "completion failed" and submit it for manual review.

8. The intelligent-driven multimodal data fusion method for data middleware as described in claim 1, characterized in that, S5 includes: Model architecture: Encoder: Each modality uses an independent pre-trained encoder, with a uniform output dimension of 512. Decoupling module: The 512-dimensional encoded output of each modality is passed through a decomposition network, which outputs a shared semantic vector. and mode-specific vectors The decomposition network structure consists of two fully connected layers; the first layer: 512→1024; the second layer: 1024→512; the output 512 dimensions are split into 256 dimensions. And 256-dimensional ; Decoder: From The original encoded features are reconstructed using a decoder, and MSE loss constraint is applied. Loss function and training strategy: (1) Decompose constraint loss Minimize Mutual Information (MINE): Minimize and Mutual information between them ensures the separation of shared and unique information; the mutual information neural estimator MINE is used to estimate mutual information. And minimize; Reconstruction loss MSE: from The original features are reconstructed by the decoder, and the loss is ,in The original feature vector output by the encoder. The feature vector reconstructed by the decoder; Cross-modal contrastive learning loss InfoNCE: different modalities They should have high similarity; positive example: different modalities of the same data sample. Negative examples: different samples Loss is recorded as ; Masked Modeling Loss (MLM): Randomly masks the input features of a particular modality, requiring the model to extract features from other modalities. The system predicts blocked content to enhance the robustness of cross-modal semantic understanding; the loss is denoted as... ; (2) Multi-target loss balancing and progressive training: The total loss is: The progressive training strategy is as follows: Phase 1: The first 50 epochs, using only Minimize mutual information without using it; make each modality Initial alignment; Phase 2: Starting from the 51st epoch, a mutual information minimization term is added with an initial weight of 0.1; simultaneously, the weights are dynamically adjusted. The weights are decreased by 0.05 every 10 epochs from 0.4 until they reach 0.2; the mutual information weights are increased by 0.05 every 10 epochs from 0.1 until they reach 0.3; training continues until the loss converges on the validation set. Fusion layer: First layer: Calculate each mode Attention weights; calculate the first Modal attention score: Among them, the weight matrix bias Attention vector , The attention dimension is hidden; then the normalized weights are calculated using the softmax function: The final weighted summation yields the first layer of fused features: Second layer: As a query, each modality As keys and values, they are input into the Transformer decoder, which outputs a final unified semantic vector. Training data and initialization: Each modal encoder is initialized using publicly available pre-trained weights; the decoupling module, decoder, and fusion layer are randomly initialized using Xavier; end-to-end fine-tuning and optimization are performed using real enterprise business data. .

9. The intelligent-driven multimodal data fusion method for data middleware as described in claim 1, characterized in that, S6 includes: Fusion quality verification: Calculate the following three metrics: (1) Fusion confidence during offline evaluation: defined as the cosine similarity between the output unified semantic vector and the standard semantic vector in the validation set; quality indicators during online operation: if real-time annotations cannot be obtained, the model embedded confidence is used as a proxy indicator, which is positively correlated with the offline cosine similarity. (2) Association error rate: Regular sampling is conducted for manual review, and the deviation ratio between the association results and the manual annotation is calculated; the mean of the association confidence is used as a proxy indicator in online feedback; as a node variable in the causal graph, it participates in the average causal effect ACE estimation to determine whether the association error has a significant causal impact on the fusion confidence. (3) Completion failure rate: The proportion of completion results rejected by consistency check in step S4 to all completion requests; As an intermediate node in the cause-effect graph, it connects completion parameters and fusion confidence to help locate the root cause of the completion process; The calculation results of the three indicators are all written to the system log for offline analysis and cause-effect graph updates; Causal inference feedback module: (1) Construction of cause-effect graph Cause-effect graph nodes include: synchronization delay (sync_delay), synchronization anomaly rate (sync_anomaly), association error rate (link_error), completion generation parameters, completion consistency threshold (consistency_th), completion failure rate (completion_fail_rate), and fusion confidence (fusion_conf). The edges of the cause-effect graph represent the causal direction, and the initial structure is predefined based on domain expert knowledge; After accumulating 1000 new fused samples, the current causal graph is first used for root cause localization, and then the PC algorithm is used to learn and correct the structure of the causal graph to ensure that the causal relationship is continuously updated as the system runs. If the sample accumulation rate is too fast, the execution frequency of the PC algorithm will be suppressed by time interval limitation to control the system load. The PC algorithm uses Fisher's Z test with a significance level set to 0.01, and continuous variables are first transformed by Gaussian kernel. The causal graph is updated using the most recent 2000 samples. Root cause localization; when the fusion confidence is below a threshold, the average causal effect (ACE) of each preceding node on the fusion confidence is estimated using the most recent 1000 fusion samples; a dual machine learning approach is employed, with the following specific steps: Step 1: Data Preparation: Collect recent data. One fusion sample, Record each candidate root cause node The value and fusion confidence Candidate root cause nodes include: synchronization delay, synchronization anomaly rate, association error rate, C-VAE temperature coefficient, and completion consistency threshold. Step 2, Residual Calculation: For each candidate root cause node Fit using a machine learning model: Use all nodes except T as covariates. predict ; Using the same covariates predict ; in In the cause-and-effect diagram, except and All nodes outside; calculate residuals: ， Step 3, ACE estimation: linear regression ACE estimate is obtained ;in This is the average causal effect (ACE). Step 4, Significance Verification: Calculate using the bootstrap method. The 95% confidence interval; if the confidence interval does not contain 0 and the p-value after Bonferroni correction is <0.05, then the node is considered to have a significant causal effect on the fusion confidence. Parameter tuning; select the node with the largest and statistically significant ACE as the root cause, and then tune the parameters for that node: The root cause is synchronization delay: Increase the delay weight in the low-level policy reward function. Furthermore, the high-level reward function was adjusted to add a delay-related penalty, guiding the high-level strategy into "chasing mode" rather than forcibly switching modes; The root cause is the synchronization anomaly rate: increase the sensitivity of the autoencoder anomaly detection, or trigger link switching; Root cause of association error rate: Automatically adjust the lower bound of manual review in the association result fusion decision to allow more low-confidence associations to enter the completion process, rather than increasing the proportion of manual review; The root cause is to complete the generated parameters: reduce the temperature coefficient, or tighten the consistency check threshold; Root cause: Completion consistency threshold: dynamically adjust the threshold to balance completion pass rate and accuracy; All adjusted parameters are written back to the corresponding modules in steps S2, S3, S4, and S5, and the adjustment log is recorded. Feedback signal quantization, transmission, and closed-loop processing; adjusted parameters are written back to each module via configuration center or direct API call; the time, root cause, adjustment content, and fusion confidence before and after each feedback are recorded for subsequent causal graph updates and effect evaluation; after every 1000 feedbacks, the PC algorithm is re-executed to update the causal graph structure, and the new graph is used for subsequent ACE estimation.

10. A smart-driven multimodal data fusion system for data platforms, characterized in that, The system consists of 8 layers, with data flow and control flow interacting between the layers through message queues and RPC calls; Layer 1: Data source layer; stores multimodal raw data, pushes data to the preprocessing layer via Kafka, and receives processing confirmation signals; Layer 2: Preprocessing and predictive feature extraction layer; performs format unification, noise reduction, and missing data labeling; deploys the LSTM prediction module to output the predicted data volume for the next minute; the standardized data and predicted values ​​are then passed to subsequent layers. Layer 3: Intelligent Synchronization Scheduling Layer; dynamically adjusts synchronization interval and resource allocation weights based on hierarchical reinforcement learning (HRL); receives prediction values ​​from Layer 2 as input to higher-level states, outputs synchronized data streams to the association layer, and reports the synchronization status to the feedback optimization layer. Layer 4: Association Matching Layer; performs explicit ID matching and implicit semantic matching; outputs association results and 512-dimensional semantic feature vectors of each modality to the missing feature completion layer; Layer 5: Missing Completion Layer; Identifies joint missing data and uses C-VAE + consistency discriminator for generative or fallback retrieval-based completion; Outputs the completed data and optional latent space vectors to the semantic fusion layer; Layer 6: Semantic fusion layer; performs decoupled representation learning, decomposes the features of each modality into shared semantic vectors and unique vectors, and outputs a unified semantic vector through two-layer fusion; receives the completed data and outputs the fusion result to the layer; Layer 7: Verification output and feedback optimization layer; real-time calculation of fusion confidence, association error rate, and completion failure rate; when the fusion confidence is lower than the threshold, the causal inference module is triggered: using the most recent 1000 samples to estimate the average causal effect ACE of each preceding node, locate the root cause and generate a targeted adjustment signal; the adjustment signal is written back to layers 3-6; qualified fusion results are output to the service interface and storage; unqualified results can trigger a retry; Layer 8: Monitoring and maintenance layer; collects resource usage, abnormal logs, and historical fusion effects of each layer in real time; provides data to Layer 7 for cause-effect graph updates; issues critical alarms and supports link switching.

Images