Cloud-based multi-source data fusion processing method, apparatus and storage medium

By adopting a cloud-based distributed data fusion architecture and closed-loop feedback mechanism, the problems of processing latency and accuracy in multi-source data fusion are solved, and efficient and accurate data fusion and decision support are achieved.

CN123069263APending Publication Date: 2026-11-10SHENZHEN SHUMEI BIG DATA TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611238332.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-14
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing multi-source data fusion methods are prone to excessive processing latency due to single-point bottlenecks when facing large-scale, high-concurrency data streams. Furthermore, traditional fusion algorithms fail to fully consider the differences in the reliability and timeliness of data sources and lack a closed-loop feedback mechanism, resulting in unstable accuracy of fusion results.

Method used

A cloud-based distributed data fusion architecture is adopted. Computing nodes are dynamically allocated through a virtualized resource pool, and parallel processing is performed using a task scheduling strategy based on a genetic algorithm. A pre-trained quality assessment model is used to evaluate the reliability and timeliness of data in real time, dynamically generating adaptive weighting coefficients. Local fusion is performed by combining weighted averaging and Kalman filtering. Global consistency verification is performed on the master node in the cloud through a Bayesian network to generate high-quality global fused data. The strategy is then adjusted in reverse to adapt to changes in data source quality and business needs.

Benefits of technology

It significantly improves the processing efficiency and accuracy of multi-source data fusion, can adapt to changes in data source quality, maintain stable fusion performance and decision accuracy, and realizes the long-term adaptive capability of the fusion system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069263A_ABST
    Figure CN123069263A_ABST
Patent Text Reader

Abstract

This application relates to the field of data processing technology, and provides a method, apparatus, and storage medium for multi-source data fusion processing based on cloud computing. The method includes: standardizing and preprocessing raw data from multiple heterogeneous data sources to obtain standardized data with a unified format; allocating computing nodes in a distributed data fusion engine within a cloud computing environment, and distributing several sub-tasks to the corresponding computing nodes for parallel processing; for each sub-task on a computing node, using a pre-trained quality assessment model and a fusion algorithm combining weighted averaging and Kalman filtering to locally fuse the standardized data in the sub-task, obtaining a local fusion result; aggregating the local fusion results output from all computing nodes to a cloud master node to generate global fused data; inputting the global fused data into a preset decision analysis model, and adjusting the quality assessment model and the update strategy of the adaptive weighting coefficients based on the feedback results of the analysis report.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of big data processing and distributed computing, and in particular to a method, apparatus and storage medium for multi-source data fusion processing based on cloud computing. Background Technology

[0002] With the rapid development of the Internet of Things (IoT), the Industrial Internet, and smart terminals, various business systems have generated massive amounts of multi-source, heterogeneous data. These data sources are diverse, including sensors, log files, database records, and social media, and vary greatly in format, accuracy, timeliness, and reliability. How to efficiently and accurately integrate this multi-source, heterogeneous data to support upper-level decision analysis has become a core challenge in the field of big data processing.

[0003] Most existing multi-source data fusion methods employ a centralized architecture, aggregating all raw data onto a single server or cluster for unified processing. When faced with large-scale, high-concurrency data streams, these methods are prone to high processing latency due to single-point bottlenecks, making it difficult to meet real-time requirements. Furthermore, traditional fusion algorithms typically use fixed weighting rules or simple arithmetic averages, failing to adequately consider the differences in reliability and timeliness of different data sources at different times, leading to unstable accuracy of the fusion results. More critically, existing solutions lack a closed-loop feedback mechanism for the fusion process. If the quality of the data source drifts or the environment changes, the fusion strategy cannot be automatically adjusted, resulting in decision-making biases. Summary of the Invention

[0004] This application provides a cloud computing-based multi-source data fusion processing method, apparatus, and storage medium that integrates multi-source data fusion, temporal feature extraction, and reinforcement learning to achieve intelligent adaptive optimization of enterprise data services.

[0005] On the one hand, this application provides a multi-source data fusion processing method based on cloud computing, the method comprising:

[0006] Step S1: Receive raw data from multiple heterogeneous data sources and perform standardized preprocessing on the raw data to obtain standardized data with a unified format;

[0007] Step S2: Build a distributed data fusion engine in the cloud computing environment. The distributed data fusion engine dynamically allocates computing nodes through a virtualized resource pool, and divides the standardized data into several sub-tasks according to the real-time load and data volume of each data source, and distributes the several sub-tasks to the corresponding computing nodes for parallel processing.

[0008] Step S3: For each subtask on the computing node, the data credibility and timeliness of the corresponding data source are evaluated in real time using a pre-trained quality assessment model. Based on the evaluation results, adaptive weighting coefficients are dynamically generated. Then, a fusion algorithm combining weighted averaging and Kalman filtering is used to locally fuse the standardized data in the subtask to obtain the local fusion result.

[0009] Step S4: Aggregate the local fusion results output by all computing nodes to the cloud master node, and perform conflict detection and correction on the local fusion results through a global consistency verification model based on Bayesian networks to generate global fusion data;

[0010] Step S5: Input the global fusion data into the preset decision analysis model, output an analysis report to guide business decisions, and adjust the update strategy of the quality assessment model and the adaptive weighting coefficients in reverse according to the feedback results of the analysis report.

[0011] On the other hand, this application provides a cloud computing-based multi-source data fusion processing apparatus, the apparatus comprising:

[0012] The preprocessing module is used to receive raw data from multiple heterogeneous data sources and perform standardized preprocessing on the raw data to obtain standardized data with a unified format.

[0013] The parallel processing module is used to build a distributed data fusion engine in a cloud computing environment. The distributed data fusion engine dynamically allocates computing nodes through a virtualized resource pool, and divides the standardized data into several sub-tasks according to the real-time load and data volume of each data source, and distributes the several sub-tasks to the corresponding computing nodes for parallel processing.

[0014] The fusion module is used to evaluate the data credibility and timeliness of the corresponding data source in real time for each subtask on the computing node using a pre-trained quality assessment model, and dynamically generate adaptive weighting coefficients based on the evaluation results. Then, a fusion algorithm combining weighted averaging and Kalman filtering is used to perform local fusion on the standardized data in the subtask to obtain the local fusion result.

[0015] The generation module is used to aggregate the local fusion results output by all computing nodes to the cloud master node, and to perform conflict detection and correction on the local fusion results through a global consistency verification model based on Bayesian network, thereby generating global fusion data.

[0016] The adjustment module is used to input the global fusion data into a preset decision analysis model, output an analysis report to guide business decisions, and adjust the quality assessment model and the update strategy of the adaptive weighting coefficients in reverse according to the feedback results of the analysis report.

[0017] Thirdly, this application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the technical solution of the cloud computing-based multi-source data fusion processing method described above.

[0018] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described cloud computing-based multi-source data fusion processing method.

[0019] As can be seen from the technical solutions provided in this application, on the one hand, by constructing a distributed data fusion engine, dynamically allocating computing nodes using a virtualized resource pool, and combining a task scheduling strategy based on a genetic algorithm, parallel processing of standardized data is achieved. This solution can adaptively divide sub-tasks according to the real-time load and data volume of each data source, effectively avoiding performance bottlenecks in centralized processing and significantly improving the processing efficiency and real-time response capability of large-scale multi-source data fusion. On the other hand, a pre-trained quality assessment model is introduced on each computing node to evaluate the data credibility and timeliness of the corresponding data source in real time, and dynamically generate adaptive weighting coefficients based on the evaluation results. On this basis, a fusion algorithm combining weighted average and Kalman filtering is used for local fusion. Since the weighting coefficients can be dynamically adjusted with changes in data source quality, this solution overcomes the shortcomings of traditional fixed-weight fusion methods that cannot adapt to fluctuations in data source quality, and improves the accuracy and robustness of local fusion results. On the third hand, all local fusion results are collected through a cloud master node, and a global consistency verification model based on Bayesian networks is used for conflict detection and correction. This model effectively identifies and corrects contradictory information between different computing nodes caused by differences in data sources or noise interference, thereby generating high-quality global fusion data and providing a reliable data foundation for subsequent decision analysis. Fourthly, the generated global fusion data is input into a pre-set decision analysis model, which outputs an analysis report. Based on the feedback from the analysis report, the quality assessment model and the update strategy of the adaptive weighting coefficients are adjusted in reverse. This closed-loop feedback mechanism enables the fusion system to continuously self-optimize, maintaining stable fusion performance and decision accuracy even in the face of dynamic changes in data source quality or adjustments in business needs, achieving long-term adaptive capability of the fusion method. In summary, the technical solution of this application improves the efficiency and accuracy of multi-source data fusion through dynamic task scheduling and closed-loop feedback optimization. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a cloud computing-based multi-source data fusion processing method provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the structure of the cloud computing-based multi-source data fusion processing device provided in the embodiments of this application;

[0023] Figure 3 This is a schematic diagram of the device provided in the embodiments of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] In this specification, adjectives such as "first" and "second" are used only to distinguish one element or action from another, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be construed as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0026] For ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.

[0027] Most existing multi-source data fusion methods employ a centralized architecture, aggregating all raw data onto a single server or cluster for unified processing. When faced with large-scale, high-concurrency data streams, these methods are prone to high processing latency due to single-point bottlenecks, making it difficult to meet real-time requirements. Furthermore, traditional fusion algorithms typically use fixed weighting rules or simple arithmetic averages, failing to adequately consider the differences in reliability and timeliness of different data sources at different times, leading to unstable accuracy of the fusion results. More critically, existing solutions lack a closed-loop feedback mechanism for the fusion process. If the quality of the data source drifts or the environment changes, the fusion strategy cannot be automatically adjusted, resulting in decision-making biases.

[0028] To address the aforementioned problems in existing technologies, this application proposes a multi-source data fusion processing method based on cloud computing, the flowchart of which is attached. Figure 1 As shown, it mainly includes steps S1 to S5, which are detailed below:

[0029] Step S1: Receive raw data from multiple heterogeneous data sources and perform standardized preprocessing on the raw data from multiple heterogeneous data sources to obtain standardized data with a unified format.

[0030] In practical applications, data sources may include various heterogeneous systems such as sensors, databases, log files, and API interfaces, each with different data formats, encoding methods, time granularities, and measurement units. Directly feeding such raw data into the fusion engine would lead to inconsistent parsing and computation in subsequent processing. Therefore, this step first performs standardization preprocessing on the raw data. Specifically, raw data from multiple heterogeneous data sources is received and standardized preprocessed to obtain standardized data with a unified format. The core of this step is to eliminate the heterogeneity of data sources, laying the foundation for subsequent parallel processing and fusion. Standardization preprocessing includes, but is not limited to: format conversion (e.g., converting JSON, XML, and CSV to an internal structured format), encoding standardization (e.g., UTF-8), timestamp alignment (e.g., converting to UTC millisecond-level timestamps), and numerical normalization (e.g., Min-Max scaling or Z-score normalization).

[0031] As one embodiment of this application, standardizing and preprocessing the raw data from multiple heterogeneous data sources to obtain standardized data with a unified format can be achieved through the following steps S11 to S13:

[0032] Step S11: Parse the metadata header of the original data and extract the data structure description information.

[0033] Metadata headers typically contain information such as data source, field names, data types, timestamp format, and encoding method. For example, for a piece of sensor data in JSON format, its metadata header might contain field definitions such as "sensor_id", "timestamp", "value", and "unit". By parsing the metadata header, the system can automatically identify the structural characteristics of the data, providing a basis for subsequent missing field identification and outlier detection. This process is similar to "schema parsing" in databases, but it is for streaming or batch heterogeneous data.

[0034] Step S12: Based on the data structure description information, identify missing fields and outliers in the original data, and perform interpolation completion and smoothing filtering operations.

[0035] After obtaining the data structure description, the system checks each data record one by one: if a field is empty or has a null value, it is marked as a missing field; if the value of a field is outside a reasonable range (for example, the temperature sensor reading is -200°C), it is marked as an outlier. For missing fields, interpolation completion methods are used (e.g., linear interpolation, Lagrange interpolation, or statistical imputation based on historical data); for outliers, they are first removed, and then smoothed by smoothing filters (e.g., median filtering, Gaussian filtering) to reduce the impact of noise.

[0036] Considering that traditional data completion methods (such as mean imputation and linear interpolation) do not take into account the inherent correlation between multiple data sources, the completion of missing fields in heterogeneous data sources is often inaccurate, resulting in low quality of the underlying data for subsequent fusion. Therefore, as an embodiment of this application, based on data structure description information, identifying missing fields and outliers in the original data and performing interpolation completion and smoothing filtering operations can be achieved through steps S121 to S123, as detailed below.

[0037] Step S121: Construct a data association graph, which is generated based on the historical co-occurrence frequency of multiple heterogeneous data sources.

[0038] In this embodiment, the data association graph is an undirected weighted graph where nodes represent different data sources or fields, and the edge weights represent the frequency with which two data sources or fields appear simultaneously in historical data. For example, if temperature sensor A and humidity sensor B report data simultaneously for 80% of the time period, the edge weight between them is 0.8. This graph reflects the implicit relationships between multi-source data, providing prior knowledge for missing value completion.

[0039] Step S122: Calculate the conditional probability distribution of the missing field based on the data association graph.

[0040] For a missing field, the system finds other fields connected to it in its association graph (i.e., fields with high co-occurrence frequency). Using the current known values ​​of these fields as conditions, the system calculates the conditional probability distribution of the missing field's various possible values ​​through Bayesian formulas or Markov network inference. For example, if temperature sensor T is missing, but humidity sensor H and pressure sensor P both have readings at the same time, then P(T|H,P) can be calculated based on the joint distribution of (T,H,P) in historical data.

[0041] Step S123: Sample from the conditional probability distribution to generate complete data, and write the complete data into the missing field.

[0042] A Monte Carlo sampling method (e.g., Gibbs sampling or rejection sampling) is used to extract a value from the conditional probability distribution obtained in step S122 as imputation data and write it into the missing field. This method preserves the original statistical properties and cross-source correlation of the data better than simple mean imputation or linear interpolation, thereby improving the quality of standardized data.

[0043] Step S13: Map the completed and filtered data onto a predefined standardized data model to generate standardized data.

[0044] A predefined standardized data model is a unified data structure template that specifies the names, data types, value ranges, and time formats of all fields. For example, the "temperature" field is uniformly named "temperature", the unit is degrees Celsius (°C), and the timestamp is uniformly a Unix millisecond integer. Through mapping operations, data from all data sources is converted into instances of this template, thus forming standardized data with a unified format for use in subsequent steps.

[0045] Step S2: Build a distributed data fusion engine in the cloud computing environment. The distributed data fusion engine dynamically allocates computing nodes through a virtualized resource pool, and according to the real-time load and data volume of each data source, it adopts a task scheduling strategy based on genetic algorithm to divide the standardized data into several sub-tasks, and distributes the sub-tasks to the corresponding computing nodes for parallel processing.

[0046] In traditional centralized data fusion architectures, all standardized data is transmitted to a single server for processing. When data volume surges, processing bottlenecks easily emerge, leading to a sharp increase in latency. Furthermore, the load and data volume of different data sources change dynamically over time, and a fixed number of computing resources cannot adapt to these fluctuations, resulting in either idle resources or insufficient processing capacity. Therefore, a distributed processing mechanism capable of elastic scaling and dynamic scheduling is urgently needed. To address these issues, this application adopts the following technical solution: A distributed data fusion engine is built in a cloud computing environment. This engine dynamically allocates computing nodes through a virtualized resource pool and, based on the real-time load and data volume of each data source, uses a task scheduling strategy based on a genetic algorithm to divide the standardized data into several sub-tasks, which are then distributed to corresponding computing nodes for parallel processing.

[0047] Specifically, the distributed data fusion engine is deployed on cloud computing infrastructure, utilizing containerization technologies (e.g., Docker, Kubernetes) to build a virtualized resource pool. This resource pool contains a set of elastically scalable computing nodes, each of which can be a virtual machine or a container instance. The engine continuously monitors the real-time load (e.g., data generation rate per second) and data volume (e.g., total size of data to be processed) of each data source, using this as input to drive a task scheduling strategy based on a genetic algorithm. A genetic algorithm is an optimization algorithm that simulates the natural evolutionary process and is suitable for solving complex combinatorial optimization problems. In this application, the genetic algorithm divides standardized data into several sub-tasks, each containing a certain number of data records or data blocks, and assigns them to specific computing nodes. The core steps of the algorithm are as follows:

[0048] Encoding: Encode a possible task partitioning scheme as a chromosome, where genes in the chromosome represent the computing node number to which each data block is assigned.

[0049] Fitness function: Taking into account the CPU utilization, memory usage, and network I / O latency of each computing node, the fitness value is calculated through a weighted combination. The higher the fitness, the better the partitioning scheme can achieve load balancing and efficient processing.

[0050] Selection, crossover, and mutation: Through operations such as roulette wheel selection, single-point crossover, and random mutation, a new generation of population is generated iteratively to gradually approach the optimal solution.

[0051] Termination condition: When the preset maximum number of iterations or the fitness value converges, output the individual with the highest fitness as the optimal subtask partitioning scheme.

[0052] After implementing this partitioning scheme, the engine distributes each subtask to the corresponding computing nodes, and each node processes the allocated data independently and in parallel, thereby making full use of the elasticity of cloud computing resources and significantly improving processing efficiency.

[0053] In one embodiment of this application, the division of standardized data into several sub-tasks using a task scheduling strategy based on a genetic algorithm can be achieved through steps S21 to S24, as detailed below:

[0054] Step S21: Initialize the population. Each individual in the population represents a partitioning scheme that divides standardized data into a set of subtasks.

[0055] The population size can be dynamically set according to the data scale and the number of computing nodes, for example, 50 to 200 individuals. Each individual is generated randomly, that is, a computing node is randomly assigned to each data block, thus forming an initial set of partitioning schemes. To ensure population diversity, heuristic methods can also be used to generate some excellent individuals (e.g., based on polling or minimum load priority strategies) to accelerate the convergence process.

[0056] Step S22: Calculate the fitness value of an individual. The fitness value is determined based on a weighted combination of the CPU utilization, memory usage, and network I / O latency of the computing node.

[0057] The design of the fitness function directly affects the search direction of the genetic algorithm. Specifically, for each partitioning scheme represented by an individual, the system simulates the execution of that scheme, estimates the CPU utilization, memory usage, and network I / O latency of each computing node, and then calculates a weighted sum according to preset weights (e.g., α=0.4, β=0.3, γ=0.3). The reciprocal or negative value of this weighted sum is used as the fitness value, so that the scheme with a more balanced load and lower latency has a higher fitness.

[0058] Specifically, the fitness value of an individual can be calculated through the following steps S221 to S223:

[0059] Step S221: Obtain the heartbeat packets of each computing node in the current cloud computing environment to monitor their liveness status.

[0060] Obtain heartbeat packets from each compute node in the current cloud computing environment to monitor their liveness. Each compute node periodically (e.g., every 1 second) sends a heartbeat packet to the master node, containing information such as node ID, timestamp, and current load. The master node maintains a heartbeat timeout timer; if it does not receive a heartbeat from a node for more than a preset threshold (e.g., 3 seconds), it determines that the node may have failed.

[0061] Step S222: If a computing node is detected to have timed out, it is removed from the available resource pool.

[0062] If a compute node's heartbeat times out, it will be removed from the available resource pool. Once a node is determined to be faulty, the master node will immediately remove it from the virtualization resource pool and will no longer assign new subtasks to it. At the same time, subtasks already assigned to that node but not yet completed will be marked as "pending reassignment" and await subsequent rescheduling.

[0063] Step S223: After removing the faulty node, recalculate the fitness value of the remaining individuals and increase the weight coefficient of network I / O latency.

[0064] After removing the faulty node, the fitness values ​​of the remaining individuals are recalculated, and the weighting factor for network I / O latency is increased. Since the node topology has changed, the original fitness values ​​are no longer accurate. Therefore, the system re-evaluates the fitness of each individual in the current population, while appropriately increasing the weighting factor for network I / O latency (e.g., from 0.3 to 0.5) to guide the genetic algorithm to prioritize partitioning schemes that reduce cross-node data transmission, thereby reducing communication overhead caused by node failures.

[0065] From the scheme for calculating the fitness value of individuals in the examples of steps S221 to S223 above, it can be seen that the scheduling strategy can adaptively avoid faulty nodes and dynamically adjust the optimization target (increase the network I / O latency weight) when the node topology changes, so that the partitioning scheme searched by the genetic algorithm is always oriented towards the currently available optimal resources.

[0066] Step S23: Iteratively update the population through selection, crossover, and mutation operations until the maximum number of iterations or fitness value converges.

[0067] The population is iteratively updated through selection, crossover, and mutation operations until the maximum number of iterations or the fitness value converges. Selection uses tournament selection or roulette wheel selection to retain individuals with high fitness; crossover randomly swaps parts of the genes of two parent individuals to produce offspring; mutation randomly changes the value of a gene (i.e., changes the allocation node of a data block) with a small probability. Through multiple generations of iteration, the average fitness of the population gradually increases, eventually converging to an approximately optimal solution.

[0068] Step S24: Determine the partitioning scheme corresponding to the individual with the highest fitness as the optimal subtask partitioning scheme, and execute the partitioning.

[0069] The partitioning scheme corresponding to the individual with the highest fitness is determined as the optimal subtask partitioning scheme, and the partitioning is executed. After the termination condition is met, the individual with the highest fitness is selected from the final population, and the mapping relationship between each data block and the computing node is decoded. The engine divides the standardized data into several subtasks according to this mapping relationship and distributes them to the corresponding computing nodes to start parallel processing.

[0070] Step S3: For each subtask on the computing node, the data credibility and timeliness of the corresponding data source are evaluated in real time using a pre-trained quality assessment model. Based on the evaluation results, adaptive weighting coefficients are dynamically generated. Then, a fusion algorithm combining weighted averaging and Kalman filtering is used to locally fuse the standardized data in the subtask to obtain the local fusion result.

[0071] In distributed parallel processing architectures, each computing node is responsible for processing subtasks from different data sources. However, the data quality from different data sources varies considerably and changes dynamically over time. For example, some sensors may experience reading drift due to aging or environmental interference, some network links may experience packet loss or latency, and some data sources may be maliciously tampered with. If all data sources are fused indiscriminately, low-quality data will severely contaminate the fusion results, leading to decision-making errors. Traditional fusion methods typically employ fixed-weighted averaging or simple sorting based solely on timestamps, which cannot adapt to the dynamic changes in data source quality. While Kalman filtering can handle temporal noise, it assumes that all observations have the same noise covariance, which does not align with the reality of multi-source heterogeneous data. Therefore, a method that can perceive data source quality in real time and dynamically adjust the fusion strategy is urgently needed.

[0072] To address the aforementioned issues, the technical solution adopted in this application is as follows: For each subtask on a computing node, a pre-trained quality assessment model is used to evaluate the data reliability and timeliness of the corresponding data source in real time. Adaptive weighting coefficients are dynamically generated based on the evaluation results. Then, a fusion algorithm combining weighted averaging and Kalman filtering is used to locally fuse the standardized data in the subtask, obtaining a local fusion result. This solution, on the one hand, uses a pre-trained quality assessment model to quantify the reliability and timeliness of each data source in real time, enabling the fusion system to "distinguish between good and bad." The dynamic generation of weighting coefficients based on the evaluation results gives higher weights to high-quality data and automatically suppresses low-quality data, making the fusion result robust to degradation from individual data sources. On the other hand, combining adaptive weighting with Kalman filtering utilizes the smoothing ability of Kalman filtering for temporal noise and introduces prior knowledge of data source quality, ensuring high accuracy of the local fusion result in dynamic environments.

[0073] The key technical features of this step are described in detail below.

[0074] 1) Quality Assessment Model: This is a pre-trained machine learning model used to quantify the credibility and timeliness of each data source at the current moment. This model can be trained based on various features, including historical accuracy and data integrity, which will be explained in detail below:

[0075] 1.1) Historical accuracy: The degree to which the data from this data source matches the true value (or high-precision reference value) over a past period of time.

[0076] 1.2) Data integrity: The proportion of missing fields in the data reported by this data source.

[0077] 1.3) Outlier rate: The frequency at which data from this data source is marked as an outlier.

[0078] 1.4) Timestamp consistency: The degree of deviation between the data reporting time and the actual time.

[0079] 1.5) Signal-to-noise ratio: The ratio of signal strength to background noise (applicable to sensors).

[0080] Quality assessment models can employ classification or regression algorithms, such as logistic regression, support vector machines, random forests, or lightweight neural networks. During training, historical data and their corresponding ground truth labels (e.g., manually labeled data quality levels) are used as samples to learn the mapping from features to quality scores. In actual operation, the model receives real-time features from the current data source and outputs two key metrics: data credibility (a score between 0 and 1, where 1 represents complete credibility) and timeliness (a metric measured in data latency, with lower values ​​being better).

[0081] 2) Adaptive Weighting Coefficient: Based on the reliability and timeliness of the quality assessment model output, the system dynamically generates weighting coefficients for each data source. The formula for calculating the weighting coefficients can be designed as follows:

[0082]

[0083] in, The weighting coefficients for the i-th data source are... For credibility score, The timeliness score (timeliness can be converted into a score inversely proportional to delay). and Here, n is the preset balancing parameter, and n is the total number of data sources participating in the fusion on this computing node. Through this formula, high-quality, timely data sources receive greater weight, while low-quality or outdated data sources are automatically suppressed.

[0084] 3) Fusion Algorithm Combining Weighted Average and Kalman Filtering: After obtaining the adaptive weighting coefficients, this application employs a fusion algorithm combining weighted average and Kalman filtering to locally fuse the standardized data in the subtask. The specific process is as follows:

[0085] 3.1) Weighted Average: For observations from multiple data sources at the same time, a weighted average is calculated using adaptive weighting coefficients, which serves as the initial fusion estimate for that time. This step enables rapid fusion of multi-source information and utilizes weights to suppress the influence of low-quality data.

[0086] 3.2) Kalman Filtering: The weighted average is used as the observation value for Kalman filtering. This is combined with the system's state transition equation (a dynamic model describing data changes) and the observation noise covariance matrix (adjusted according to the weighting coefficients; higher weights result in lower observation noise) to perform prediction-update iterations. Kalman filtering can smooth out time-series noise and provide a measure of uncertainty in state estimation (covariance matrix).

[0087] 3.3) Output Local Fusion Results: The output of the Kalman filter is the local fusion result of the computation node at the current time, including the state estimate and the corresponding uncertainty. This result contains both weighted fusion information from multiple sources and has undergone time-series filtering, resulting in high accuracy and stability.

[0088] Through the above fusion algorithm, each computing node independently processes its assigned subtasks, generating high-quality local fusion results, which provide reliable input for subsequent global consistency verification.

[0089] Step S4: Aggregate the local fusion results output by all computing nodes to the cloud master node, and use a Bayesian network-based global consistency verification model to detect and correct conflicts in the local fusion results, generating global fusion data.

[0090] In step S3, each computing node independently completed the local fusion of its assigned sub-tasks and output its own local fusion results. However, since the data sources processed by different computing nodes may overlap or be related, and there are differences in processing latency and data source quality among the nodes, these local fusion results may have logical conflicts or inconsistencies. For example, temperature sensor nodes in two adjacent areas may report significantly different temperature values ​​at the same time; or, different time segments from the same data source may be assigned contradictory weights in two nodes. If conflicting local results are directly merged into global data, the final decision analysis report will lose credibility. Traditional methods often use majority voting or a simple timestamp priority principle to handle such conflicts. When the number of data sources is small or most data sources are faulty simultaneously, majority voting may adopt incorrect viewpoints; timestamp priority assumes that the latest data is the most reliable, ignoring the differences in the credibility of the data sources themselves, which may lead to "garbage in, garbage out". Therefore, there is an urgent need for a mechanism that can model causal relationships, quantify uncertainty, and intelligently resolve conflicts.

[0091] To address the aforementioned issues, this application employs the following technical solution: The local fusion results output from all computing nodes are aggregated to a cloud master node. A global consistency verification model based on Bayesian networks is used to detect and correct conflicts in the local fusion results, generating globally fused data. In this solution, Bayesian networks can explicitly model the conditional dependencies between local fusion results and quantify the confidence propagation paths between different node outputs using conditional probability tables, thus providing a theoretical basis for conflict detection. When a state difference exceeds a threshold, Bayesian inference can be used to locate the most likely conflict source node and take targeted measures (downweighting or retransmission) instead of blindly discarding data. By coordinating the contradictions between nodes through probabilistic inference, the final globally fused data is logically self-consistent, providing a reliable data foundation for subsequent decision-making.

[0092] Specifically, the cloud master node collects the local fusion results from all computing nodes and inputs them into a pre-built Bayesian network model. A Bayesian network is a probabilistic graphical model that represents the conditional dependencies between random variables using a directed acyclic graph. In this application, the nodes of the Bayesian network represent the local fusion results of each computing node, directed edges represent causal or correlational relationships between results from different nodes, and a conditional probability table quantifies the probability of child nodes taking each value given a parent node's state. Through Bayesian inference, the system can determine which local results conflict, locate the most likely source of conflict based on probability propagation, and then take corrective measures (e.g., weight reduction or requesting retransmission), ultimately generating globally consistent fused data.

[0093] As an embodiment of this application, conflict detection and correction of local fusion results using a Bayesian network-based global consistency verification model can be achieved through the following steps S41 to S44:

[0094] Step S41: Construct a conditional probability table to represent the causal relationships between local fusion results.

[0095] The conditional probability table is a core component of Bayesian networks, defining the probability distribution of each node in the network under various combinations of values ​​of its parent node. In this application, the construction of the conditional probability table is based on historical data and domain knowledge. For example, if node A (processing data from sensor X) and node B (processing data from sensor Y) have the same output result 90% of the time historically, then in the conditional probability table, when node A takes the value v, the probability that node B takes the same value v is set to 0.9, and the probability that it takes a different value is set to 0.1. The conditional probability table can be automatically obtained from historical data through offline learning (such as maximum likelihood estimation or the EM algorithm), or it can be set by domain experts based on experience.

[0096] Step S42: Calculate the state difference between any two local fusion results at the same time.

[0097] State differences are used to quantify the degree of inconsistency between two local fusion results. For numerical data, state differences can be absolute differences or relative errors; for categorical data, state differences can be 0 (consistent) or 1 (inconsistent). The system pairs all nodes and calculates the state differences at the same time, forming a state difference matrix. This matrix is ​​the direct basis for conflict detection.

[0098] Step S43: If the state difference exceeds the preset threshold, a conflict event is triggered, and the source node of the conflict is inferred using the conditional probability table.

[0099] When the state difference between a pair of nodes exceeds a preset threshold (e.g., a temperature difference exceeding 5°C, or a relative error exceeding 10%), the system determines that a conflict event has occurred. At this point, the system uses the conditional probability table of a Bayesian network to perform reverse reasoning, calculating the posterior probability that each node is a conflict source. Specifically, using the conflict event as observational evidence, the probability distribution of each node is updated using the Bayesian formula, and the node with the highest posterior probability is inferred to be the most likely conflict source.

[0100] To further improve the accuracy of conflict source identification, this application also introduces a secondary verification mechanism based on information entropy. That is, the refined scheme of step S43 above can be implemented through steps S431 to S433, as detailed below:

[0101] Step S431: Perform time series alignment on the local fusion results.

[0102] Because different computing nodes may have varying processing speeds and network latency, the local fusion results output by each node may not be perfectly aligned on the time axis. Therefore, before calculating entropy, it is necessary to align the time series of each node. Alignment methods can include linear interpolation, nearest neighbor matching, or dynamic time warping algorithms to map the time series of all nodes onto a unified common time axis.

[0103] Step S432: Calculate the entropy value of the aligned data.

[0104] Information entropy is a metric that measures the degree of uncertainty and disorder in data. For an aligned time series, the system calculates the data entropy value for each node within a time window. A higher entropy value indicates that the data at that node is more disordered and unpredictable, and more likely to be the source of conflict. The formula for calculating entropy is: ,in, For data values The probability of it appearing within the window.

[0105] Step S433: If the data entropy value is greater than the preset entropy threshold, the node data is determined to be in a chaotic state and marked as an untrustworthy source.

[0106] The preset entropy threshold can be determined based on the statistical characteristics of historical data, such as taking the 95th percentile of the historical entropy values ​​of all nodes. When the data entropy value of a node exceeds this threshold, it means that its data exhibits high disorder or random fluctuations. The system judges it to be in a chaotic state and marks it as an untrustworthy source. This label will be used for subsequent weight reduction processing or data source switching decisions.

[0107] The technical solutions described in steps S431 to S433 above can further identify low-quality sources of data chaos from the conflict sources inferred by the Bayesian network, avoiding misjudging random noise as valid information. The entropy value determination and Bayesian inference form a dual verification mechanism, significantly improving the accuracy of conflict source identification.

[0108] Step S44: Reduce the weight of the local fusion results output by the conflict source node or request retransmission.

[0109] Based on the inference result of step S43, the system takes corresponding measures for conflict source nodes. If the posterior probability of a conflict source is high or it has been marked as an untrusted source, its output local fusion result is downweighted (i.e., assigned a lower weight in subsequent global fusion), or a retransmission request is directly sent to the node, requiring it to re-report data in the next time window. In this way, the system can effectively suppress the impact of poor-quality data on the global fusion result, ensuring that the final generated global fusion data is logically consistent.

[0110] Step S5: Input the global fusion data into the preset decision analysis model, output an analysis report to guide business decisions, and adjust the quality assessment model and the update strategy of adaptive weighting coefficients in reverse according to the feedback results of the analysis report.

[0111] After processing steps S1 to S4, the system has generated high-quality, globally consistent fused data. This data will be used to support upper-level business decisions, such as equipment failure early warning, supply chain optimization, and market trend forecasting. However, the real environment is dynamic: the quality of data sources may drift over time (e.g., sensor aging, increased network congestion), and business needs may also change. If the fusion system is an open-loop structure—that is, data inflow → processing → result output followed by an end—then the system's fusion performance will gradually degrade as the environment changes, and the credibility of the analysis reports will decrease accordingly.

[0112] Traditional methods typically address performance degradation by periodically retraining the model manually or setting fixed threshold alarms. However, these methods are slow to respond, costly, and cannot achieve automated, continuous optimization. Therefore, there is an urgent need for a closed-loop feedback mechanism that can automatically optimize the fusion process based on decision results. To solve the above problems, this application adopts the following solution: global fusion data is input into a pre-set decision analysis model, which outputs an analysis report to guide business decisions. Simultaneously, the quality assessment model and the update strategy of the adaptive weighting coefficients are adjusted in reverse based on the feedback results of the analysis report.

[0113] Specifically, the decision analysis model is a prediction or classification model tailored to a specific business scenario, such as a random forest-based equipment failure prediction model or an LSTM-based sales prediction model. This model receives globally fused data as input and outputs an analysis report containing prediction results, confidence scores, and key influencing factors. Simultaneously, the system continuously monitors the accuracy metrics of the analysis report (such as the deviation between predicted and actual values, classification accuracy, etc.) and uses these metrics as feedback signals to influence the front-end quality assessment model and the update strategy of the adaptive weighting coefficients. Through this closed-loop feedback, the system can automatically identify performance degradation and trigger optimization actions, thereby achieving continuous self-improvement in the fusion process.

[0114] Specifically, the strategy of adjusting the quality assessment model and the adaptive weighting coefficients in reverse based on the feedback results of the analysis report can be implemented through steps S51 to S54, as detailed below:

[0115] Step S51: Monitor the accuracy of the analysis report output by the decision analysis model.

[0116] Accuracy metrics are key measures of the quality of a decision analysis model's output. For classification tasks, accuracy metrics can be precision, recall, or F1 score; for regression tasks, they can be mean squared error (MSE) or mean absolute percentage error (MAPE). The system calculates these metrics in real time or periodically (e.g., hourly, daily) and compares them to preset target values. When an accuracy metric falls below the target value, it indicates that the fusion system may be experiencing performance degradation, requiring subsequent tuning processes.

[0117] Step S52: If the accuracy index is lower than the preset target value, a model drift alarm is generated.

[0118] The preset target value can be set according to business needs, such as a classification accuracy of no less than 90% or a prediction error of no more than 5%. Once the accuracy metric is detected to fall below the target value, the system immediately generates a model drift alarm, indicating that the current quality assessment model and weighting coefficients may no longer be suitable for the latest data distribution. This alarm will serve as a signal to trigger subsequent backtracking and adjustment operations.

[0119] Step S53: In response to the model drift alarm, backtrack the historical change curves of the adaptive weighting coefficients.

[0120] Upon receiving a model drift alert, the system retrieves historical data on the adaptive weighting coefficients of each data source over a past period (e.g., the past 24 hours or 7 days) and plots their change curves. By analyzing the trend of the curves, it can determine whether the weighting coefficients have experienced abnormal fluctuations (e.g., sharp oscillations, monotonically increasing or decreasing), and which data source weight changes are temporally correlated with the decrease in accuracy. This backtracking analysis provides a basis for subsequent targeted adjustments.

[0121] To further enhance the intelligence level of backtracking adjustments, this application also introduces an adversarial incremental training method based on key feature dimensions. That is, in response to model drift alarms, the historical change curve of the adaptive weighting coefficients can be backtracked through the following steps S531 to S533:

[0122] Step S531: Extract the key feature dimensions that cause the accuracy metric to decline.

[0123] When accuracy declines, the system identifies the feature dimensions that have the greatest impact on the prediction results through feature importance analysis or gradient analysis. For example, in a device failure prediction scenario, it may be found that the contribution of the features "vibration frequency" and "temperature" has decreased significantly, indicating that the data quality of these features may have deteriorated.

[0124] Step S532: Construct adversarial samples based on key feature dimensions.

[0125] Adversarial samples are generated by applying small perturbations to the original data, aiming to induce the model to make incorrect predictions. In this application, the system applies targeted perturbations (e.g., adding ±5% noise to the "vibration frequency" feature) to the key feature dimensions extracted in step S531, generating a batch of adversarial samples. These samples simulate extreme cases of data source quality degradation, effectively exposing the vulnerabilities of the quality assessment model.

[0126] Step S533: Incrementally train the quality assessment model using adversarial samples.

[0127] The generated adversarial samples and their correct quality labels (e.g., "low confidence") are used as training data to incrementally train the quality assessment model (i.e., updating the model parameters with new samples without forgetting existing knowledge). In this way, the quality assessment model can learn to identify new degradation patterns, thereby more accurately assessing the quality of data sources in future fusion processes. Incremental training can employ methods such as Elastic Weight Consolidation (EWC) or progressive neural networks to prevent catastrophic forgetting.

[0128] Step S54: Adjust the retraining frequency and learning rate parameters of the quality assessment model.

[0129] In addition to incremental training using adversarial examples, the system also needs to adjust the model maintenance strategy at a macro level. Specifically, the system dynamically adjusts the retraining frequency (e.g., from once a week to once a day) and learning rate parameter (e.g., from 0.01 to 0.001) of the quality assessment model based on the magnitude and duration of the accuracy decline. By increasing the retraining frequency and reducing the learning rate, the model can adapt to new data distributions more finely while avoiding excessive oscillations.

[0130] During the dynamic update of the adaptive weighting coefficients, excessive fluctuations (oscillations) in the coefficients can lead to instability in the local fusion results, resulting in jitter in the global fusion data and affecting the continuity of decision analysis. Therefore, to further ensure the stability of the adaptive weighting coefficients, this application also introduces a smoothing mechanism based on the momentum term. That is, the adjustment of the retraining frequency and learning rate parameters of the quality assessment model in the above embodiment can be achieved through steps S541 to S543, as detailed below:

[0131] Step S541: Calculate the variance of the adaptive weighting coefficients.

[0132] Variance is a statistic that measures the degree of data fluctuation. The system calculates the variance of the adaptive weighting coefficients for each data source within a recent time window to quantify their stability. The larger the variance, the more drastic the fluctuation of the weighting coefficients, which may lead to instability in the local fusion results.

[0133] Step S542: If the variance is greater than the preset stability threshold, the weighted coefficient is determined to be oscillating.

[0134] The preset stability threshold can be set based on the normal fluctuation range of historical data, for example, twice the median of the historical variance of all data sources. When the variance of the weighting coefficient of a data source exceeds this threshold, the system determines that the weighting coefficient of that data source is in an oscillating state and needs to be smoothed.

[0135] Step S543: Introduce a momentum term to smooth the update formula for the adaptive weighting coefficients.

[0136] The momentum term is a commonly used technique in deep learning optimization. By incorporating the direction information from the previous update into the update formula, it can effectively suppress oscillations and accelerate convergence. Specifically, the update formula for the weighted coefficients is modified as follows:

[0137]

[0138]

[0139] in, For momentum, The momentum coefficient (usually, (0.9 can be taken) For learning rate, This represents the gradient of the loss function with respect to the weighted coefficients. By introducing a momentum term, the update direction of the weighted coefficients becomes smoother, avoiding drastic changes caused by short-term random fluctuations, thereby improving the stability of the local fusion results.

[0140] Considering that a data source frequently being marked as untrusted indicates a potential persistent failure or permanent degradation, simply downgrading its ranking without addressing the root cause will leave the system carrying inefficient or even harmful data sources for an extended period, hindering overall fusion performance. Therefore, Figure 1 The technical solution in the example may also include the technical solution in steps S61 and S62 as shown below.

[0141] Step S61: Count the number of times each node is marked as an untrustworthy source within the preset time window.

[0142] In step S433, the system marks nodes whose data entropy values ​​exceed a preset entropy threshold as untrustworthy sources. The system continuously records the number of times each node is marked and performs cumulative statistics within a preset time window (e.g., the past 1 hour or 24 hours). This statistical data reflects the long-term reliability performance of each data source.

[0143] Step S62: If the number of times a node is marked as an untrusted source exceeds the upper limit, the data source switching process is triggered to switch the data source corresponding to the node to a backup data source.

[0144] When a node is marked as an untrusted source more than a preset limit (e.g., 10 times / hour) within a time window, it indicates that the data source corresponding to that node may have experienced a persistent failure or permanent degradation, and simply downgrading it is no longer sufficient to resolve the issue. In this case, the system automatically triggers a data source switching process, switching the data source corresponding to that node from the primary data source to a backup data source (e.g., from the primary sensor to a backup sensor, or from the primary API interface to a mirror interface). After the switch is complete, the system records the switch event and continues to monitor the performance of the new data source. If the backup data source also performs poorly, the system can continue to try other backup sources or issue a manual intervention alert.

[0145] From the above appendix Figure 1 The example of a cloud-based multi-source data fusion processing method demonstrates that, on the one hand, by constructing a distributed data fusion engine, dynamically allocating computing nodes using a virtualized resource pool, and combining a task scheduling strategy based on a genetic algorithm, parallel processing of standardized data is achieved. This scheme can adaptively divide sub-tasks according to the real-time load and data volume of each data source, effectively avoiding performance bottlenecks in centralized processing and significantly improving the processing efficiency and real-time response capability of large-scale multi-source data fusion. On the other hand, a pre-trained quality assessment model is introduced on each computing node to evaluate the data credibility and timeliness of the corresponding data source in real time, and dynamically generate adaptive weighting coefficients based on the evaluation results. On this basis, a fusion algorithm combining weighted averaging and Kalman filtering is used for local fusion. Since the weighting coefficients can be dynamically adjusted according to changes in data source quality, this scheme overcomes the shortcomings of traditional fixed-weight fusion methods that cannot adapt to fluctuations in data source quality, improving the accuracy and robustness of local fusion results. Thirdly, all local fusion results are collected through a cloud master node, and a global consistency verification model based on Bayesian networks is used for conflict detection and correction. This model effectively identifies and corrects contradictory information between different computing nodes caused by differences in data sources or noise interference, thereby generating high-quality global fusion data and providing a reliable data foundation for subsequent decision analysis. Fourthly, the generated global fusion data is input into a pre-set decision analysis model, which outputs an analysis report. Based on the feedback from the analysis report, the quality assessment model and the update strategy of the adaptive weighting coefficients are adjusted in reverse. This closed-loop feedback mechanism enables the fusion system to continuously self-optimize, maintaining stable fusion performance and decision accuracy even in the face of dynamic changes in data source quality or adjustments in business needs, achieving long-term adaptive capability of the fusion method. In summary, the technical solution of this application improves the efficiency and accuracy of multi-source data fusion through dynamic task scheduling and closed-loop feedback optimization.

[0146] Please see the appendix Figure 2This application provides a cloud computing-based multi-source data fusion processing device, which includes a preprocessing module 201, a parallel processing module 202, a fusion module 203, a generation module 204, and an adjustment module 205, as detailed below:

[0147] The preprocessing module 201 is used to receive raw data from multiple heterogeneous data sources and perform standardized preprocessing on the raw data from multiple heterogeneous data sources to obtain standardized data with a unified format.

[0148] Parallel processing module 202 is used to build a distributed data fusion engine in a cloud computing environment. The distributed data fusion engine dynamically allocates computing nodes through a virtualized resource pool, and according to the real-time load and data volume of each data source, it adopts a task scheduling strategy based on genetic algorithm to divide standardized data with a unified format into several sub-tasks, and distributes these sub-tasks to the corresponding computing nodes for parallel processing.

[0149] The fusion module 203 is used to evaluate the data credibility and timeliness of the corresponding data source in real time for each subtask on the computing node using a pre-trained quality assessment model, and dynamically generate adaptive weighting coefficients based on the evaluation results. Then, a fusion algorithm combining weighted averaging and Kalman filtering is used to perform local fusion of the standardized data in the subtask to obtain the local fusion result.

[0150] The generation module 204 is used to aggregate the local fusion results output by all computing nodes to the cloud master node, and to perform conflict detection and correction on the local fusion results through a global consistency verification model based on Bayesian network, thereby generating global fusion data.

[0151] The adjustment module 205 is used to input global fusion data into a preset decision analysis model, output an analysis report to guide business decisions, and adjust the quality assessment model and the update strategy of adaptive weighting coefficients in reverse according to the feedback results of the analysis report.

[0152] From the above appendix Figure 2As illustrated by the example of a cloud-based multi-source data fusion processing device, on the one hand, by constructing a distributed data fusion engine, dynamically allocating computing nodes using a virtualized resource pool, and combining a task scheduling strategy based on a genetic algorithm, parallel processing of standardized data is achieved. This scheme can adaptively divide sub-tasks according to the real-time load and data volume of each data source, effectively avoiding performance bottlenecks in centralized processing and significantly improving the processing efficiency and real-time response capability of large-scale multi-source data fusion. On the other hand, a pre-trained quality assessment model is introduced on each computing node to evaluate the data credibility and timeliness of the corresponding data source in real time, and dynamically generate adaptive weighting coefficients based on the evaluation results. On this basis, a fusion algorithm combining weighted averaging and Kalman filtering is used for local fusion. Since the weighting coefficients can be dynamically adjusted according to changes in data source quality, this scheme overcomes the shortcomings of traditional fixed-weight fusion methods that cannot adapt to fluctuations in data source quality, and improves the accuracy and robustness of local fusion results. Thirdly, all local fusion results are collected through a cloud master node, and a global consistency verification model based on Bayesian networks is used for conflict detection and correction. This model effectively identifies and corrects contradictory information between different computing nodes caused by differences in data sources or noise interference, thereby generating high-quality global fusion data and providing a reliable data foundation for subsequent decision analysis. Fourthly, the generated global fusion data is input into a pre-set decision analysis model, which outputs an analysis report. Based on the feedback from the analysis report, the quality assessment model and the update strategy of the adaptive weighting coefficients are adjusted in reverse. This closed-loop feedback mechanism enables the fusion system to continuously self-optimize, maintaining stable fusion performance and decision accuracy even in the face of dynamic changes in data source quality or adjustments in business needs, achieving long-term adaptive capability of the fusion method. In summary, the technical solution of this application improves the efficiency and accuracy of multi-source data fusion through dynamic task scheduling and closed-loop feedback optimization.

[0153] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 3 in this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for a cloud-based multi-source data fusion processing method. When the processor 30 executes the computer program 32, it implements the steps described in the above-described embodiment of the cloud-based multi-source data fusion processing method, for example... Figure 1 Steps S1 to S5 are shown. Alternatively, when processor 30 executes computer program 32, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the preprocessing module 201, parallel processing module 202, fusion module 203, generation module 204, and adjustment module 205 are shown.

[0154] For example, the computer program 32 of the cloud computing-based multi-source data fusion processing method mainly includes: receiving raw data from multiple heterogeneous data sources, and performing standardized preprocessing on the raw data from multiple heterogeneous data sources to obtain standardized data with a unified format; constructing a distributed data fusion engine in the cloud computing environment, the distributed data fusion engine dynamically allocates computing nodes through a virtualized resource pool, and according to the real-time load and data volume of each data source, adopts a task scheduling strategy based on a genetic algorithm to divide the standardized data with a unified format into several sub-tasks, and distributes these sub-tasks to the corresponding computing nodes for parallel processing; for each sub-task on the computing node, using pre-trained... The quality assessment model evaluates the reliability and timeliness of the corresponding data source in real time, and dynamically generates adaptive weighting coefficients based on the assessment results. Then, a fusion algorithm combining weighted averaging and Kalman filtering is used to locally fuse the standardized data in the subtasks, obtaining local fusion results. The local fusion results output from all computing nodes are aggregated to the cloud master node, and a Bayesian network-based global consistency verification model is used to detect and correct conflicts in the local fusion results, generating globally fused data. The globally fused data is input into a preset decision analysis model, outputting an analysis report to guide business decisions. Simultaneously, the update strategy of the quality assessment model and adaptive weighting coefficients is adjusted in reverse based on the feedback from the analysis report. The computer program 32 can be divided into one or more modules / units, which are stored in memory 31 and executed by processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 32 in the electronic device 3.For example, computer program 32 can be divided into the functions of a preprocessing module 201, a parallel processing module 202, a fusion module 203, a generation module 204, and an adjustment module 205 (a module in the virtual device). The specific functions of each module are as follows: the preprocessing module 201 is used to receive raw data from multiple heterogeneous data sources and perform standardized preprocessing on the raw data from multiple heterogeneous data sources to obtain standardized data with a unified format; the parallel processing module 202 is used to build a distributed data fusion engine in a cloud computing environment. The distributed data fusion engine dynamically allocates computing nodes through a virtualized resource pool, and according to the real-time load and data volume of each data source, it uses a task scheduling strategy based on a genetic algorithm to divide the standardized data with a unified format into several sub-tasks, and distributes these sub-tasks to the corresponding computing nodes for parallel processing; fusion... Module 203 is used to evaluate the data credibility and timeliness of the corresponding data source in real time for each subtask on the computing node using a pre-trained quality assessment model, and dynamically generate adaptive weighting coefficients based on the evaluation results. Then, a fusion algorithm combining weighted averaging and Kalman filtering is used to perform local fusion on the standardized data in the subtask to obtain the local fusion result. Module 204 is used to aggregate the local fusion results output by all computing nodes to the cloud master node, and perform conflict detection and correction on the local fusion results through a global consistency verification model based on Bayesian networks to generate global fusion data. Module 205 is used to input the global fusion data into a preset decision analysis model, output an analysis report to guide business decisions, and adjust the update strategy of the quality assessment model and the adaptive weighting coefficients in reverse according to the feedback results of the analysis report.

[0155] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0156] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0157] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or RAM. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0158] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed. That is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0159] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0160] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0161] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0164] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, all or part of the processes in the above-described embodiments can also be implemented by a computer program instructing related hardware. The computer program for the cloud computing-based multi-source data fusion processing method can be stored in a storage medium. When executed by a processor, this computer program can implement the steps of the various method embodiments described above, namely, receiving raw data from multiple heterogeneous data sources, performing standardized preprocessing on the raw data from multiple heterogeneous data sources to obtain standardized data with a unified format; constructing a distributed data fusion engine in a cloud computing environment; the distributed data fusion engine dynamically allocates computing nodes through a virtualized resource pool, and according to the real-time load and data volume of each data source, uses a task scheduling strategy based on a genetic algorithm to divide the standardized data with a unified format into several sub-tasks, and then... Subtasks are distributed to corresponding computing nodes for parallel processing. For each subtask on a computing node, a pre-trained quality assessment model is used to evaluate the data reliability and timeliness of the corresponding data source in real time. Based on the evaluation results, adaptive weighting coefficients are dynamically generated. Then, a fusion algorithm combining weighted averaging and Kalman filtering is used to locally fuse the standardized data in the subtasks to obtain local fusion results. The local fusion results output from all computing nodes are aggregated to the cloud master node, and a global consistency verification model based on Bayesian networks is used to detect and correct conflicts in the local fusion results, generating globally fused data. The globally fused data is input into a pre-set decision analysis model, which outputs an analysis report to guide business decisions. Simultaneously, the update strategy of the quality assessment model and adaptive weighting coefficients is adjusted in reverse based on the feedback from the analysis report. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. Storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of storage media can be appropriately added to or removed according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, storage media may not include electrical carrier signals and telecommunication signals.

[0165] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application. The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the protection scope of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multi-source data fusion processing method based on cloud computing, characterized in that, The method includes: Step S1: Receive raw data from multiple heterogeneous data sources and perform standardized preprocessing on the raw data to obtain standardized data with a unified format; Step S2: Build a distributed data fusion engine in the cloud computing environment. The distributed data fusion engine dynamically allocates computing nodes through a virtualized resource pool, and divides the standardized data into several sub-tasks according to the real-time load and data volume of each data source, and distributes the several sub-tasks to the corresponding computing nodes for parallel processing. Step S3: For each subtask on the computing node, the data credibility and timeliness of the corresponding data source are evaluated in real time using a pre-trained quality assessment model. Based on the evaluation results, adaptive weighting coefficients are dynamically generated. Then, a fusion algorithm combining weighted averaging and Kalman filtering is used to locally fuse the standardized data in the subtask to obtain the local fusion result. Step S4: Aggregate the local fusion results output by all computing nodes to the cloud master node, and perform conflict detection and correction on the local fusion results through a global consistency verification model based on Bayesian networks to generate global fusion data; Step S5: Input the global fusion data into the preset decision analysis model, output an analysis report to guide business decisions, and adjust the update strategy of the quality assessment model and the adaptive weighting coefficients in reverse according to the feedback results of the analysis report.

2. The multi-source data fusion processing method based on cloud computing according to claim 1, characterized in that, The standardization preprocessing of the original data to obtain standardized data with a unified format includes: Step S11: Parse the metadata header of the original data and extract the data structure description information; Step S12: Based on the data structure description information, identify missing fields and outliers in the original data, and perform interpolation completion and smoothing filtering operations; Step S13: Map the completed and filtered data onto a predefined standardized data model to generate the standardized data.

3. The multi-source data fusion processing method based on cloud computing according to claim 2, characterized in that, Step S12 includes: Step S121: Construct a data association graph, which is generated based on the historical co-occurrence frequency of the multiple heterogeneous data sources; Step S122: Calculate the conditional probability distribution of the missing field based on the data association graph; Step S123: Sample from the conditional probability distribution to generate complete data, and write the complete data into the missing field.

4. The multi-source data fusion processing method based on cloud computing according to claim 1, characterized in that, The process of detecting and correcting conflicts in the local fusion results using a Bayesian network-based global consistency verification model includes: Step S41: Construct a conditional probability table to represent the causal relationships between the local fusion results; Step S42: Calculate the state difference between any two local fusion results at the same time. Step S43: If the state difference exceeds a preset threshold, a conflict event is triggered, and the conflict source node is inferred using the conditional probability table; Step S44: Reduce the weight of the local fusion results output by the conflict source node or request retransmission.

5. The multi-source data fusion processing method based on cloud computing according to claim 4, characterized in that, If the state difference exceeds a preset threshold, a conflict event is triggered, and the conflict source node is inferred using the conditional probability table, including: Step S431: Perform time series alignment on the local fusion results; Step S432: Calculate the entropy value of the aligned data; Step S433: If the data entropy value is greater than the preset entropy threshold, the node data is determined to be in a chaotic state and marked as an untrustworthy source.

6. The multi-source data fusion processing method based on cloud computing according to claim 1, characterized in that, The update strategy for adjusting the quality assessment model and the adaptive weighting coefficients in reverse based on the feedback results of the analysis report includes: Step S51: Monitor the accuracy index of the analysis report output by the decision analysis model; Step S52: If the accuracy index is lower than the preset target value, a model drift alarm is generated; Step S53: In response to the model drift alarm, backtrack the historical change curves of the adaptive weighting coefficients; Step S54: Adjust the retraining frequency and learning rate parameters of the quality assessment model.

7. The multi-source data fusion processing method based on cloud computing according to claim 6, characterized in that, In response to the model drift alarm, the historical change curves of the adaptive weighting coefficients are traced back, including: Step S531: Extract the key feature dimensions that cause the decrease in the accuracy metric; Step S532: Construct adversarial samples based on the aforementioned key feature dimensions; Step S533: Incrementally train the quality assessment model using the adversarial samples.

8. A cloud computing-based multi-source data fusion processing device, characterized in that, The device includes: The preprocessing module is used to receive raw data from multiple heterogeneous data sources and perform standardized preprocessing on the raw data to obtain standardized data with a unified format. The parallel processing module is used to build a distributed data fusion engine in a cloud computing environment. The distributed data fusion engine dynamically allocates computing nodes through a virtualized resource pool, and divides the standardized data into several sub-tasks according to the real-time load and data volume of each data source, and distributes the several sub-tasks to the corresponding computing nodes for parallel processing. The fusion module is used to evaluate the data credibility and timeliness of the corresponding data source in real time for each subtask on the computing node using a pre-trained quality assessment model, and dynamically generate adaptive weighting coefficients based on the evaluation results. Then, a fusion algorithm combining weighted averaging and Kalman filtering is used to perform local fusion on the standardized data in the subtask to obtain the local fusion result. The generation module is used to aggregate the local fusion results output by all computing nodes to the cloud master node, and to perform conflict detection and correction on the local fusion results through a global consistency verification model based on Bayesian network, thereby generating global fusion data. The adjustment module is used to input the global fusion data into a preset decision analysis model, output an analysis report to guide business decisions, and adjust the quality assessment model and the update strategy of the adaptive weighting coefficients in reverse according to the feedback results of the analysis report.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.