A multi-source heterogeneous data-oriented big data cleaning and fusion method

Abstract

The application relates to the technical field of data processing, and discloses a multi-source heterogeneous data-oriented big data cleaning and fusion method, which comprises the following steps: writing asynchronous observation values into a time attenuation buffer matrix, combining a dynamic attenuation rate constant to calculate state weights; calculating physical conduction delay, updating a field dependence matrix and updating the dynamic attenuation rate constant; calculating conditional information entropy based on the updated matrix and state weights, mapping the conditional information entropy into confidence weight to complete data cleaning; performing topological edge pruning on the updated matrix based on the average value of the confidence weight, calculating topological survival rate and a regularization coefficient; and performing weighted least square fusion on the observation values by using the confidence weight and the regularization coefficient, and outputting a state vector. The application retains time sequence characteristics to improve the accuracy of data cleaning, avoids divergence by regularization punishment, and realizes stable big data cleaning and fusion for multi-source heterogeneous data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method for cleaning and fusing big data from multiple sources and heterogeneous data. Background Technology

[0002] Data generated by various sensors deployed in industrial sites differ significantly in dimensions and sampling periods, constituting multi-source heterogeneous data. Big data cleaning and fusion for this type of data refers to identifying and processing invalid values ​​from continuous observation sequences with asynchronous time characteristics and noise interference, while simultaneously integrating multi-source collaborative information for calculation, ultimately outputting fused state parameters that objectively reflect the current real operating conditions.

[0003] Existing conventional methods for processing multi-source heterogeneous data typically employ data interpolation algorithms to force sequence alignment along the time axis, followed by hard thresholding based on pre-defined upper and lower limits to complete data cleaning. After preprocessing, conventional methods extract the remaining valid observation values ​​to construct an observation matrix, and then use the traditional weighted least squares method to solve the objective functional to obtain the final data fusion result.

[0004] The above processing methods have several limitations in practical applications. Interpolation-forced alignment ignores the objective delay time of spatial medium transmission, easily causing distortion of the underlying features of the original data. Simultaneously, hard threshold cleaning lacks cross-correlation verification between multiple data sources; relying solely on isolated numerical judgments can easily misjudge effective spike data reflecting sudden changes in equipment operating conditions as anomalies and directly discard them. Furthermore, the large-scale discarding of local numerical values ​​caused by hard threshold cleaning, coupled with data disconnections due to local equipment failures, results in a large number of zero elements in the weight diagonal matrix entering the fusion stage. This rank deficiency phenomenon caused by incomplete observation data leads to matrix multiplication approaching a singular state and causing divergence in the inversion operation, thus failing to obtain and output stable data fusion results.

[0005] Therefore, this invention proposes a big data cleaning and fusion method for multi-source heterogeneous data to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a big data cleaning and fusion method for multi-source heterogeneous data. It solves the problems of data low-level feature distortion, accidental deletion of effective peak states, and large amounts of incomplete data caused by forced time interpolation and single-point hard threshold removal when processing multi-source heterogeneous data in existing solutions. This leads to rank deficiency of the fusion matrix and divergence in computational solution.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for cleaning and fusing big data from multiple heterogeneous sources, comprising the following steps: Asynchronous observations from multi-source heterogeneous sensors are written into a time decay buffer matrix, and the state weights of the asynchronous observations in the time decay buffer matrix are calculated in combination with the dynamic decay rate constant. Calculate the physical conduction delay between the multi-source heterogeneous sensors, update the neighborhood dependency matrix using the physical conduction delay, and calculate the dynamic attenuation rate constant using the physical conduction delay; Based on the updated domain dependency matrix and the state weights, the conditional information entropy of the target asynchronous observation is calculated, and the conditional information entropy is mapped to confidence weights to complete the data cleaning for the asynchronous observation. Calculate the statistical window average of the confidence weights, perform topological edge pruning on the updated domain dependency matrix based on the statistical window average, and calculate the topological survival rate of the updated domain dependency matrix after performing topological edge pruning. The regularization coefficient is calculated based on the topological survival rate. The confidence weight and the regularization coefficient are used to perform weighted least squares data fusion on the asynchronous observations, and the fused state vector is output.

[0008] Preferably, the step of calculating the state weight of the asynchronous observation in the time decay buffer matrix in conjunction with the dynamic decay rate constant includes: Extract the original timestamps of the asynchronous observations; Calculate the time difference between the current system time and the original timestamp; The exponential decay function value is calculated with the natural constant as the base and the negative of the product of the dynamic decay rate constant and the time difference as the exponent. The value of the exponential decay function is used as the state weight and assigned to the asynchronous observation value written into the time decay buffer matrix.

[0009] Preferably, the step of calculating the physical conduction delay between the multi-source heterogeneous sensors includes: Extract the time series of upstream and downstream sensors within the sliding window of the stream processing; Calculate the discrete cross-correlation function between the upstream sensor time series and the downstream sensor time series; Find the time offset that makes the discrete cross-correlation function reach its global maximum and determine the physical conduction delay between the multi-source heterogeneous sensors.

[0010] Preferably, the step of calculating the dynamic attenuation rate constant using the physical conduction delay includes: Divide the preset basic attenuation scalar by the maximum value of the physical conduction delay and the first preset minimum positive real number compensation term, and use the quotient as the dynamic attenuation rate constant. The calculated dynamic decay rate constant is fed back.

[0011] Preferably, the step of calculating the conditional information entropy of the target asynchronous observation based on the updated domain dependency matrix and the state weights includes: Determine the target asynchronous observation and address the corresponding associated sensor subset; Calculate the weighted probability distribution based on the state weights; The conditional probability is obtained by dividing the joint probability density in the weighted probability distribution by the marginal probability density, and then the conditional information entropy of the asynchronous observation of the target is calculated by summing the probabilities.

[0012] Preferably, the step of mapping the conditional information entropy to confidence weights to complete the data cleaning for the asynchronous observations includes: Calculate the difference between the conditional information entropy and the preset entropy tolerance threshold, and multiply the preset sensitivity adjustment coefficient by the difference to obtain the first product; The first exponent value is calculated with the natural constant as the base and the first product as the exponent; The confidence weight is calculated by dividing constant 1 by the sum of constant 1 and the first exponent value. The confidence weights are bound to the target asynchronous observations and the data is written to disk to complete the data cleaning for the asynchronous observations.

[0013] Preferably, the step of performing topological edge pruning on the updated domain dependency matrix based on the statistical window average includes: The average value of the statistical window is compared with the preset pruning tolerance threshold, and the total number of confidence weights is simultaneously verified to ensure that the preset minimum sample size requirement is met. If the average value of the statistical window is lower than the pruning tolerance threshold and the total number of confidence weights is greater than or equal to the minimum sample size requirement, obtain the row and column index coordinates of the failed upstream and downstream sensor nodes in the updated domain dependency matrix. Locate the memory region corresponding to the row index and column index coordinates, force the original element values ​​to be overwritten to zero, and perform the topology edge trimming.

[0014] Preferably, the step of calculating the topological survival rate of the updated neighborhood dependency matrix after performing topological edge pruning includes: Count the total number of non-zero elements in the updated domain dependency matrix after performing the topological edge pruning. The topology survival rate is obtained by dividing the total number of non-zero elements after the topology edge pruning by the sum of the initial total number of non-zero elements before pruning and the second preset minimal positive real number compensation term.

[0015] Preferably, the step of calculating the regularization coefficient based on the topology survival rate includes: Multiply the preset sensitivity adjustment scalar by the topology survival rate and take the negative number as the second product; The second exponent value is calculated with the natural constant as the base and the second product as the exponent; The third product is obtained by multiplying the preset penalty scaling factor by the second exponent value; The regularization coefficient is calculated by adding the preset baseline regularization constant to the third product.

[0016] Preferably, the step of performing weighted least squares data fusion on the asynchronous observations using the confidence weights and the regularization coefficients, and outputting the fused state vector, includes: The asynchronous observations within the current fusion window are aligned to a unified timestamp using a time alignment algorithm. Extract the aligned asynchronous observations, and combine the extracted observation values ​​column by column to construct an observation column vector; The confidence weights are arranged on the main diagonal in the order of the corresponding observation data to construct a weight diagonal matrix; A weighted least squares objective functional containing a regularization penalty term is established based on the state observation matrix, the observation column vector, the weight diagonal matrix, the preset regularization constraint matrix, and the regularization coefficients. The fused state vector is obtained by solving the extreme value of the weighted least squares objective functional containing the regularization penalty term, and the fused state vector is written into the specified cache as output.

[0017] This invention provides a method for cleaning and fusing big data from multiple sources and heterogeneous data. It has the following beneficial effects: 1. This invention writes asynchronous observations from multi-source heterogeneous sensors into a time decay buffer matrix and calculates state weights, while continuously updating the neighborhood dependency matrix and dynamic decay rate constant based on physical propagation delay. This method achieves dynamic data acquisition while preserving the original physical time sequence, overcomes the feature distortion problem caused by conventional interpolation alignment, corrects time delay misalignment caused by spatial distance, and lays an objective data foundation for big data cleaning and fusion of multi-source heterogeneous data.

[0018] 2. This invention calculates conditional information entropy based on an updated domain dependency matrix and state weights, and maps it to continuous confidence weights. This process utilizes the spatial collaborative correlation between multi-source heterogeneous data for cross-validation, overcoming the shortcomings of traditional single hard thresholds that are prone to misjudgment. The system thereby assigns extremely low weights to anomalous mutations, filtering out isolated noise while retaining effective spikes from real-world conditions, thus improving the accuracy of big data cleaning for multi-source heterogeneous data.

[0019] 3. This invention performs topology edge pruning and calculates the topology survival rate by averaging the confidence weights, then adaptively adjusts the regularization coefficient, and finally uses it and the confidence weights to perform weighted least squares fusion. The above design dynamically introduces the sensor network connectivity state as a penalty term into the calculation, compensating for the main diagonal elements of the matrix at the algebraic level. This avoids matrix singularity and divergence risks caused by local device failures, achieving stable solutions for multi-source heterogeneous data fusion equations and resulting in stable data fusion outputs. Attached Figure Description

[0020] Figure 1 This is a system architecture diagram of a big data cleaning and fusion system for multi-source heterogeneous data according to the present invention; Figure 2 This is a flowchart of a big data cleaning and fusion method for multi-source heterogeneous data according to the present invention; Figure 3 This is a schematic diagram of the big data cleaning and fusion processing effect of the present invention for multi-source heterogeneous data; wherein, sub-figure (a) is a comparison diagram of pressure observation values ​​and fused values; sub-figure (b) is a trend diagram of confidence weight changes.

[0021] Among them, 10 is the data access module; 20 is the latency tracking module; 30 is the data cleaning module; 40 is the topology management module; and 50 is the data fusion module. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] See attached document Figure 1 The present invention provides a big data cleaning and fusion system for multi-source heterogeneous data, including: a data access module 10; a latency tracking module 20; a data cleaning module 30; a topology management module 40; and a data fusion module 50.

[0024] The data access module 10 is used to receive asynchronous observations from multi-source heterogeneous sensors. The data access module 10 writes the acquired asynchronous observations into a time decay buffer matrix and calculates the state weight of the asynchronous observations in the time decay buffer matrix in conjunction with the dynamic decay rate constant.

[0025] The time delay tracking module 20 is used to calculate the physical conduction delay between multiple heterogeneous sensors. The time delay tracking module 20 updates the neighborhood dependency matrix using the physical conduction delay and calculates the dynamic attenuation rate constant using the physical conduction delay. The time delay tracking module 20 feeds back the calculated dynamic attenuation rate constant to the data access module 10.

[0026] The data cleaning module 30 reads the updated domain dependency matrix. Based on the updated domain dependency matrix and state weights, the data cleaning module 30 calculates the conditional information entropy of the target asynchronous observations. The data cleaning module 30 maps the conditional information entropy to confidence weights to complete the data cleaning for the asynchronous observations.

[0027] The topology management module 40 is used to calculate the statistical window average of the confidence weights. Based on this statistical window average, the topology management module 40 performs topology edge pruning on the domain dependency matrix. After performing topology edge pruning, the topology management module 40 calculates the topology survival rate of the domain dependency matrix.

[0028] The data fusion module 50 receives the topology survival rate. Based on the topology survival rate, the data fusion module 50 calculates the regularization coefficient. The data fusion module 50 then performs weighted least squares data fusion on the asynchronous observations using confidence weights and the regularization coefficient, outputting a fused state vector.

[0029] See attached document Figure 2 This invention provides a method for cleaning and fusing big data from multiple sources and heterogeneous data. This method relies on the aforementioned system for cleaning and fusing big data from multiple sources and heterogeneous data, and includes the following steps: S10: Write the asynchronous observations from the multi-source heterogeneous sensors into the time decay buffer matrix, and calculate the state weight of the asynchronous observations in the time decay buffer matrix in combination with the dynamic decay rate constant. S20: Calculate the physical conduction delay between multi-source heterogeneous sensors, update the neighborhood dependency matrix using the physical conduction delay, and calculate the dynamic attenuation rate constant using the physical conduction delay; S30: Calculate the conditional information entropy of the target asynchronous observation based on the updated domain dependency matrix and state weights, and map the conditional information entropy to confidence weights to complete the data cleaning for the asynchronous observations. S40, calculate the statistical window average of the confidence weights, perform topological edge pruning on the domain dependency matrix based on the statistical window average, and calculate the topological survival rate of the domain dependency matrix after performing topological edge pruning; S50 calculates the regularization coefficient based on topological survival rate, and performs weighted least squares data fusion on asynchronous observations using confidence weights and regularization coefficients, outputting a fused state vector.

[0030] To further clarify the implementation of each technical aspect of the present invention, the following will provide a detailed description of the implementation of each functional module involved above and its internal processing flow.

[0031] See attached document Figure 1 and Figure 2 In step S10, this embodiment performs the operation of writing asynchronous observations from multi-source heterogeneous sensors into a time decay buffer matrix, and calculating the state weights of asynchronous observations in the time decay buffer matrix in conjunction with the dynamic decay rate constant. In industrial settings, different types of sensors are limited by physical dimensions and hardware sampling periods, resulting in significantly discrete data streams on the time axis. In this embodiment, asynchronous observations refer to the real physical measurement data collected by each multi-source heterogeneous sensor under its own independent sampling clock, bearing the original time stamp and without any manual resampling or interpolation modification.

[0032] To achieve unified addressing and management of multi-source heterogeneous data at the underlying level, a contiguous two-dimensional storage space is pre-allocated in the memory pool to construct a time decay buffer matrix. The time decay buffer matrix is ​​a two-dimensional data structure used to temporarily store asynchronously arriving data and record its timeliness. Its row dimension maps to all multi-source heterogeneous sensor nodes in the network, and its column dimension maps to cache slots used to store historical data and their associated attribute labels. At the data access layer, the raw data packets uploaded by the multi-source heterogeneous sensor network are intercepted and subjected to low-level unpacking and protocol parsing operations. After extracting the asynchronous observation values ​​containing actual physical measurement information, these values ​​are directly written into the cache slots in the time decay buffer matrix that match the corresponding sensors.

[0033] After completing the above data writing operations, to avoid the distortion of underlying physical transmission characteristics caused by the forced time alignment of conventional polynomial interpolation, this embodiment employs a continuous time decay mechanism to maintain the state of asynchronously arriving data streams at the probabilistic level. The process of calculating the state weights of asynchronous observations in the time decay buffer matrix in conjunction with the dynamic decay rate constant is specifically executed through the following sub-steps: S101, Extract the raw timestamp of the asynchronous observation. While parsing and acquiring the asynchronous observation, simultaneously read the absolute timestamp generated by the underlying hardware when the value completes analog-to-digital conversion at the edge sensor, and define this absolute timestamp as the raw timestamp. For data packet parsing and field extraction, those skilled in the art can call the corresponding conversion driver according to the specific communication protocol specifications. Its underlying binary parsing logic is well-known in the field and will not be elaborated upon here.

[0034] S102, calculate the time difference between the current system time and the original timestamp. Within any running cycle that triggers the data processing logic, record the absolute time of the current high-precision global reference clock as the current system time. Based on the obtained current system time and original timestamp, subtract the original timestamp from the current system time to obtain a time interval variable greater than or equal to zero. This time interval variable is the time difference. This time difference objectively characterizes, in a physical sense, the aging degree of the corresponding asynchronous observation value since its generation in memory, and its value monotonically increases as the time reference advances.

[0035] S103, the exponential decay function value is calculated using the natural constant as the base and the negative of the product of the dynamic decay rate constant and the time difference as the exponent. The state of physical equipment in industrial settings dynamically evolves over time, causing the confidence level of historical measurement data in representing the current actual operating conditions to gradually decrease. Based on this general principle, a corresponding dynamic decay rate constant is pre-configured for each multi-source heterogeneous sensor node. This constant is a real number with a strictly greater than zero value, used to characterize the time span by which a specific physical quantity maintains its effectiveness in space. Due to the fundamental differences in the inertia of different physical signals, in one embodiment, a smaller dynamic decay rate constant is configured for relatively slow-changing data sources such as temperature sensors to slow down the decay rate of their state weights; while for high-frequency abrupt signals such as pressure or vibration, a larger dynamic decay rate constant is configured to accelerate their aging. The dynamic decay rate constant matching the target sensor that generates asynchronous observations is retrieved as the control parameter for the decay slope, thereby constructing a continuous exponential probabilistic evaluation model. The corresponding mathematical expression is as follows: ; In the formula, Indicates the number is The target sensor in the current system time The value of the exponential decay function is between 0 and 1. Represented by natural constant An exponential function with base 0; Indicates target sensor The initial value of the corresponding dynamic decay rate constant can be preset by engineering experience and dynamically calibrated and updated by subsequent time delay closed-loop tracking logic. This represents the calculated current system time. The time difference from the original timestamp. According to the boundary conditions of this expression, when the time difference is zero, the exponential decay function value takes the maximum limit of 1, which indicates that the observation data has absolute confidence when it is first generated. As the time difference increases monotonically, the exponential decay function value shows a smooth nonlinear decrease.

[0036] In step S104, the exponential decay function value is assigned as a state weight to the asynchronous observation written into the time decay buffer matrix. Based on the above calculation results, the calculated exponential decay function value is defined as the state weight of the data, and the state weight is bound to the memory address of the asynchronous observation already written to disk using a structure in the time decay buffer matrix. Functionally, the state weight replaces the conventional mechanism of directly discarding expired data in the traditional process. This allows for the quantitative assessment of the contribution of each data source to the current global state by extracting the remaining historical observation data and their corresponding state weights from the time decay buffer matrix, even in the case of a large amount of sensor data not arriving synchronously at any high-frequency observation moment. By implementing the above steps, the dynamic acquisition and state weighting of asynchronous observations are completed without disrupting the original physical time sequence.

[0037] See attached document Figure 1 and Figure 2 After completing the data acquisition and state weighting operations in step S10, this embodiment executes step S20 to calculate the physical conduction delay between the multi-source heterogeneous sensors, update the neighborhood dependency matrix using the physical conduction delay, and calculate the dynamic attenuation rate constant using the physical conduction delay.

[0038] In industrial settings, due to the inherent distances between equipment, it takes time for changes in the state of upstream equipment to be transmitted downstream. This results in a time lag when different sensors capture the same operational event. To correct this time misalignment caused by spatial intervals, this embodiment establishes an adaptive adjustment mechanism for time parameters, specifically including the following sub-steps: S201, extract the upstream and downstream sensor time series within the streaming processing sliding window. In this embodiment, the system allocates a data observation interval of a specified time length in memory to construct a streaming processing sliding window. The upper limit of the window's time span continuously slides as the current system clock advances. To avoid computational waste caused by blind matching, the system determines the upstream and downstream sensor nodes with sequential transmission relationships based on the fluid or signal transmission direction determined by the field device topology diagram. Based on the node identification information, the system extracts continuous historical measurement values ​​within the time covered by the streaming processing sliding window from the time decay buffer matrix. The extracted historical measurement values ​​are arranged in chronological order to construct the upstream and downstream sensor time series respectively. To eliminate interference from inconsistent sampling frequencies of multi-source heterogeneous sensors on time series matching, the system uses a discrete time grid to perform local interpolation resampling on the upstream and downstream sensor time series to unify them to a common time step.

[0039] S202, calculate the discrete cross-correlation function between the upstream and downstream sensor time series. To quantify the waveform similarity of the two data series with time lags under different misalignment states, this embodiment introduces the principle of cross-correlation analysis to construct an evaluation model. Let the upstream sensor time series with a unified time step be the baseline variable set, and the downstream sensor time series with a unified time step be the variable set to be matched. The corresponding mathematical expressions are as follows: ; In the formula, Indicates the time offset as The discrete cross-correlation function value at time; This indicates the total number of sampling points contained within the extracted streaming sliding window; Indicates the first in the upstream sensor time series Data from each sampling point; This indicates that the downstream sensor time series has been shifted. Data of corresponding sampling points after each discrete time step; The time offset variable is used for iterative calculations, and its value range is set between the lengths of the positive and negative flow processing sliding window. Using the above model, the correlation index between the two sets of data under different offset states can be obtained.

[0040] S203, find the time offset that makes the discrete cross-correlation function reach its global maximum and determine the physical conduction delay between the multi-source heterogeneous sensors. The function value obtained in the preceding steps objectively reflects the degree of overlap between the two sets of data under a specific misalignment state. The system performs a global comparison of all calculated discrete cross-correlation function values ​​to extract peak extrema. In industrial environments with strong noise interference, multiple local spurious extrema are prone to occur. To avoid judgment errors, after locking the value of the independent variable corresponding to the absolute maximum value, the system compares and verifies it with a preset historical reasonable delay range. In this embodiment, the preset historical reasonable delay range refers to the normal signal conduction time range determined based on engineering experience. For example, the theoretical time for fluid to flow through upstream and downstream sensors in a certain pipeline is 5 to 8 seconds, so the range can be set to [4, 10] seconds. If the value of the independent variable is within this range, it is confirmed as the time offset corresponding to the global maximum value. This time offset represents the actual time consumed by the change in operating conditions from upstream to downstream. The system defines this value as the physical conduction delay between the multi-source heterogeneous sensors. .

[0041] S204, updating the domain dependency matrix using physical propagation delay. As a preferred approach, a domain dependency matrix representing the connectivity of nodes is pre-constructed in memory. The domain dependency matrix is ​​a two-dimensional square matrix composed of directed edge weights. Its row and column indices map to different sensor nodes in the network, and the non-zero elements within the matrix record the propagation direction and specific time span between intersecting nodes. After confirming the relationship between upstream and downstream nodes, the system generates addressing coordinates based on the row number of the upstream sensor and the column number of the downstream sensor. Based on these coordinates, the system locates the corresponding memory block in the domain dependency matrix and replaces the original fixed preset value at that location with the calculated physical propagation delay between the multi-source heterogeneous sensors, thus completing the operation of updating the domain dependency matrix using physical propagation delay.

[0042] S205 utilizes physical conduction delay to calculate the dynamic attenuation rate constant and provides feedback. Changes in actual operating conditions are often accompanied by variations in medium flow rate or conduction efficiency. When the conduction time increases, the effective state of the data generated by the upstream sensor in the system should be correspondingly extended to ensure that the historical state of the upstream has not yet decayed to zero when downstream data is generated. To achieve adaptive tracking of the data attenuation rate to actual operating conditions, a preset basic attenuation scalar with a strictly positive value is introduced, and the following division calculation is performed: ; In the formula, This represents the calculated updated dynamic decay rate constant; This represents the preset basic attenuation scalar, the value of which is preset by engineers based on the maximum tolerable delay benchmark of the business scenario, and is usually between 0.1 and 1.0; This represents the physical conduction delay between the multi-source heterogeneous sensors calculated in the previous step; This is a function to find the maximum value. This is a preset, minimal positive real number compensation term, whose value ranges from 10. -6 Up to 10 -3 In one embodiment, to match the floating-point arithmetic precision limitations of the system and prevent division overflow caused by a denominator of zero, Set to 10 -5 Because the physical transmission delay between multiple heterogeneous sensors may approach zero under instantaneous communication or extremely short-range conditions, to prevent the denominator from becoming extremely small during division operations and causing program malfunctions, the system employs a maximum value function to safely clamp the lower limit of the denominator within a certain range. .

[0043] After deriving the quotient by dividing the preset basic attenuation scalar by the physical transmission delay between the multi-source heterogeneous sensors using the above logic, the system feeds this quotient back to step S10 as the updated dynamic attenuation rate constant. The system replaces the initially configured dynamic attenuation rate constant in step S10 with the updated dynamic attenuation rate constant. Step S10 then uses the updated dynamic attenuation rate constant to calculate the state weight of newly accessed asynchronous observations in the time attenuation buffer matrix. Through this feedback loop, the longer the transmission time, the smaller the corresponding dynamic attenuation rate constant, which slows down the attenuation of historical data in the time attenuation buffer matrix, thereby maintaining the causal validity of upstream and downstream data.

[0044] See attached document Figure 1 and Figure 2 After completing the closed-loop calibration and topology update of the time domain parameters in step S20, this embodiment executes step S30, which calculates the conditional information entropy of the target asynchronous observation based on the updated domain dependency matrix and state weights, and maps the conditional information entropy to confidence weights to complete the data cleaning for the asynchronous observation.

[0045] Traditional industrial data cleaning processes often employ hard threshold discrimination, where a single sensor's measurement value is directly discarded when it exceeds manually set upper or lower limits. This approach is highly susceptible to misclassifying extremely valid data reflecting sudden changes in conditions as abnormal noise, thus compromising the causal integrity of the data sequence. To overcome this deficiency, this embodiment abandons isolated, single-dimensional judgments and introduces the principle of cross-validation based on information theory. Cross-validation is performed based on the spatial correlation between multiple heterogeneous sensors to achieve smooth, fault-tolerant cleaning. This process specifically includes the following sub-steps: S301, determine the target asynchronous observation and address the corresponding subset of associated sensors. In this embodiment, the system extracts one original measurement data point requiring legality evaluation within the current operating cycle from the time decay buffer matrix, and explicitly defines the extracted original measurement data point as the target asynchronous observation. To obtain the objective reference context for evaluating the target asynchronous observation, the system reads the updated domain dependency matrix from step S20. By querying the row and column vectors of the node that generated the target asynchronous observation in the domain dependency matrix, the system accurately filters out other sensor nodes with non-zero propagation edge weights. The system combines the nodes that have deterministic causal connectivity in spatial distribution and propagation paths to construct the corresponding subset of associated sensors.

[0046] S302, calculate the weighted probability distribution by combining state weights. Simply obtaining spatially related nodes is insufficient to accurately characterize the dynamic evolution of the equipment system; the system also needs to introduce data validity parameters in the time dimension. The system extracts the historical observation data sequence of the related sensor subset within the current evaluation statistical window from the time decay buffer matrix. Since data closer to the current time more accurately represents the current actual operating conditions, the system retrieves the state weights assigned to each historical observation data point. It should be noted that the state weights used here are the weight values ​​initially calculated and bound and stored in step S10; if the system enters a subsequent cyclic evaluation period, the state weights are recalculated and overwritten using the dynamic decay rate constant updated in step S20. The system uses the extracted state weights as sample weight factors in the underlying statistical algorithm and employs a weighted kernel density estimation method to calculate the joint probability density of the target asynchronous observation value and the related sensor subset data sequence, as well as the marginal probability density of the related sensor subset data sequence. In this embodiment, as a preferred implementation, the weighted kernel density estimation method uses a Gaussian kernel function as the basic smoothing function to ensure the continuity and differentiability of the probability density output. For the specific underlying mathematical logic of calculating the probability distribution of weighted kernel density estimation, those skilled in the art can directly call conventional statistical analysis algorithm libraries to execute it. The calculation and fitting process is a well-known technology in this field and will not be elaborated here.

[0047] S303, calculate the conditional information entropy of the target asynchronous observation based on the updated domain dependency matrix and state weights. After obtaining the relevant weighted probability density distribution, the system quantifies the rationality of the target asynchronous observation according to information theory rules. In this embodiment, conditional information entropy refers to the uncertainty value of the target asynchronous observation given the known operating states of the associated sensor subset. The system uses the joint probability density divided by the marginal probability density to obtain the conditional probability, and then accumulates them to calculate the conditional information entropy. The mathematical expression corresponding to this calculation step is as follows: ; In the formula, This represents the conditional information entropy of the calculated asynchronous observations of the target; This represents the total number of discrete state spaces divided within the current evaluation statistics window; Indicates the target sensor at the first The values ​​that can be taken in each state interval; Indicates the associated sensor subset in the corresponding first... A vector of combined values ​​for each state interval; This represents the calculated joint probability density, for the th Discrete probability values ​​obtained by performing definite integral mapping on each state interval; This represents the calculated marginal probability density, for the th Discrete probability values ​​obtained by performing definite integral mapping on each state interval; Represents logarithmic operations to the base 2; These are preset, small positive real-valued compensation parameters. Considering that the marginal probability density may approach zero under certain extreme static conditions, to prevent division operations from causing the system's denominator to become zero and logarithmic divergence collapse, the system forcibly superimposes a minimum value into the denominator term. In one embodiment, algebraic smoothing is performed. The value is set to 10. -6 .

[0048] The general technical principle of the above calculation logic is as follows: if the asynchronous observation of the target conforms to the transmission law of the actual working condition, the co-occurrence probability of the asynchronous observation of the target in the current context is extremely high, and the corresponding conditional information entropy calculation result is close to the low extreme value; conversely, if the asynchronous observation of the target is an isolated glitch noise caused by the sudden hardware failure of the sensor itself or communication interference, the conditional information entropy will increase significantly due to the destruction of the original cooperative correlation.

[0049] S304 maps the conditional information entropy to confidence weights. The conditional information entropy obtained in the previous step fluctuates significantly in its numerical range and exhibits a negative correlation with the traditional concept of weights, making it unsuitable for direct use in subsequent data fusion systems. Therefore, as a preferred approach, the system introduces a nonlinear mapping activation function based on the logistic equation to convert the conditional information entropy into confidence weights within a normalized interval. Confidence weights are used to objectively characterize the reliability of asynchronous observations reflecting the actual working conditions. The corresponding mathematical expression is as follows: ; In the formula, This represents the confidence weight of the mapping output, and the value of the confidence weight is strictly constrained to the interval (0, 1). Represented by natural constant Exponential operations with base 0; This represents the preset sensitivity adjustment coefficient. The sensitivity adjustment coefficient is greater than zero and is used to control the steepness of the weight decay curve. Based on the strength requirements of noise suppression, the system usually sets the sensitivity adjustment coefficient between 2.0 and 5.0. This is the conditional information entropy obtained in the previous step; This represents the preset entropy tolerance threshold. The entropy tolerance threshold is determined by statistically extracting a baseline conditional information entropy sequence during a historical period of normal and stable equipment operation without failure. The expected value of this baseline conditional information entropy sequence is calculated, and then the sum of this value and twice the standard deviation is used to establish the entropy tolerance threshold. According to this mapping rule, when the conditional information entropy is much smaller than the entropy tolerance threshold, the data highly conforms to local operating patterns, and the resulting confidence weight smoothly approaches 1. Once the conditional information entropy exceeds the entropy tolerance threshold, the confidence weight rapidly decays exponentially to near zero.

[0050] S305, data cleaning for asynchronous observations is completed. Based on the dynamic evaluation and mapping logic described above, the system binds the calculated confidence weights in memory with the corresponding target asynchronous observations and writes them to disk. By assigning extremely low weights to high-uncertainty data, the system effectively suppresses the proportion of abnormal measurement data and high-frequency measurement noise in the global state. Thus, this embodiment abandons the traditional threshold discrimination method of directly removing abnormal data, and instead utilizes the continuous confidence coefficients obtained from multi-source cross-validation. While fully preserving the temporal continuity and objective causal integrity of the original sensor sequence, it completes data cleaning for asynchronous observations, thereby providing a highly robust numerical evaluation basis for the subsequent global state evaluation of the system.

[0051] See attached document Figure 1 and Figure 2 After completing the data cleaning for asynchronous observations in step S30, this embodiment executes step S40 to calculate the statistical window average of the confidence weights, perform topological edge pruning on the neighborhood dependency matrix based on the statistical window average, and calculate the topological survival rate of the neighborhood dependency matrix after performing topological edge pruning.

[0052] In real-world industrial settings, sensor correlation failures are often a continuous degradation process caused by dielectric deposits or equipment aging. The confidence weights derived from a single cleaning operation may be affected by data fluctuations caused by random factors. Directly adjusting the system structure based on these single weights can easily lead to frequent oscillations in the algorithm logic. By performing smoothing statistics on multi-source sequence data over time, the true state of the transmission relationships between nodes can be reflected more objectively and accurately. The above process specifically includes the following sub-steps: S401, Calculate the statistical window average of the confidence weights. In this embodiment, the system sets a fixed time span in memory as the statistical evaluation window. For upstream and downstream sensor nodes with edge relationships in the domain dependency matrix, the system extracts all confidence weights generated for the observation data between the two nodes within the statistical evaluation window. To ensure strict temporal alignment of multi-source data, the system only extracts weight data belonging to the same alignment evaluation period and constructs them together as a weight set. After obtaining the weight set, the system calculates the arithmetic mean of all extracted weight values ​​to obtain the statistical window average. The mathematical expression corresponding to this calculation step is as follows: ; In the formula, This represents the calculated average value of the statistical window. This represents the total number of confidence weights included within the extracted statistical evaluation window; Indicates the index subscript of the weight sequence; Indicates the first The system extracts confidence weights. Through this mean calculation logic, the system effectively filters out drastic single-time weight fluctuations caused by isolated noise. Its technical purpose is to provide a stable and reliable evaluation benchmark for subsequent topology adjustments.

[0053] S402 performs topological edge pruning on the domain dependency matrix based on the statistical window average. When the transmission relationship between upstream and downstream nodes is substantially blocked due to pipeline blockage or equipment downtime in actual working conditions, continuing to retain this connection in subsequent data fusion algorithms would lead to serious error propagation and wasted computational resources. Therefore, the system needs to cut off low-confidence topological connections to isolate failed nodes. As a preferred approach, to avoid one-sided judgments based solely on a single extreme value, the system introduces multi-dimensional judgment logic. Specifically, the system compares the calculated statistical window average with a preset pruning tolerance threshold and simultaneously checks whether the total number of confidence weights meets the preset minimum sample size requirement. If the statistical window average is lower than the pruning tolerance threshold, and the total number of confidence weights is greater than or equal to the minimum sample size requirement, the system determines that the corresponding spatial transmission relationship has failed.

[0054] Based on the determination result, the system obtains the row and column index coordinates of the failed upstream and downstream sensor nodes in the domain dependency matrix. The system accurately locates the memory region corresponding to these coordinates and forcibly overwrites the original element value at that location with zero, thereby completing the topological edge pruning operation on the domain dependency matrix. Regarding the determination of the above-mentioned determination parameters, engineers extract the lower bound value of the weight fluctuation as a reference by analyzing historical data of devices in a sub-optimal or slightly congested state offline. In this embodiment, the pruning tolerance threshold is typically between 0.2 and 0.4. The minimum sample size requirement is set according to the law of large numbers in statistics. In one embodiment, the minimum sample size requirement is set to 30 to ensure that the statistical results have sufficient significance and anti-interference ability.

[0055] S403, calculate the topology survival rate of the neighborhood dependency matrix after topology edge pruning. After pruning local edges, the system needs to evaluate the connectivity and remaining effective information of the global network structure. Therefore, this embodiment introduces the topology survival rate, which is the ratio of the number of effective propagation paths retained in the network to the total number of propagation paths initially configured. It is used to macroscopically measure the overall robustness of the current sensor network. The system counts the total number of non-zero elements in the neighborhood dependency matrix after topology edge pruning and divides this number by the initial total number of non-zero elements before pruning to obtain the topology survival rate. The corresponding mathematical expression is as follows: ; In the formula, This represents the calculated topology survival rate; This represents the total number of sensor nodes in the sensor network represented by the domain dependency matrix; and These represent the row index and column index variables of the matrix, respectively. This represents the first element in the neighborhood dependency matrix after performing topological edge pruning. Line number The element values ​​of the column; This represents the initial domain dependency matrix before pruning. Line number The element values ​​of the column; This is an indicator function that takes the value 1 when the condition within the parentheses is true (i.e., the element value is greater than zero), and takes the value less than 0 when the condition is false. ; This is a pre-defined minimal positive real number compensation term.

[0056] Considering that under extreme network-wide failure conditions, the number of non-zero elements in the initial matrix may approach zero due to widespread failure, to prevent division operations from causing the denominator to be zero and program crashes, the system forcibly adds a minimum value to the denominator. Performing algebraically safe smoothing, in one embodiment The value is set to 10. -6 Through the above logic, the system outputs a topology survivability value between 0 and 1. The closer the topology survivability value is to 1, the more intact the spatial propagation topology of the system is; the lower the topology survivability value, the more failure propagation paths exist in the network. At this point, the system has completed the dynamic mapping and feedback from the underlying observation data to the global topology state.

[0057] See attached document Figure 1 and Figure 2 After completing the topological edge pruning operation for the domain dependency matrix in step S40, this embodiment executes step S50, calculates the regularization coefficient based on the topological survival rate, performs weighted least squares data fusion on the asynchronous observations using the confidence weight and the regularization coefficient, and outputs the fused state vector.

[0058] Conventional multi-source data fusion methods can output stable state estimation results when processing complete measurement data. However, in actual operation, sensor connection breaks or large-scale data loss often occur, leading to rank deficiency in the directly constructed fusion information matrix, which in turn causes singularity errors during matrix inversion. To address this, this embodiment introduces a Tikhonov regularization mechanism into the data fusion architecture. By dynamically introducing the prior network topology connectivity state as a penalty term into the fusion calculation, the system can effectively constrain the fluctuation space of the solution and ensure the solvability of the equation system when the multi-source observation sequence is incomplete. The above process specifically includes the following sub-steps: S501, Calculate the regularization coefficient based on the topology survival rate. To control model complexity and suppress noise amplification, this embodiment introduces a penalty weight parameter called the regularization coefficient. To achieve adaptive adjustment of this parameter, the system reads the topology survival rate calculated in step S40 and establishes a negative correlation mapping logic between the topology survival rate and the regularization coefficient. The general technical principle is as follows: when the topology survival rate is high, it indicates good connectivity of the underlying sensor network, and the existing observation data is sufficient to support the accurate solution of the state equation. In this case, the system weakens the regularization penalty to preserve the original characteristics of the data; conversely, when the topology survival rate is low, it indicates that a large number of transmission paths have failed. The system increases the regularization penalty and uses historical prior smoothing constraints to compensate for the lack of current observation information. The mathematical expression corresponding to this mapping calculation is as follows: ; In the formula, This represents the calculated regularization coefficient; This represents the preset baseline regularization constant, used to provide basic matrix diagonal element compensation. Engineers typically set this baseline regularization constant between 0.01 and 0.05 for offline calibration based on the system's noise floor level. This represents the preset penalty scaling factor, used to limit the maximum increment of the penalty item, and its value range is set from 0.1 to 1.0; Represented by natural constant Exponential operations with base 0; This represents a sensitivity adjustment scalar used to control the nonlinear growth rate of the regularization intensity as the degree of topology disruption increases. In one embodiment, the value of this sensitivity adjustment scalar is set to 5.0. This indicates that the topology survival rate, calculated in step S40, is between 0 and 1.

[0059] In step S502, weighted least squares data fusion is performed on asynchronous observations using confidence weights and regularization coefficients. Before performing the fusion calculation, to address the time misalignment problem caused by inconsistent acquisition frequencies of multi-source data, the system uses a time alignment algorithm to align all asynchronous observations within the current fusion window to a unified timestamp. The specific mathematical logic for multi-source data time-series interpolation alignment can be directly implemented using conventional signal resampling algorithms, which are well-known techniques in the field and will not be elaborated here. After time alignment, the system extracts the aligned asynchronous observations and constructs an observation column vector by combining the extracted observation values ​​column by column. To reflect the reliability differences of different sensors under the current operating environment, the system retrieves the confidence weights mapped out in step S30 and arranges each confidence weight on the main diagonal of a diagonal matrix according to the order of the corresponding observation data, constructing a weighted diagonal matrix.

[0060] Based on the parameter matrix constructed above, the system establishes a weighted least squares objective functional including a regularization penalty term. By solving for the extrema of this objective functional, the system derives the closed-form solution for the optimal state estimate. The mathematical expression corresponding to this fusion calculation step is as follows: ; In the formula, This represents the calculated fusion state vector; The state observation matrix represents the system's state, which reflects the linear projection relationship between the state to be estimated and the underlying measurement data. In this embodiment, each row vector of the state observation matrix is ​​composed of the spatial position coordinates and sensitivity coefficients of the corresponding sensor nodes, so as to achieve an objective linear mapping of multidimensional physical quantities from the state space to the observation space. This represents the transpose of the state observation matrix; This represents a weighted diagonal matrix constructed based on the confidence level weights. This represents the regularization coefficient calculated in step S501; This represents the preset regularization constraint matrix. As a preferred approach, the system uses a unit diagonal matrix as the regularization constraint matrix. This represents an observation column vector consisting of a stack of aligned asynchronous observations; This represents the matrix inversion operation.

[0061] S503, output the fused state vector. Regarding the technical objective of the above fusion formula, the core of the conventional weighted least squares method lies in solving the matrix product. The inverse matrix. When some sensor data is significantly downweighted or even removed due to cleaning operations or topology pruning, the weight diagonal matrix... The presence of numerous zero elements in the matrix will cause the matrix product to approach a singular matrix, making inversion impossible. In this case, the system forces the superposition of positive definite matrices. Using dynamically allocated regularization coefficients This operation elevates the elements on the main diagonal of the matrix. Algebraically, this eliminates matrix singularities and the risk of divergence during inversion caused by local feature degradation.

[0062] Based on the aforementioned dynamic weighting and adaptive penalty logic, the system utilizes matrix algebra operations to accurately solve the objective equation. The system defines the final calculated set of multi-dimensional state parameters as a fused state vector. The fused state vector is a set of values ​​representing the complete operational attributes of the target device or system at a specific moment. In this embodiment, the fused state vector ultimately includes, but is not limited to, basic operating condition characteristic parameters such as the actual temperature, pressure, flow rate, and vibration frequency of key device nodes; the specific parameter dimensions depend on the initial sensing configuration of the system. The system writes the fused state vector into a designated cache as the final output. This output integrates multi-dimensional indicators such as spatial topological correlation, temporal smoothing characteristics, and local confidence assessment, thereby providing high-fidelity state information for subsequent system decisions and device control.

[0063] To aid in understanding the big data cleaning and fusion method for multi-source heterogeneous data proposed in this invention, a specific application example is given below, based on the actual operation scenario of a chemical plant's cooling water circulation network.

[0064] The pipeline network is equipped with multi-source heterogeneous sensors, including pressure sensor node A located upstream and pressure sensor node B located downstream. The hardware clock accuracy of the big data cleaning and fusion system for multi-source heterogeneous data is at the millisecond level.

[0065] Data Ingestion and State Assignment: Downstream pressure sensor node B generates an asynchronous observation with an initial measurement value of 1.2 MPa. The data access module extracts the original timestamp of this asynchronous observation. The big data cleaning and fusion system for multi-source heterogeneous data records the current system time and subtracts the original timestamp from the current system time to calculate the time difference. The time is 0.5 seconds. The big data cleaning and fusion system for multi-source heterogeneous data retrieves the dynamic decay rate constant initially configured on node B. (Set to 0.8), calculate the value of the exponential decay function with the natural constant e as the base: The big data cleaning and fusion system for multi-source heterogeneous data assigns 0.6703 as a state weight to the asynchronous observation and writes it into the cache slot corresponding to the time decay buffer matrix.

[0066] Delay-based closed-loop tracking and topology update: The big data cleaning and fusion system for multi-source heterogeneous data extracts the upstream sensor time series of upstream sensor node A and the downstream sensor time series of downstream sensor node B within a sliding window of the streaming processing. This window covers data points with a common time step. The system calculates the discrete cross-correlation function (CFC) of the upstream and downstream sensor time series and finds that the time offset corresponding to the global maximum value of the CFC is 4 seconds. Since 4 seconds falls within the preset historical reasonable delay range [3 seconds, 6 seconds] of this pipeline segment, the system determines this as the physical conduction delay between the multi-source heterogeneous sensors. Subsequently, the big data cleaning and fusion system for multi-source heterogeneous data addresses the memory block in row A and column B of the domain dependency matrix, and updates the original preset element values ​​in the domain dependency matrix using a physical propagation delay of 4. Simultaneously, the big data cleaning and fusion system for multi-source heterogeneous data retrieves the preset basic decay scalar. and the smallest positive real number compensation term Perform division to calculate and update the dynamic decay rate constant: The big data cleaning and fusion system for multi-source heterogeneous data feeds back the updated dynamic decay rate constant of 0.1 to the data access stage, replacing the original initial value of 0.8 of node B, so that the decay of the state weight of historical data slows down as the transmission time increases.

[0067] Data cleaning based on cross-validation: In subsequent operation cycles, node B generated a raw measurement data value of 2.8 MPa due to local electromagnetic interference. The big data cleaning and fusion system for multi-source heterogeneous data identified this raw measurement data as the target asynchronous observation. The system then read the updated domain dependency matrix and filtered out its associated sensor subset. Next, it extracted the historical observation data sequence and corresponding state weights of the associated sensor subset within the current evaluation statistical window, and calculated the weighted probability distribution. Finally, combining the target asynchronous observation and the associated sensor subset, the system calculated the conditional information entropy of the target asynchronous observation. A big data cleaning and fusion system for multi-source heterogeneous data retrieves a preset entropy tolerance threshold. With sensitivity adjustment coefficient The confidence weights are calculated using a nonlinear mapping activation function. The big data cleaning and fusion system for multi-source heterogeneous data binds the confidence weight to the target asynchronous observation and stores it on disk. The confidence weight rapidly approaches 0 over time (see attached figure for specific trends). Figure 3 As shown in subgraph (b) in the figure, data cleaning for asynchronous observations is completed through weight reduction processing.

[0068] Network topology edge pruning and evaluation: Suppose a branch of the pipeline experiences a substantial blockage. A big data cleaning and fusion system for multi-source heterogeneous data calculates the statistical window average of the corresponding node within the statistical evaluation window, using 50 confidence-weighted parameters. The result is 0.15. The big data cleaning and fusion system for multi-source heterogeneous data compares the statistical window average of 0.15 with the preset pruning tolerance threshold of 0.3. Since 0.15 is lower than 0.3, and the total confidence weight of 50 is greater than the preset minimum sample size requirement of 30, the system locates the coordinates of this branch node in the domain dependency matrix, forcibly overwrites its element values ​​to zero, and performs topological edge pruning. Before pruning, the initial number of non-zero elements in the domain dependency matrix was 20; after pruning, the number of non-zero elements in the domain dependency matrix is ​​reduced to 19. The system introduces a minimal positive real number compensation term. The topology survival rate was calculated as follows: .

[0069] Adaptive data fusion and output: The topology survivability calculated by the big data cleaning and fusion system for multi-source heterogeneous data is 0.95. The preset baseline regularization constant is then retrieved. Penalty scaling factor and sensitivity adjustment scalar Calculate the regularization coefficient: The big data cleaning and fusion system for multi-source heterogeneous data extracts aligned asynchronous observations to construct an observation column vector, and arranges the confidence weights from the previous mapping output on the main diagonal to construct a weighted diagonal matrix. Using the above parameter matrix and regularization coefficients, the system performs weighted least squares data fusion calculations. The regularization coefficients effectively increase the number of elements on the main diagonal, avoiding matrix divergence caused by extremely low confidence weights. Finally, the system outputs a fused state vector, which contains continuous pressure parameters of the current pipeline network, objectively restoring the true operating conditions while removing abnormal spikes through weight reduction (comparison results are attached). Figure 3 (as shown in subgraph (a)).

[0070] Based on the data processing effects of the above application embodiments, corresponding data processing effect diagrams are generated. (See attached diagram.) Figure 3 As shown, the Figure 3 It contains two subgraphs arranged vertically.

[0071] Appendix Figure 3 Subplot (a) shows a comparison between pressure observations and fused numerical values. The horizontal axis represents the system's operating time series (in seconds), and the vertical axis represents the pressure parameter (in MPa). The thin solid line marked with hollow circles represents the unprocessed asynchronous observations, and this curve shows a significant peak at the 5th second of the time series. The thick solid line represents the pressure parameter in the final output fused state vector of the big data cleaning and fusion system for multi-source heterogeneous data; this curve is smooth and unaffected by the peak of the thin solid line.

[0072] Appendix Figure 3 Subgraph (b) in the figure is a trend chart of confidence weight changes. The horizontal axis represents the system running time series (unit: seconds), and the vertical axis represents the confidence weight values. The dotted line represents the confidence weight mapped out by the big data cleaning and fusion system for multi-source heterogeneous data. The value of this curve approaches 0 at the moment when the asynchronous observations show a peak. Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for cleaning and fusing big data from multiple heterogeneous sources, characterized in that, Includes the following steps: Asynchronous observations from multi-source heterogeneous sensors are written into a time decay buffer matrix, and the state weights of the asynchronous observations in the time decay buffer matrix are calculated in combination with the dynamic decay rate constant. Calculate the physical conduction delay between the multi-source heterogeneous sensors, update the neighborhood dependency matrix using the physical conduction delay, and calculate the dynamic attenuation rate constant using the physical conduction delay; Based on the updated domain dependency matrix and the state weights, the conditional information entropy of the target asynchronous observation is calculated, and the conditional information entropy is mapped to confidence weights to complete the data cleaning for the asynchronous observation. Calculate the statistical window average of the confidence weights, perform topological edge pruning on the updated domain dependency matrix based on the statistical window average, and calculate the topological survival rate of the updated domain dependency matrix after performing topological edge pruning. The regularization coefficient is calculated based on the topological survival rate. The confidence weight and the regularization coefficient are used to perform weighted least squares data fusion on the asynchronous observations, and the fused state vector is output.

2. The big data cleaning and fusion method for multi-source heterogeneous data according to claim 1, characterized in that, The steps for calculating the state weights of the asynchronous observations in the time decay buffer matrix in conjunction with the dynamic decay rate constant include: Extract the original timestamps of the asynchronous observations; Calculate the time difference between the current system time and the original timestamp; The exponential decay function value is calculated with the natural constant as the base and the negative of the product of the dynamic decay rate constant and the time difference as the exponent. The value of the exponential decay function is used as the state weight and assigned to the asynchronous observation value written into the time decay buffer matrix.

3. The big data cleaning and fusion method for multi-source heterogeneous data according to claim 2, characterized in that, The steps for calculating the physical conduction delay between the multi-source heterogeneous sensors include: Extract the time series of upstream and downstream sensors within the sliding window of the stream processing; Calculate the discrete cross-correlation function between the upstream sensor time series and the downstream sensor time series; Find the time offset that makes the discrete cross-correlation function reach its global maximum and determine the physical conduction delay between the multi-source heterogeneous sensors.

4. The big data cleaning and fusion method for multi-source heterogeneous data according to claim 3, characterized in that, The steps for calculating the dynamic attenuation rate constant using the physical conduction delay include: Divide the preset basic attenuation scalar by the maximum value of the physical conduction delay and the first preset minimum positive real number compensation term, and use the quotient as the dynamic attenuation rate constant. The calculated dynamic decay rate constant is fed back.

5. A big data cleaning and fusion method for multi-source heterogeneous data according to claim 1, characterized in that, The steps for calculating the conditional information entropy of the target asynchronous observation based on the updated domain dependency matrix and the state weights include: Determine the target asynchronous observation and address the corresponding associated sensor subset; Calculate the weighted probability distribution based on the state weights; The conditional probability is obtained by dividing the joint probability density in the weighted probability distribution by the marginal probability density, and then the conditional information entropy of the asynchronous observation of the target is calculated by summing the probabilities.

6. A big data cleaning and fusion method for multi-source heterogeneous data according to claim 5, characterized in that, The steps of mapping the conditional information entropy to confidence weights to complete the data cleaning for the asynchronous observations include: Calculate the difference between the conditional information entropy and the preset entropy tolerance threshold, and multiply the preset sensitivity adjustment coefficient by the difference to obtain the first product; The first exponent value is calculated with the natural constant as the base and the first product as the exponent; The confidence weight is calculated by dividing constant 1 by the sum of constant 1 and the first exponent value. The confidence weights are bound to the target asynchronous observations and the data is written to disk to complete the data cleaning for the asynchronous observations.

7. A method for cleaning and fusing big data from multiple heterogeneous sources according to claim 1, characterized in that, The steps of performing topological edge pruning on the updated domain dependency matrix based on the statistical window average include: The average value of the statistical window is compared with the preset pruning tolerance threshold, and the total number of confidence weights is simultaneously verified to ensure that the preset minimum sample size requirement is met. If the average value of the statistical window is lower than the pruning tolerance threshold and the total number of confidence weights is greater than or equal to the minimum sample size requirement, obtain the row and column index coordinates of the failed upstream and downstream sensor nodes in the updated domain dependency matrix. Locate the memory region corresponding to the row index and column index coordinates, force the original element values ​​to be overwritten to zero, and perform the topology edge trimming.

8. A big data cleaning and fusion method for multi-source heterogeneous data according to claim 7, characterized in that, The steps for calculating the topological survival rate of the updated neighborhood dependency matrix after performing topological edge pruning include: Count the total number of non-zero elements in the updated domain dependency matrix after performing the topological edge pruning. The topology survival rate is obtained by dividing the total number of non-zero elements after the topology edge pruning by the sum of the initial total number of non-zero elements before pruning and the second preset minimal positive real number compensation term.

9. A method for cleaning and fusing big data from multiple heterogeneous sources according to claim 1, characterized in that, The steps for calculating the regularization coefficient based on the topology survivability include: Multiply the preset sensitivity adjustment scalar by the topology survival rate and take the negative number as the second product; The second exponent value is calculated with the natural constant as the base and the second product as the exponent; The third product is obtained by multiplying the preset penalty scaling factor by the second exponent value; The regularization coefficient is calculated by adding the preset baseline regularization constant to the third product.

10. A method for cleaning and fusing big data from multiple heterogeneous sources according to claim 9, characterized in that, The steps of performing weighted least squares data fusion on the asynchronous observations using the confidence weights and the regularization coefficients, and outputting the fused state vector, include: The asynchronous observations within the current fusion window are aligned to a unified timestamp using a time alignment algorithm. Extract the aligned asynchronous observations, and combine the extracted observation values ​​column by column to construct an observation column vector; The confidence weights are arranged on the main diagonal in the order of the corresponding observation data to construct a weight diagonal matrix; A weighted least squares objective functional containing a regularization penalty term is established based on the state observation matrix, the observation column vector, the weight diagonal matrix, the preset regularization constraint matrix, and the regularization coefficients. The fused state vector is obtained by solving the extreme value of the weighted least squares objective functional containing the regularization penalty term, and the fused state vector is written into the specified cache as output.

Images