A consistency determination method fusing uncertainty estimation

Abstract

The application relates to the field of data determination, and discloses a consistency determination method fusing uncertainty estimation, which is used for consistency determination of data. The method comprises the following steps: obtaining measurement results and uncertainty information of multiple data sources, converting the measurement results and the uncertainty information into normalized mean and standard deviation, and recording sample size; for each pair of data sources, the Wasserstein distance of a probability model is calculated, the global consistency index is obtained by weighted average of the Wasserstein distances, and after the data set is combined, Monte Carlo resampling is carried out, virtual data sources are obtained, corresponding indexes are calculated, a critical threshold is determined, the global consistency index is compared with the threshold, consistency is determined, and when the consistency is not determined, an abnormal data source is located. The application proposes processing methods for the cases of missing uncertainty information, inconsistent data distribution assumption, weight value construction and overall data set outlier, ensures effective operation of the method in various complex scenes, and provides a comprehensive and reliable solution for consistency determination of multiple data sources.

Description

Technical Field

[0001] This invention relates to the field of data determination, and more particularly to a consistency determination method that integrates uncertainty estimation. Background Technology

[0002] In scientific research and engineering applications, data consistency assessment is a crucial step in ensuring data quality and improving model reliability. With the rapid development of sensor technology, the Internet of Things (IoT), and big data, measurement data from various data sources are increasingly prevalent in diverse application scenarios. In fields such as environmental monitoring, industrial control, medical diagnosis, and financial analysis, it is often necessary to fuse data from multiple sensors, systems, or institutions to obtain more comprehensive and accurate information. However, due to differences in the accuracy of measuring equipment, environmental noise interference, data transmission errors, and variations in measurement methods or assumptions from different data sources, measurement results from various data sources often exhibit uncertainty and may be inconsistent. If this inconsistency is not detected and addressed in a timely manner, it may lead to erroneous decisions or analytical results, thereby affecting the performance and reliability of the entire system.

[0003] Traditional consistency assessment methods are usually based on simple statistical indicators, such as mean, variance, or correlation analysis. However, these methods often ignore the uncertainty of the data and are difficult to fully reflect the differences between data sources. Comparing only the mean may ignore the dispersion of the data distribution, while comparing only the variance may ignore the location shift of the data.

[0004] Furthermore, traditional methods often lack a comprehensive consideration of data source reliability when processing multi-source data, leading to significant influence of abnormal or low-quality data sources on consistency determination results. In recent years, with the development of probabilistic models and uncertainty quantification techniques, how to integrate uncertainty information from data and construct more robust and comprehensive consistency determination methods has become a research hotspot in the field of data fusion.

[0005] Therefore, we propose a consistency determination method that integrates uncertainty estimation to address the above problems. Summary of the Invention

[0006] This invention provides a consistency determination method that integrates uncertainty estimation for determining data consistency.

[0007] The first aspect of the present invention provides a consistency determination method for integrating uncertainty estimation, the consistency determination method for integrating uncertainty estimation includes: acquiring measurement results and uncertainty information from multiple data sources, and determining the probability model parameters and sample size of the corresponding data sources; calculating the distribution distance between data sources based on the probability model parameters; calculating a global consistency index based on the sample size and the distribution distance; and determining whether the multiple data sources are consistent based on the comparison result of the global consistency index and a critical threshold.

[0008] Optionally, in a first implementation of the first aspect of the present invention, determining the probability model parameters of the corresponding data source includes: converting the measurement results and the uncertainty information into dimensionless normalized mean and normalized standard deviation; the normalized mean and the normalized standard deviation together constitute the probability model parameters.

[0009] Optionally, in a second implementation of the first aspect of the present invention, the method includes: identifying the representation type of the uncertainty information, converting it into an equivalent sample standard deviation through a conversion rule corresponding to the representation type; and linearly scaling the mean estimate extracted from the measurement results and the equivalent sample standard deviation based on a normalized reference range to obtain a normalized mean and a normalized standard deviation.

[0010] Optionally, in a third implementation of the first aspect of the present invention, the step of calculating the distribution distance between data sources based on the probability model parameters includes: for each pair of data sources, using the corresponding normalized mean and normalized standard deviation, calculating the Wasserstein distance between the probability models of the two data sources; and using the Wasserstein distance as the distribution distance.

[0011] Optionally, in a fourth implementation of the first aspect of the present invention, calculating the Wasserstein distance between the probability models of the two data sources includes: determining the correlation coefficient between the two data sources and using the correlation coefficient as an adjustment parameter; calculating the square of the difference between the two normalized means, the square of the difference between the two normalized standard deviations, and the product of the product of the two normalized standard deviations and a factor less than twice the difference of the adjustment parameter; summing the three calculated above and then taking the square root to obtain the Wasserstein distance.

[0012] Optionally, in a fifth implementation of the first aspect of the present invention, the method further includes: verifying whether the normalized mean and the normalized standard deviation of the two data sources conform to the normal distribution assumption; if not, converting the measurement results of the two data sources into discrete empirical distributions, and calculating the Wasserstein distance by performing point-by-point matching of the quantile sequences of the two discrete empirical distributions.

[0013] Optionally, in a sixth implementation of the first aspect of the present invention, the step of calculating the global consistency index based on the sample size and the distribution distance includes: constructing a weight value for each pair of data sources, wherein the weight value is positively correlated with the smaller sample size of the two data sources and negatively correlated with the product of the normalized standard deviations of the two data sources; and using the weight value to perform a weighted summation of the distribution distances of all paired data sources, and dividing by the total weight to obtain the global consistency index.

[0014] Optionally, in a seventh implementation of the first aspect of the present invention, the construction of the weight value further includes: calculating a reliability adjustment coefficient based on the normalized standard deviation of each data source and the sample size; multiplying the reliability adjustment coefficients of each pair of data sources to obtain a joint reliability factor; and using the joint reliability factor to correct the weight value.

[0015] Optionally, in the eighth implementation of the first aspect of the present invention, the critical threshold is determined by: merging the measurement results of all data sources into a whole dataset; performing resampling on the whole dataset to obtain multiple sets of virtual data sources; calculating the global consistency index corresponding to each set of virtual data sources to obtain multiple global consistency index sample values; and determining the critical threshold based on the statistical quantiles of all the global consistency index sample values.

[0016] Optionally, in the ninth implementation of the first aspect of the present invention, when there is a missing data source with missing uncertainty information, the method further includes: estimating the standard deviation estimate of the missing data source based on the statistical characteristic value of the normalized standard deviation of other data sources with valid uncertainty information; marking the sample size of the missing data source as invalid, and the data source marked as invalid does not participate in the construction of weight values.

[0017] The mechanism of this invention is as follows: under the original data structure, the statistical critical threshold is adaptively obtained to achieve an objective estimate of the consistency judgment boundary; when the judgment is inconsistent, the abnormal source can be automatically located based on the average Wasserstein distance of each data source, and it has robust handling of missing uncertainty and extended adaptability to non-normal distribution. Beneficial effects: It breaks down the analytical barriers caused by differences in uncertainty representation among different data sources, enabling various types of data to be compared and integrated under a unified framework, greatly expanding the scope of data applicability, and laying the foundation for consistency analysis of multi-source data; When constructing weight values, the product of the smaller sample size and normalized standard deviation from both data sources is considered. This allows data sources with smaller sample sizes but higher measurement accuracy to play a greater role in consistency determination, while reducing the influence of data sources with larger sample sizes but higher measurement dispersion, thus more reasonably reflecting the contribution of each data source to overall consistency. By fully considering the randomness and uncertainty of data, the critical threshold is made more representative and reliable, improving the accuracy of consistency determination. When inconsistencies are found, the average Wasserstein distance between each data source and other data sources is calculated to accurately locate the most outlier data source. This provides a clear objective for subsequent data calibration or removal operations, contributing to improved overall data quality. For data that does not conform to the normal distribution assumption, the Wasserstein distance is calculated using a discrete empirical distribution. This broadens the applicability of the method, enabling it to effectively determine consistency even in complex data distribution scenarios. Outlier preprocessing is performed on the overall dataset to remove extreme values, resulting in a virtual data source and a calculated critical threshold. This reduces the interference of outliers on consistency determination and improves the stability and reliability of the method in cases containing anomalous data. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of an embodiment of the consistency determination method that integrates uncertainty estimation in this invention. Figure 2 This is a schematic diagram illustrating the calculation of the Wasserstein distance between probability models for every two data sources. Figure 3 This is a schematic diagram of another embodiment of the consistency determination method that integrates uncertainty estimation in this invention. Figure 4 This is a schematic diagram of an embodiment of the consistency determination device that integrates uncertainty estimation in this invention. Detailed Implementation

[0019] This invention provides a consistency determination method incorporating uncertainty estimation for data consistency determination. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the consistency determination method integrating uncertainty estimation in this invention includes: 101. Obtain measurement results and their associated uncertainty information from multiple data sources, and convert the measurement results and uncertainty information into dimensionless normalized mean and normalized standard deviation, respectively, while recording the sample size of each data source; the conversion is based on the measurement range or known physical upper and lower bounds of each data source to obtain the normalized statistics of each data source, where the normalized mean and normalized standard deviation together constitute the probability model parameters of the data source.

[0021] It is understood that the executing entity of this invention can be a consistency determination device that integrates uncertainty estimation, or it can be a terminal or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as an example of the executing entity.

[0022] It should be noted that, taking the internal temperature monitoring of a chemical plant reactor as an example, the reactor's designed physical operating temperature has a lower limit of 1000 degrees Celsius and an upper limit of 1500 degrees Celsius, with a temperature range of 500 degrees Celsius.

[0023] Currently, there are three temperature sensors (i.e., three data sources) independently monitoring the same area in the network. The server obtains the following information and the original discrete data points: Data source A: The measurement mean is 1250 degrees Celsius, with an unnormalized standard deviation of 5 degrees Celsius and a sample size of 100.

[0024] Data source B: The measurement mean is 1255 degrees Celsius, with an unnormalized standard deviation of 8 degrees Celsius and a sample size of 80.

[0025] Data source C: The measurement mean is 1240 degrees Celsius, with an unnormalized standard deviation of 10 degrees Celsius and a sample size of 120.

[0026] The transformation logic is as follows: the normalized mean equals the original mean minus the lower bound divided by the span; the normalized standard deviation equals the original standard deviation divided by the span.

[0027] For data source A: 1250 minus 1000 equals 250, which is then divided by 500 to obtain the normalized mean of 0.5; 5 divided by 500 yields the normalized standard deviation of 0.01. The sample size is 100.

[0028] For data source B: The value exceeds the lower bound of 255; dividing by 500 yields a normalized mean of 0.51; dividing 8 by 500 yields a normalized standard deviation of 0.016. The sample size is 80.

[0029] For data source C: The value exceeds the lower bound of 240; dividing by 500 yields a normalized mean of 0.48; dividing 10 by 500 yields a normalized standard deviation of 0.02. The sample size is 120.

[0030] While extracting the aforementioned probabilistic model parameters, all the original discrete temperature data points collected by each of the three sensor networks were simultaneously converted into normalized measurement results according to the same rule (subtracting the lower bound and dividing by the span) and saved separately. These three sets of normalized statistics (A: 0.5 and 0.01; B: 0.51 and 0.016; C: 0.48 and 0.02) constitute their respective probabilistic model parameters.

[0031] 102. For each pair of data sources, use the normalized mean and normalized standard deviation of the two data sources to calculate the Wasserstein distance between the probability models of the two data sources. The Wasserstein distance is a dimensionless real number that represents the combined difference between the two probability models in terms of location and dispersion.

[0032] It should be noted that the server currently possesses parameters for three dimensionless probability models: A (mean 0.5, standard deviation 0.01), B (mean 0.51, standard deviation 0.016), and C (mean 0.48, standard deviation 0.02). The server performs pairwise combinations and comprehensive calculations: Pairing A and B together, the system assesses their deviation from each other in terms of center position, as well as their differences in fluctuation range, and integrates these two differences. Since A and B are relatively close in both mean and standard deviation, the system outputs a dimensionless value representing their combined difference, and the Wasserstein distance of the A-B pair is calculated to be approximately 0.0117.

[0033] Pairing A with C, the system comparison revealed a relatively increased deviation between A's mean of 0.5 and C's mean of 0.48; simultaneously, C's measurement uncertainty was significantly higher than A's. After inputting the location and scattering features into the processing module, the Wasserstein distance between them increased to approximately 0.0224, reflecting that the overall difference between A and C was greater than that between A and B.

[0034] Pairing B with C, the mean of B (0.51) and the mean of C (0.48) show the largest reading range among the three groups, and their inherent uncertainties also differ. Through the same comprehensive calculation process, the Wasserstein distance between B and C is approximately 0.0303.

[0035] The interactive comparison of all sensors was completed, and the three extracted Wasserstein distances not only represent the reading difference, but also incorporate the error range of the sensors themselves.

[0036] 103. Based on the normalized standard deviation and sample size of each data source, construct a weight value for each pair of data sources. This weight value is proportional to the smaller sample size of the two data sources and inversely proportional to the product of the normalized standard deviations of the two data sources. Then, use this weight value to perform a weighted average of all paired Wasserstein distances to obtain a global consistency index, which is a single dimensionless value.

[0037] It should be noted that the extracted parameters are: A (sample size 100, standard deviation 0.01); B (sample size 80, standard deviation 0.016); C (sample size 120, standard deviation 0.02). The distances are: A / B 0.0117, A / C 0.0224, B / C 0.0303.

[0038] To optimize underlying computing power consumption and prevent numerical overflow when calculating weights, the server directly introduces a scaling factor of one ten-thousandth in the logical operations for the initial weight calculation. For pairing A and B, a small sample size of 80 is extracted, and the product of their standard deviations is 0.00016. Dividing 80 by 0.00016 and then multiplying by one ten-thousandth yields an optimized initial weight of 5.

[0039] For pairing A and C, a small sample size of 100 is extracted, and the product of their standard deviations is 0.0002. Dividing 100 by 0.0002 and then multiplying by one ten-thousandth yields an optimized initial weight of 5.

[0040] For pairing B and C, a small sample size of 80 is extracted, and the product of their standard deviations is 0.00032. Dividing 80 by 0.00032 and then multiplying by one ten-thousandth yields an optimized initial weight of 2.5.

[0041] The server adds the three initial weights together to get a total weight of 12.5. Each initial weight is divided by the total weight to convert it into a normalized proportion: the proportion of pairing A and B is 40% (i.e., 0.4), the proportion of pairing A and C is 40% (i.e., 0.4), and the proportion of pairing B and C is 20% (i.e., 0.2).

[0042] Perform a weighted average calculation: multiply the distance between A and B (0.0117) by 0.4, add the distance between A and C (0.0224) by 0.4, add the distance between B and C (0.0303) by 0.2, and add them together to obtain a single dimensionless value of 0.0197, which is the global consistency index of the reactor temperature sensor network at that moment.

[0043] 104. Merge the normalized measurement results of all data sources into a single dataset. Perform Monte Carlo resampling on this dataset to obtain multiple sets of virtual data sources with the same number of data sources and matching sample sizes. Repeatedly calculate the global consistency index corresponding to each set of virtual data sources. Based on the results of these repeated calculations, determine a critical threshold, which is the statistical quantile of the global consistency index.

[0044] It should be noted that the system retrieves the discrete data points of the normalized measurement results from each data source, which were converted and saved synchronously in step 101. The system merges the 100 data points from source A, the 80 data points from source B, and the 120 data points from source C into a single dataset containing 300 normalized temperature data points. This dataset represents the statistical distribution of the combined measurement results from each sensor.

[0045] Monte Carlo resampling was performed on the entire dataset consisting of these 300 data points. Using random sampling with replacement, three virtual data sources were simulated: 100 data points were randomly selected to form virtual source A, 80 to form virtual source B, and 120 to form virtual source C. Since the data were completely shuffled and redistributed, the differences between them were solely due to purely random sampling. This operation represents a "null hypothesis" distribution in a physical and statistical sense, assuming that the generated virtual baseline represents "all data sources originally from the same population." The calculated global consistency index then characterizes the baseline limit when there is no systematic bias.

[0046] For this set of virtual data sources, the server repeatedly executes the aforementioned logic of calculating normalization parameters, pairwise Wasserstein distances, and sample size weighting to obtain the virtual global consistency index under this random sampling. To obtain a rigorous benchmark, the server repeats the above process 10,000 times, collecting 10,000 virtual global consistency indices generated by random fluctuations.

[0047] The 10,000 values ​​are sorted in ascending order, and the 95th percentile is selected based on project requirements. The system locates the value corresponding to the 9500th position and reads the result as 0.0225. This 0.0225 is the determined critical threshold, meaning that the deviation caused by purely random factors has a 95% probability of not exceeding this value.

[0048] 105. Compare the global consistency index with the critical threshold. If the global consistency index is less than the critical threshold, it is determined that there is consistency among all data sources; otherwise, it is determined that there is inconsistency. Under the assumption that there is at most one significantly deviating anomaly source, the most deviating anomaly data source is located based on the average Wasserstein distance between each data source and all other data sources.

[0049] It should be noted that during a new monitoring cycle in the system's operation, environmental factors caused data fluctuations. The server retrieved the Wasserstein distances between each pair of data sources in the new cycle: the distance between data source A and data source B was 0.0120, the distance between data source A and data source C was 0.0450, and the distance between data source B and data source C was 0.0480. The weighting ratios remained the same as those calculated in step 103: 0.4, 0.4, and 0.2. The server calculated the current global consistency index: 0.0120 multiplied by 0.4, plus 0.0450 multiplied by 0.4, plus 0.0480 multiplied by 0.2, resulting in a current global consistency index of 0.0324.

[0050] Consistency determination: The index 0.0324 is directly compared with the simulated critical threshold of 0.0225. Since 0.0324 is greater than 0.0225, exceeding the allowable random fluctuation range, the server determines that there is inconsistency between the three temperature sensor networks.

[0051] Initiate the anomaly localization procedure to calculate the arithmetic mean of the distances between each individual data source and all other data sources. The specific investigation data and results are shown in Table 1 below: Table 1 According to the table analysis, the average distance values ​​of data sources A and B are similar and at a low level, while the average distance of data source C (0.0465) is the highest, showing a deviation. Data source C is identified as an abnormal data source, and an alarm is issued to prompt on-site investigation of sensor C, thus completing the closed loop of consistency judgment and traceability.

[0052] Please see Figure 2 and Figure 3 Another embodiment of the consistency determination method incorporating uncertainty estimation in this invention includes: 201. Obtain measurement results and their associated uncertainty information from multiple data sources. Convert the measurement results and uncertainty information into dimensionless normalized mean and normalized standard deviation, respectively, and record the sample size of each data source. The conversion is based on the measurement range or known physical upper and lower bounds of each data source to obtain the normalized statistics of each data source. The normalized mean and normalized standard deviation together constitute the probability model parameters of the data source.

[0053] Specifically, for each data source, a mean estimate is extracted from the measurement results it provides, and the uncertainty representation type is identified from the uncertainty information it provides. Uncertainty representation types include sample standard deviation, standard error, confidence interval half-width, or maximum permissible error. Based on the uncertainty representation type, the uncertainty information is converted into an equivalent sample standard deviation using the corresponding conversion rules, while retaining the sample size of the data source. The conversion rules are as follows: when the uncertainty is the standard error, the standard error is multiplied by the arithmetic square root of the sample size to obtain the sample standard deviation; when the uncertainty is the confidence interval half-width, the half-width is divided by the quantile factor at the corresponding confidence level to obtain the sample standard deviation; when the uncertainty is the maximum permissible error, the maximum permissible error is divided by a fixed constant to obtain the sample standard deviation. The premise here is the assumption that the error follows a uniform distribution (if the error is known to follow a normal, rectangular, or other distribution, the corresponding conversion factor should be used).

[0054] Determine the normalized reference range for each data source. This range is determined based on the measurement range of the data source or on known physical upper and lower bounds shared by all data sources. If such a range cannot be obtained, the minimum and maximum values ​​of the measurement results from all data sources are used as a temporary range. Based on the normalized reference range, the mean estimate and sample standard deviation are linearly scaled to obtain dimensionless normalized mean and normalized standard deviation. The linear scaling should ensure that the normalized mean is between 0 and 1, and the normalized standard deviation is the original sample standard deviation divided by the width of the normalized reference range.

[0055] Each data source ultimately yields a set of parameters, including the normalized mean, normalized standard deviation, and sample size. The normalized mean is used to calculate the Wasserstein distance between data sources, while the normalized standard deviation and sample size are used to construct the weight values.

[0056] It should be noted that, taking the industrial natural gas pipeline pressure monitoring scenario as an example, the known lower limit of the physical operating pressure is 0 MPa, the upper limit is 10 MPa, and the reference range width is 10 MPa. Three data sources are currently connected, and the specific processing procedure is as follows: Extract the mean, sequence, and transform the uncertainty; Data source A: Provides a sequence containing 5 consecutive measurements (4.48, 4.49, 4.50, 4.51, 8.50 MPa, where 8.50 is a suspected interference jump value), the mean is estimated to be 4.5 MPa; the associated uncertainty is the "sample standard deviation" of 0.2 MPa based on the equipment factory calibration or historical prior assessment, with a sample size of 100. Due to type matching, the equivalent sample standard deviation is 0.2 MPa.

[0057] Data source B: Provides 5 consecutive measurements (mean estimated at 4.6 MPa); uncertainty is “standard error” of 0.025 MPa, and sample size is 64. Multiplying 0.025 by the square root of 64 (8) yields an equivalent sample standard deviation of 0.2 MPa.

[0058] Data source C: Provides 5 continuous measurements (mean estimated at 4.4 MPa); uncertainty is "maximum permissible error" of 0.3464 MPa, sample size is 50. Dividing this by a constant (approximately 1.732) corresponding to a uniform distribution yields an equivalent sample standard deviation of 0.2 MPa.

[0059] Linear scaling based on reference range: For data source A: the mean 4.5 divided by the width 10 yields a normalized mean of 0.45; the equivalent standard deviation 0.2 divided by 10 yields a normalized standard deviation of 0.02; simultaneously, each of its five original sequences is divided by 10 to obtain normalized sequences (0.448, 0.449, 0.450, 0.451, 0.850), and the sample size is kept to 100.

[0060] For data sources B and C: Using the same transformation method, data source B yields a normalized mean of 0.46 and a normalized standard deviation of 0.02, retaining a sample size of 64; data source C yields a normalized mean of 0.44 and a normalized standard deviation of 0.02, retaining a sample size of 50. The measurement sequences for both are also simultaneously scaled and saved. Thus, the heterogeneous data are mapped to a dimensionless space.

[0061] 202. For each pair of data sources, use the normalized mean and normalized standard deviation of the two data sources to calculate the Wasserstein distance between the probability models of the two data sources. The Wasserstein distance is a dimensionless real number used to simultaneously characterize the combined differences between the two probability models in terms of location and dispersion dimensions. Specifically, it is necessary to obtain the normalized mean and normalized standard deviation of two data sources: the normalized mean reflects the central position of the measurement results of the data source, and the normalized standard deviation reflects the dispersion of the measurement results of the data source. Determine if there is a known correlation coefficient between the two data sources: if it exists, use the correlation coefficient as an adjustment parameter to correct the overlap between the two probability models; if it does not exist, set the adjustment parameter to 0, indicating that the two data sources are independent. Based on the normalized mean, normalized standard deviation, and adjustment parameter, the Wasserstein distance between two data source probability models is calculated. The calculation steps are as follows: First, calculate the square of the difference between the two normalized means; second, calculate the square of the difference between the two normalized standard deviations; third, calculate the product of the two normalized standard deviations and multiply this product by 2 (1 minus the adjustment parameter) to obtain the combined term; then, sum the results of the above three steps; finally, take the square root of this sum to obtain the Wasserstein distance. The formula for calculating the Wasserstein distance distribution between the core probability models is expressed as follows: ; in, , For the normalized mean, , To normalize the standard deviation, Correlation coefficient (adjustment parameter); The Wasserstein distance is output as a dimensionless feature distance value, which reflects the overall similarity between the two probabilistic models in terms of positional offset and scattering differences, and is used for weighted average calculation in subsequent steps. Furthermore, before calculating the Wasserstein distance between the probability models of the two data sources, it is necessary to first verify whether the normalized mean and normalized standard deviation of the two data sources conform to the normal distribution assumption. If they do, the Wasserstein distance is calculated based on the normal distribution in accordance with claim 3. If they do not, the original measurement results of the two data sources (if they have been retained) or the quantile point set obtained by resampling from the overall dataset according to the sample size are used as two discrete empirical distributions. By matching the quantile sequences of these two discrete empirical distributions point by point, the squared difference at each quantile point is calculated and the average is taken. Then, the square root of this average is taken to obtain the Wasserstein distance between the discrete empirical distributions. The Wasserstein distance corresponding to this discrete empirical distribution is also output as a dimensionless feature distance value for subsequent weighted average calculation. It should be noted that, in the context of natural gas pipeline scenarios, the system currently stores three model parameters: A (mean 0.45, standard deviation 0.02), B (mean 0.46, standard deviation 0.02), and C (mean 0.44, standard deviation 0.02). The system performed tests on the three sets of normalized sequences, confirming that their background noise distributions all conformed to the normality assumption. Pairwise calculations were then performed. Regarding the pairing of A and B: The system search revealed that A and B are installed at adjacent nodes on the same main pipeline segment, indicating a known positive physical correlation. Therefore, the physical correlation coefficient was extracted. The system constructs a distance determination formula for fusion adjustment parameters (since the mean and standard deviation are dimensionless, the equation maintains dimensionless self-consistency): Substituting the data, we calculate: the squared mean difference is ; the squared standard deviation is 0; the combined term is . The sum of the three terms is 0.0005. Taking the square root, we find that the Wasserstein distance between A and B is approximately 0.02236 (denoted as 0.0224). Regarding the pairing of A and C: A and C are independent branches without physical coupling, and the adjustment parameter is set to 0. Substituting into the formula: the squared difference of the mean is ; the squared difference of the standard deviation is 0; the combination term without correlation is . The summation result is 0.0009, and taking the square root, the exact Wasserstein distance between A and C is 0.0300. Similarly, the distance between B and C (independent and uncoupled) can be calculated to be approximately 0.0346. 203. Based on the normalized standard deviation and sample size of each data source, construct a weight value for each pair of data sources. (Compared to the basic scenario, this embodiment introduces a joint reliability factor as an optimization and enhancement scheme for weight construction, targeting multi-source heterogeneous complex interference scenarios). If there are data sources with missing uncertainty information during the parameter acquisition stage, a missing information processing branch is triggered simultaneously (see the special processing logic described later), eliminating weight constructions with invalid states. This weight value is directly proportional to the smaller sample size of the two data sources and inversely proportional to the product of the normalized standard deviations of the two data sources; subsequently, this weight value is used to perform a weighted average of the Wasserstein distances of all pairs to obtain a global consistency index. This index is a single dimensionless value. Specifically, the normalized standard deviation and sample size of each data source need to be obtained; For each pair of data sources, the smaller sample size is determined as the first factor, and the normalized standard deviations of the two data sources are multiplied to obtain the second factor. Dividing the first factor by the second factor yields the weight value for that pair of data sources. The weight value is dimensionless and increases with the smaller sample size, and decreases with the product of the two normalized standard deviations. Wasserstein distance is used as the weighting factor for Wasserstein measurements, and the weight values ​​of each pair of data sources are used as the corresponding weighting coefficients. The weighted sum of the Wasserstein distances across all paired data sources is obtained, and the sum of the weight values ​​across all paired data sources is obtained, resulting in a weighted sum. The global consistency index is obtained by dividing the weighted sum by the weighted sum. The global consistency index is a dimensionless real number used to characterize the overall consistency of measurement results from all data sources.

[0062] Furthermore, before constructing the weight values ​​for each pair of data sources, a reliability adjustment coefficient must be calculated for each data source based on its normalized standard deviation and sample size. This reliability adjustment coefficient is directly proportional to the square root of the sample size and inversely proportional to the normalized standard deviation. When determining the weight values ​​for each pair of data sources, the reliability adjustment coefficients of each pair are multiplied together to obtain a joint reliability factor; then, the joint reliability factor is multiplied by the original weight value calculated based on the product of the smaller sample size and the normalized standard deviation to obtain the corrected weight value. All subsequent weighted average calculations use the corrected weight value instead of the original weight value.

[0063] It should be noted that the extraction parameters are as follows: A (sample size 100, standard deviation 0.02); B (sample size 64, standard deviation 0.02); C (sample size 50, standard deviation 0.02). Pairing distance: A-B = 0.0224, A-C = 0.0300, B-C = 0.0346.

[0064] Calculate the individual reliability coefficient; extract the square root of the sample size and divide it by the standard deviation. Among them, the calculation result of A is 10 divided by 0.02, which is 500; B is 8 divided by 0.02, which is 400; C is approximately 7.07 divided by 0.02, which is 353.5.

[0065] Calculate the original weights; for pairing A and B: the joint reliability factor is 500 multiplied by 400, which gives 200,000; then multiply by the smaller sample size of 64, and divide by the standard deviation product of 0.0004, resulting in an original weight of 320,000,000,000.

[0066] For pairing A and C: the joint reliability factor is 500 multiplied by 353.5, which gives 176750; multiply by 50 and divide by 0.0004, the original weight is approximately 22093750000.

[0067] For pairing B and C: the joint reliability factor is 400 multiplied by 353.5, which gives 141400; multiply by 50 and divide by 0.0004, the original weight is approximately 17675000000.

[0068] Overflow Prevention Normalization Mapping and Weighting: To address the risk of memory overflow caused by billions of values ​​in algorithm engineering, the system performs maximum value normalization mapping. Using the maximum weight value (32000000000 for A and B) as the benchmark, all values ​​are divided by this benchmark and multiplied by 100 to smoothly map to the interval [1, 100].

[0069] The final weights after mapping are: A and B = 100.00; A and C = approximately 69.04; B and C = approximately 55.23. The system calculates the total weight to be 224.27.

[0070] The weighted average is calculated as follows: (0.0224×100.00+0.0300×69.04+0.0346×55.23)÷224.27=(2.24+2.0712+1.9109)÷224.27=6.2221÷224.27≈0.0277.

[0071] The current global consistency index is 0.0277, eliminating the risk of weight overflow.

[0072] 204. Merge the normalized measurement results of all data sources into a single dataset. Perform Monte Carlo resampling on this dataset to obtain multiple sets of virtual data sources with the same number of data sources and matching sample sizes. Repeatedly calculate the global consistency index corresponding to each set of virtual data sources. Determine the critical threshold based on the repeated calculation results. This critical threshold is the statistical quantile of the global consistency index.

[0073] Specifically, the normalized mean of all data sources is used as the normalized measurement result, which is then combined with the sample size of each data source to form a unified dataset. This unified dataset contains information on the central location and sample size of all data sources.

[0074] Multiple resampling with replacement is performed on the overall dataset. Each resampling yields a set of dummy data, with the same number of data sources as the original data sources, and the sample size of each dummy data source matches the sample size of its corresponding original data source. The number of resampling operations is a preset positive integer.

[0075] For each set of virtual data obtained from resampling, the global consistency index corresponding to the set of virtual data is calculated according to the method of claim 1, thereby obtaining a global consistency index sample value with the same number of resampling times.

[0076] All global consistency index sample values ​​are sorted in ascending order, and the sample value at the preset percentile position after sorting is selected as the critical threshold. The critical threshold is a dimensionless real number. It is used to compare with the global consistency index to determine whether there is consistency between data sources.

[0077] Furthermore, before performing Monte Carlo resampling, outlier preprocessing is required on the overall dataset: For each data source containing multiple normalized measurements, the median and corresponding absolute median deviation of the measurements within that data source are calculated; each measurement is compared with the median of that data source, and if the deviation of a measurement from the median exceeds a preset multiple of the absolute median deviation, it is marked as an outlier and removed from the normalized measurements of that data source; after all data sources have completed outlier removal, the remaining unmarked measurements constitute the purified overall dataset; subsequent Monte Carlo resampling operations use the purified overall dataset as the sampling population, and the resulting virtual data source and corresponding global consistency index sample values ​​are calculated based on the purified data. The resulting critical threshold has robustness to extreme values ​​in the original data and is used for comparative analysis.

[0078] It should be noted that the continuous normalized measurement sequence retained in step 201 is extracted. The sequence for user A is (0.448, 0.449, 0.450, 0.451, 0.850). The system calculates the median of this sequence to be 0.450, and the absolute median difference to be 0.001. Assuming a preset tolerance factor of 10 (i.e., a threshold of 0.01), since the difference between the measured value 0.850 and the median is 0.400, far exceeding the threshold, the system identifies it as an outlier and removes it. User A retains 4 valid measurements.

[0079] Similarly, for B (5 values ​​distributed around 0.460) and C (5 values ​​distributed around 0.440), neither exceeded the threshold, and each retained 5 valid values. The system merged these 14 clean values ​​into a purified overall dataset as the sampling population.

[0080] The system initiates 1000 Monte Carlo resampling runs. In each loop, a center position is randomly selected from these 14 values ​​and assigned to virtual data sources A, B, and C, which inherit the original sample size (100, 64, 50) and standard deviation. The system recalculates the virtual global consistency index for these virtual combinations according to the logic in step 203.

[0081] After 1000 calculations, the system collects 1000 virtual indicators generated by random disturbances. These are sorted in ascending order, and the 95th percentile (the value at the 950th position) is extracted. Assuming this value is 0.0320, it is set as the system's critical threshold after excluding extreme value interference.

[0082] 205. Compare the global consistency index with the critical threshold. If the global consistency index is less than the critical threshold, it is determined that there is consistency among all data sources; otherwise, it is determined that there is inconsistency, and the most deviated abnormal data source is located based on the average Wasserstein distance of each data source to all other data sources.

[0083] Specifically, if the global consistency index is less than the critical threshold after comparison, a judgment result indicating consistency across all data sources is output; if the global consistency index is greater than or equal to the critical threshold, a judgment result indicating inconsistency among data sources is output, and an abnormal data source location operation is performed. In the abnormal data source location operation, for each data source, the average Wasserstein distance between that data source and all other data sources is calculated to obtain the average deviation of that data source. All data sources are sorted from largest to smallest according to their average deviation, and the data source ranked first is marked as the most deviating abnormal data source. To cope with complex scenarios, when multiple abnormal sources are acting simultaneously, an alternative iterative elimination enhancement strategy can be added (i.e., after eliminating the current most deviating data source, the global consistency index is recalculated until the remaining data sources are in a consistent state). Furthermore, the output global consistency index needs to be mapped to an operable industrial quality control standard (such as the corresponding acceptable relative error tolerance level on-site) to enhance the practical guiding significance of the judgment.

[0084] It should be noted that this causal localization logic is mainly for scenarios with independent random errors superimposed with abnormal biases. If there is systematic common-mode drift caused by environmental common-mode interference in the scenario (synchronous shift of readings of each sensor), the judgment index may not change significantly. In this case, it is necessary to introduce an external reference absolute truth condition for auxiliary verification.

[0085] It should be noted that an anomaly occurred in the pipeline network during subsequent high-frequency monitoring. The system extracted the pairing distances for the new cycle: the distance between A and B was 0.0220, the distance between A and C was amplified to 0.0550 due to the anomaly, and the distance between B and C was amplified to 0.0580. The normalized weights determined in step 203 were used (A and B 100.00, A and C 69.04, B and C 55.23, total weight 224.27).

[0086] The system calculates the current global consistency index: (0.0220×100.00+0.0550×69.04+0.0580×55.23) / 224.27=(2.20+3.7972+3.20334) / 224.27=9.20054 / 224.27≈0.0410.

[0087] The current indicator 0.0410 is compared with the critical threshold 0.0320. Since 0.0410 is greater than 0.0320, exceeding the allowable random fluctuation limit, the system determines that there is a consistency problem and triggers a troubleshooting operation. The arithmetic mean deviation of each node is shown in Table 2 below: Table 2 After sorting the deviations in descending order, data source C had the highest average deviation (0.0565). Based on this, the system marked data source C as an abnormal device and issued an alarm work order containing the device identifier, thus achieving closed-loop management from mathematical calculation to on-site guidance.

[0088] 206. When uncertainty information about a data source is missing, the system estimates the standard deviation of the missing data source using robust statistical methods based on a sample set of normalized standard deviations from all other data sources. The specific steps are as follows: calculate the median of the normalized standard deviations from all other data sources and use this median as the estimated standard deviation of the missing data source; simultaneously, mark the sample size of that data source as invalid. This invalid state indicates that: when constructing weight values, only data sources with valid sample sizes are included in the weighted average; the pairwise distance between the missing data source and other data sources is not included in the weighted average; the missing data source only participates in the calculation of Wasserstein distance and when merging the overall dataset.

[0089] Furthermore, during the Monte Carlo resampling process, for missing data sources marked as having invalid sample sizes, their virtual data are not directly generated from the overall dataset through resampling. Instead, they are randomly drawn from the normalized measurement results of all data sources with valid sample sizes, with the number of samples equal to the original sample size recorded when the missing data source was acquired. When constructing weight values ​​for the virtual data sources corresponding to this randomly drawn virtual data, their sample sizes are still marked as invalid, but the normalized standard deviation is included in the calculation using the median estimated in claim 7. Each set of virtual data sources thus obtained maintains the same missing data source structure and sample size distribution as the original data sources, thereby obtaining a critical threshold consistent with the original consistency criteria for subsequent comparison.

[0090] It should be noted that the newly added data source D in this period only returns the measurement mean and sample size (50), and the uncertainty information is missing. The normalized standard deviations of the other valid data sources are: A 0.018, B 0.020, and C 0.024.

[0091] The effective standard deviation sequence (0.018, 0.020, 0.024) was extracted, and the median of 0.020 was taken as the normalized standard deviation estimate of data source D. Simultaneously, the sample size of D was marked as invalid, as shown in Table 3 below. Table 3 When proceeding to step 203, the system calculates the pairing distance containing D (to ensure that step 205 can still identify D if it drifts), but forcibly skips the construction of all pairing weights containing D to avoid unverified data affecting the global consistency assessment.

[0092] The original scheme's computational redundancy was corrected: Upon entering step 204, the resampling process, the system identifies D as "invalid" and directly excludes it from the resampling loop. Instead of extracting virtual data for D from the overall dataset, the system constructs an objective consistency benchmark threshold purely based on three trusted data sources: A, B, and C. This optimization preserves the monitoring function for abnormal nodes, ensuring the overall system's efficiency and data integrity.

[0093] Figure 4 This is a schematic diagram of a consensus determination device 300 for fusion uncertainty estimation provided by an embodiment of the present invention. This consensus determination device 300 for fusion uncertainty estimation can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the consensus determination device 300 for fusion uncertainty estimation. Furthermore, the processor 310 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the consensus determination device 300 for fusion uncertainty estimation.

[0094] The consistency determination device 300 integrating uncertainty estimation may further include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4 The illustrated consistency determination device structure for fusion uncertainty estimation does not constitute a limitation on the consistency determination device for fusion uncertainty estimation. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0095] The present invention also provides a consistency determination device for fusion uncertainty estimation, the consistency determination device for fusion uncertainty estimation includes a memory and a processor, the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor performs the steps of the consistency determination method for fusion uncertainty estimation in the above embodiments.

[0096] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the consistency determination method for fusion uncertainty estimation.

[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0098] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of fusion of consistency decision with uncertainty estimation, characterized in that, include: Acquire measurement results and uncertainty information from multiple data sources, and determine the probability model parameters and sample size for the corresponding data sources; Based on the probability model parameters, the distribution distance between data sources is calculated; Calculate the global consistency index based on the sample size and the distribution distance; Based on the comparison results between the global consistency index and the critical threshold, it is determined whether the multiple data sources are consistent.

2. The consistency determination method for fusion uncertainty estimation according to claim 1, characterized in that, The determination of the probability model parameters corresponding to the data source includes: The measurement results and the uncertainty information are converted into dimensionless normalized mean and normalized standard deviation; The normalized mean and the normalized standard deviation together constitute the parameters of the probability model.

3. The consistency determination method for fusion uncertainty estimation according to claim 2, characterized in that, include: Identify the representation type of the uncertainty information, and convert it into an equivalent sample standard deviation using the transformation rule corresponding to the representation type; Based on the normalized reference range, the mean estimate extracted from the measurement results and the equivalent sample standard deviation are linearly scaled to obtain the normalized mean and normalized standard deviation.

4. The consistency determination method for fusion uncertainty estimation according to claim 2, characterized in that, The calculation of the distribution distance between data sources based on the probability model parameters includes: For each pair of data sources, the Wasserstein distance between the probability models of the two data sources is calculated using the corresponding normalized mean and normalized standard deviation. The Wasserstein distance is used as the distribution distance.

5. The consistency determination method for fusion uncertainty estimation according to claim 4, characterized in that, The calculation of the Wasserstein distance between the probabilistic models of the two data sources includes: Determine the correlation coefficient between the two data sources and use the correlation coefficient as an adjustment parameter; Calculate the square of the difference between the two normalized means, the square of the difference between the two normalized standard deviations, and the product of the product of the two normalized standard deviations and twice the difference of the adjustment parameter. Summing the three values ​​obtained above and then taking the square root yields the Wasserstein distance.

6. The consistency determination method for fusion uncertainty estimation according to claim 4, characterized in that, Also includes: Test whether the normalized mean and normalized standard deviation of the two data sources conform to the normal distribution assumption; If they do not match, the measurement results from the two data sources are transformed into discrete empirical distributions, and the Wasserstein distance is calculated by point-by-point matching of the quantile sequences of the two discrete empirical distributions.

7. The consistency determination method for fusion uncertainty estimation according to claim 2, characterized in that, The step of calculating the global consistency index based on the sample size and the distribution distance includes: For each pair of data sources, a weight value is constructed, which is positively correlated with the smaller sample size of the two data sources and negatively correlated with the product of the normalized standard deviations of the two data sources. The global consistency index is obtained by weighting and summing the distribution distances of all paired data sources using the weight values ​​and dividing by the total weights.

8. The consistency determination method for fusion uncertainty estimation according to claim 7, characterized in that, The construction of the weight values ​​also includes: The reliability adjustment factor is calculated based on the normalized standard deviation and the sample size for each data source; The reliability adjustment coefficients of each pair of data sources are multiplied together to obtain a joint reliability factor, and the weight values ​​are corrected using the joint reliability factor.

9. The consistency determination method for fusion uncertainty estimation according to claim 1, characterized in that, The critical threshold is determined in the following way: Merge the measurement results from all data sources into a single dataset; Resampling is performed on the overall dataset to obtain multiple sets of virtual data sources; Calculate the global consistency index for each group of virtual data sources to obtain multiple global consistency index sample values; The critical threshold is determined based on the statistical quantiles of all the global consistency index sample values.

10. The consistency determination method for fusion uncertainty estimation according to claim 2, characterized in that, When there is a missing data source with missing uncertainty information, it also includes: Estimate the standard deviation of the missing data source based on the statistical characteristics of the normalized standard deviation of other data sources with valid uncertainty information; The sample size of the missing data source is marked as invalid, and the data source marked as invalid does not participate in the construction of weight values.

Images