Sensor data quality problem attribution method, system, device, and medium

Abstract

This invention relates to the field of industry data diagnostic analysis technology, and discloses a method, system, device, and medium for attributing sensor data quality problems. The method includes: dividing raw data from multiple data layers into data windows to obtain window data; calculating the data features of the window data to obtain window feature vectors, and calculating the basic feature distance between each window feature vector and a preset normal window feature vector; performing feature distance weighting correction on the basic feature distance to obtain a corrected distance, and identifying abnormal data windows based on the corrected distance; performing threshold attribution based on the abnormal data windows to obtain an average contribution ratio, and calculating the fault probability using a pre-constructed fault model; calculating the final fusion contribution and consistency index based on the average contribution ratio and the fault probability, and generating the attribution result. This invention can improve the accuracy of attributing sensor data quality problems.

Description

Technical Field

[0001] This invention relates to the field of industry data diagnostic analysis technology, specifically to a method, system, device, and medium for attributing sensor data quality problems. Background Technology

[0002] With the rapid development of smart cities, sensor monitoring systems have become a core means of real-time sensing of the operational status of urban lifelines (such as water supply, drainage, gas, and bridges). High-quality data means accurate, complete, timely, and consistent measurement information that can truly reflect the actual condition of the monitored objects. For example, in water supply projects, real-time data from sensors such as flow meters and pressure gauges, combined with existing parameters such as pipe diameter and pipe age, are used to assess the overall or regional health status of the water supply network, the risk of pipe leakage, and monitor pipe leakage in real time. In today's increasingly complex, networked, and intelligent urban lifeline systems, data is no longer just a reflection of the status of individual nodes, but also the foundation for system linkage and intelligent scheduling. Therefore, when data quality problems occur, being able to pinpoint the cause of the problem transcends purely technical considerations and becomes a crucial foundational task concerning urban resilience, public safety, and governance capabilities.

[0003] However, in actual operation, sensor monitoring data often deteriorates in quality due to various factors, including sensor aging, environmental interference, unstable communication links, abnormal data storage, and database processing failures, severely impacting data reliability and availability. When data anomalies occur, the fundamental challenge for operation and maintenance lies in the inability to quickly and accurately pinpoint the root cause of the problem. Data anomalies may originate from any link in the sensor terminal, transmission network, or data platform, or they may be caused by a combination of faults in multiple links, making it difficult to effectively distinguish between them using traditional methods.

[0004] Currently, most common anomaly handling methods stop at "discovering the problem" and lack the ability to systematically "locate the problem." Traditional methods mostly rely on a single dimension or simple rules for rough judgments, such as attributing data jumps to sensors while ignoring the possibility of instantaneous network packet loss or database write delays. This lack of attribution ability leads to blind and inefficient troubleshooting, often requiring maintenance personnel to check multiple potential fault points one by one based on experience, which not only prolongs system downtime but also significantly increases maintenance costs and complexity. In addition, existing methods usually lack a hierarchical and quantitative analysis framework for fault chains, failing to clearly reveal the specific contribution of each link to the current data problem, making it difficult to support accurate maintenance decisions.

[0005] Chinese patent application CN 115936484 A discloses a method and system for evaluating the operational status of an urban lifeline safety monitoring system. The main components include: determining a set of evaluation indicators for the monitoring system's operational status; obtaining a multi-level indicator judgment matrix and weights; and then calculating the indicators, weights, and scores for each level. Based on the weights and scores of each first-level indicator, a score for the operational status of the urban lifeline safety monitoring system is obtained, and the operational status of the urban lifeline safety monitoring system is determined based on the score. This method can perform quality assessments of the data system but cannot perform multi-level attribution analysis for data problems.

[0006] In conclusion, improving the accuracy of attributing sensor data quality problems has become an urgent issue to be addressed. Summary of the Invention

[0007] The technical problem to be solved by this invention is how to improve the accuracy of attribution of sensor data quality problems.

[0008] The present invention solves the above-mentioned technical problems through the following technical means:

[0009] Obtain raw data from multiple data layers, and divide the raw data into data windows to obtain window data; Calculate the data features of the window data to obtain window feature vectors, and calculate the basic feature distance between each window feature vector and a preset normal window feature vector; The basic feature distance is weighted and corrected to obtain the corrected distance for each window data, and abnormal data windows in the window data are identified based on the corrected distance. Threshold attribution is performed based on the abnormal data window to obtain the average contribution ratio of each data level, and the failure probability of each data level is calculated using a pre-built fault model. The final fusion contribution and consistency index of each data level are calculated based on the average contribution ratio and the failure probability, and attribution results are generated based on the final fusion contribution and the consistency index.

[0010] Optionally, calculating the data features of the window data to obtain the window feature vector includes: Calculate the continuous statistical characteristics of the signal layer data in the window data to obtain the signal layer feature sequence; Calculate the device characteristics of the device layer data in the window data and compare them with a preset device layer threshold to obtain the abnormal characteristics of the device layer data; Calculate the statistical characteristics of network latency, packet loss rate, and signal strength in the transport layer data within the window data, and identify abnormal features in the statistical characteristics; Calculate the data layer features corresponding to the database layer data in the window data and compare them with a preset data layer feature threshold to obtain abnormal features; Calculate the abnormal frequency features and time-series features corresponding to the abnormal features, and generate a window feature vector based on the abnormal frequency features, the time-series features and the signal layer feature sequence.

[0011] Optionally, the step of performing feature distance weighted correction on the basic feature distance to obtain the corrected distance for each window data includes: The basic feature distance is corrected using the following formula with feature distance weighting:

[0012] in, This represents the corrected base feature distance. Represents the distance to basic features. This represents the preset abnormal weight coefficient. This represents the abnormal characteristics in each data level.

[0013] Optionally, identifying anomalous data windows in the window data based on the corrected distance includes: The normal base feature distance for each data level is calculated based on the normal window feature vector. Construct a normal distance sequence corresponding to the normal window feature vector based on the normal basic feature distance; Calculate the anomaly detection threshold based on the normal distance sequence; Abnormal data windows are identified using the aforementioned anomaly detection threshold and the corrected distance.

[0014] Optionally, the step of performing threshold attribution based on the abnormal data window to obtain the average contribution ratio of each data level includes: Calculate the total correction distance of the abnormal data window based on the correction distance; The contribution of each data level is calculated based on the window feature vector; The contribution ratio of the abnormal data window at the corresponding data level is calculated based on the total correction distance and the contribution degree. The average contribution ratio of each data level is calculated based on the single-window contribution ratio of each abnormal data window.

[0015] Optionally, calculating the failure probability of each data level using a pre-built failure model includes: Obtain the multi-level feature vector of the abnormal data window; The multi-level feature vectors are passed along the decision tree splitting rules in the fault model to obtain the accumulated values ​​of the leaf nodes; The predicted values ​​of the leaf nodes are normalized using an activation function to obtain the fault probability of each data level.

[0016] Optionally, calculating the final fusion contribution and consistency index for each data layer based on the average contribution ratio and the failure probability includes: The average contribution ratio and the failure probability are weighted and summed to obtain the final fusion contribution of each data level; Construct the intersection and union of the average contribution ratio and the failure probability; The consistency index between the average contribution ratio and the failure probability is calculated based on the intersection and the union.

[0017] To address the above problems, this invention also proposes a sensor data quality problem attribution system, the system comprising: The data window partitioning module is used to acquire raw data from multiple data layers, partition the raw data into data windows, and obtain window data. The basic feature distance calculation module is used to calculate the data features of the window data, obtain the window feature vector, and calculate the basic feature distance between each window feature vector and the preset normal window feature vector. An abnormal data window identification module is used to perform feature distance weighting correction on the basic feature distance to obtain the corrected distance of each window data, and to identify abnormal data windows in the window data according to the corrected distance; The threshold attribution and model calculation module is used to perform threshold attribution based on the abnormal data window, obtain the average contribution ratio of each data level, and calculate the failure probability of each data level using a pre-built fault model. The attribution result generation module calculates the final fusion contribution and consistency index of each data level based on the average contribution ratio and the failure probability, and generates attribution results based on the final fusion contribution and the consistency index.

[0018] The present invention also provides a processing device, characterized in that it includes at least one processor and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the above-described method for attributing sensor data quality problems by calling the program instructions.

[0019] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, the computer instructions causing the computer to perform the above-described method for attributing sensor data quality problems.

[0020] The advantages of this invention are: This invention obtains windowed data by synchronously acquiring raw data from four data layers: signal, device, transmission, and database, and dividing it according to a preset data window size. This allows for control of the data scale and reflects the timeliness of the raw data in the data layers. By quantifying the overall abnormal data status into a clear responsibility ratio for each level (e.g., "equipment layer contributes 45%, transmission layer contributes 30%)", maintenance personnel can directly pinpoint the source of the problem, completely changing the inefficient model of relying on experience to troubleshoot step by step, and achieving precise guidance for fault tracing. By employing a parallel and fusion decision-making process combining threshold rules and machine learning modules, the system significantly enhances its intelligent diagnostic capabilities for complex and latent faults while ensuring high reliability of attribution results. The system operates in parallel with a threshold analysis method based on explicit rules and a data-driven machine learning model, weightedly fusing and verifying the consistency of their independent conclusions. This approach combines the transparency and stability of rule-based methods with the flexibility and adaptability of machine learning methods, enabling the system to explicitly handle common typical faults while intelligently identifying unknown and complex patterns. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a sensor data quality problem attribution method in one embodiment of the present invention; Figure 2 This is a functional block diagram of a sensor data quality problem attribution system provided in one embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Reference Figure 1 The diagram shown is a flowchart illustrating a sensor data quality problem attribution method according to an embodiment of the present invention. In this embodiment, the sensor data quality problem attribution method includes: S1. Obtain the original data of multiple data layers, and divide the original data of the data layers into data windows to obtain window data.

[0024] In this embodiment of the invention, the data hierarchy is the source of the original data, and the original data of each data hierarchy is referred to as the original data of the data layer. The original data of the data layer includes sensor time-series data of the signal layer, device metadata of the device layer, network transmission indicators of the transmission layer, and database performance data of the database layer. Among them, the sensor time-series data includes: real-time reading sequences of each monitoring point and their timestamps; the device metadata includes: sensor installation information, equipment age, maintenance records, and historical failure counts; the network transmission indicators include: communication quality logs such as communication latency, packet loss rate, and signal strength; and the database performance data includes: system performance parameters such as response time, number of connections, storage utilization, and data integrity indicators.

[0025] Furthermore, data windowing involves dividing the raw data of the data layer into multiple layers using a preset number of time windows, resulting in windowed data containing data from multiple data layers. The number of time windows should be flexibly set according to the actual application scenario, adhering to the principle that the minimum time window length should be greater than the acquisition frequency, and the maximum time window should not be less than the periodicity characteristic. For example, in a water supply project, the minimum periodicity duration that a pressure flow meter in a water supply pipeline can represent is one day. Window times can be selected such as 3 minutes, 30 minutes, 1 hour, 3 hours, 24 hours, 1 week, etc. By dividing the raw data of the data layer into data windows for each window time, large-scale data can be divided into smaller windows for parallel processing, reducing the complexity of a single computation, and improving the interpretability of data attribution from different time dimensions.

[0026] In this embodiment of the invention, the original data of each data layer is divided according to a preset data window size to obtain window data, which can control the data scale and reflect the timeliness of the original data of the data layer.

[0027] S2. Calculate the data features of the window data to obtain the window feature vector, and calculate the basic feature distance between each window feature vector and the preset normal window feature vector.

[0028] In this embodiment of the invention, the data features include the data features corresponding to the original data of each data layer. For example, data features are extracted from the window data of the signal layer, features are extracted from the device layer data, features are extracted from the transmission layer data, and features are extracted from the database layer.

[0029] Specifically, calculating the data features of the window data to obtain the window feature vector includes: Calculate the continuous statistical characteristics of the signal layer data in the window data to obtain the signal layer feature sequence; Calculate the device characteristics of the device layer data in the window data and compare them with a preset device layer threshold to obtain the abnormal characteristics of the device layer data; Calculate the statistical characteristics of network latency, packet loss rate, and signal strength in the transport layer data within the window data, and identify abnormal features in the statistical characteristics; Calculate the data layer features corresponding to the database layer data in the window data and compare them with a preset data layer feature threshold to obtain abnormal features; Calculate the abnormal frequency features and time-series features corresponding to the abnormal features, and generate a window feature vector based on the abnormal frequency features, the time-series features and the signal layer feature sequence.

[0030] In this embodiment of the invention, when the data in the window data is signal layer data, the sensor reading sequence is: The timestamp of the corresponding time window is The signal layer feature sequence comprises the mean, variance, extrema, and slope of the signal layer data. Specifically, the mean is extracted from the sensor data sequence. ,variance ,extremum , Slope: Calculated based on first-order difference. ,in Find the maximum slope Minimum slope Constant value ratio and number of jumps.

[0031] The constant value ratio and the number of jumps are calculated using the following formulas:

[0032]

[0033] in, Indicates a constant value ratio. This indicates the number of consecutive segments with the same value in the window data. Indicates the total number of data in the window. Indicates the number of jumps. , These represent the first and second data in the window. and the One data point, Indicates an indicator function, This indicates the preset jump parameters.

[0034] In detail, the number of segments with consecutive identical values ​​is the number of constant value segments in the window data. For example, in [1, 1, 2, 2, 2, 3], the constant value segments are [1, 1], [2, 2, 2], [3], so the number of segments with consecutive identical values ​​is 3. In [5, 5, 5, 5], the constant value segments are [5, 5, 5, 5], so the number of segments with consecutive identical values ​​is 1. In [1, 2, 1, 2], the constant value segments are [1], [2], [1], [2], so the number of segments with consecutive identical values ​​is 4.

[0035] Furthermore, the indicator function is used to determine whether the corresponding data meets the conditions, i.e. ,exist When the result is 1, If the condition is not met, the result is 0.

[0036] In another embodiment of the present invention, the device feature is that the sensor age is extracted from the device layer data: Term of the relationship: Historical number of failures: Data such as device features are used as the threshold for the device layer, which is defined as follows: (Specific thresholds are determined based on sensor characteristics and expert experience). Anomalies are identified by comparing preset equipment-level thresholds: Sensor age anomaly: Abnormal object age: Historical faults and anomalies: .

[0037] Furthermore, when the data in the window data is transport layer data, the statistical characteristics include: network latency: Packet loss rate: Signal strength: The preset thresholds for transport layer network latency, packet loss rate, and signal strength are defined as follows: (The specific threshold can be determined based on network characteristics and expert experience). Statistical characteristic calculations are performed on the data: Mean network latency: Mean packet loss rate: Mean signal strength: Based on statistical characteristics and thresholds, the following abnormal features of the transport layer are identified: Network latency anomalies: Abnormal packet loss rate: Abnormal signal strength: .

[0038] In another embodiment of the present invention, data layer features are extracted from the database layer and compared with a threshold. The selected data layer feature of the database layer is: data integrity missing rate. Data consistency violation rate: Data import delay: Data duplication rate: Database response time: Database connection usage: Database storage utilization: The corresponding database layer thresholds are: (Specific thresholds are determined through database performance benchmarks and expert experience). Anomaly characteristics are given based on the thresholds: Data integrity anomalies: Data inconsistency anomalies: Data import delay anomaly: Abnormal data duplication rate: Database response time anomaly: Abnormal database connection usage: Abnormal database storage usage: .

[0039] In this embodiment of the invention, the abnormal frequency characteristics, time series characteristics, and composite abnormality ratio are calculated using the following formulas:

[0040]

[0041]

[0042] in, , , These represent the anomaly frequency characteristics corresponding to device layer data, transport layer data, and database layer data, respectively. , , These represent the data in the device layer, transport layer, and database layer, respectively. One abnormal data point. This indicates the total number of abnormal data.

[0043] Furthermore, the time-series characteristics include the longest consecutive anomaly duration. Average duration of anomalies and the proportion of composite anomalies reflecting the severity of the anomalies. Specifically, the time series characteristics are calculated using the following formula:

[0044]

[0045]

[0046] Among them, the duration of continuous abnormal features represents the length of time during which abnormal features appear continuously in the device layer data, transmission layer data, and database layer data. The number of times abnormal features appear simultaneously represents the number of times abnormal features appear simultaneously in different layers of data, such as device layer data, transmission layer data, and database layer data, at the same time. The above formulas can be used to calculate the time-series features corresponding to the device layer, database layer, and transmission layer, respectively.

[0047] Furthermore, based on the signal layer feature sequence and the abnormal frequency data and timing features corresponding to the device layer data, database layer data, and transmission layer data, a window data feature vector, i.e., a real-time feature vector, is constructed, represented as: ,in, Represents the feature vector of window data. Represents the characteristic sequence of the signal layer. It represents the total number of consecutive statistical features in the signal layer feature sequence.

[0048] In this embodiment of the invention, the normal window feature vector includes a normal baseline feature vector composed of normal baseline features and a normal feature standard deviation vector composed of normal feature standard deviations. Using data from historical normal operation periods, i.e., the baseline period selecting a time period of stable system operation and barrier-free recording, the feature vector of each time window within the baseline period is calculated according to the method described above for calculating window feature vectors. Then, the mean of each feature is calculated as the normal baseline value feature, and the standard deviation corresponding to the normal baseline value feature is calculated to obtain the normal baseline standard deviation feature. Specifically, the normal baseline feature vector is represented as: The standard deviation vector of normal features is represented as: .in, Representing the signal layer feature sequences respectively Anomaly frequency characteristics corresponding to device layer data, transport layer data, and database layer data , , and longest continuous anomaly duration Average duration of anomalies and the proportion of composite anomalies reflecting the severity of the anomalies. The corresponding normal baseline feature vector; Representing the signal layer feature sequences respectively Abnormal frequency characteristics , , Longest continuous anomaly duration Average duration of anomalies and the proportion of composite anomalies reflecting the severity of the anomalies. The corresponding normal feature standard deviation vector.

[0049] Furthermore, the calculation of the normal window feature vector should be determined according to the characteristics of the system. The baseline should not remain unchanged. The normal baseline should be recalculated periodically to adapt to system changes, or the baseline should be re-established when there are significant changes in the system's hardware or software.

[0050] In detail, the characteristics of normal baseline values ​​can be calculated using the following formula:

[0051] in, Indicates the first Each feature corresponds to a normal baseline value feature. This indicates that the time window within the base period is... The first time One characteristic, This represents the total number of time windows within the base period.

[0052] Specifically, the basic feature distance between each window feature vector and the preset normal window feature vector is calculated using the following formula:

[0053] in, Represents the distance to basic features. The first element in the window feature vector represents the... One characteristic, This represents the total number of features in the window feature vector. Indicates the first Features Corresponding normal baseline characteristics Indicates the first Features The corresponding normal characteristic standard deviation.

[0054] In detail, the fundamental feature distance includes the hierarchical fundamental feature distance of each data level and the total fundamental feature distance obtained by summing the fundamental feature distances of all levels. The signal layer is calculated separately. Equipment layer Transport layer Database layer ( The basic feature distance of the window data is obtained by summing the basic feature distances of each level relative to the feature vector of the normal window.

[0055] In this embodiment of the invention, the signal layer is acquired synchronously. Equipment layer Transport layer Database layer The feature vectors of the four data levels are calculated, and the feature distance between them and the feature vectors of the normal window is calculated, which can provide a data foundation for subsequent calculation of the average contribution ratio of the data levels.

[0056] S3. Perform feature distance weighted correction on the basic feature distance to obtain the corrected distance of each window data, and identify abnormal data windows in the window data according to the corrected distance.

[0057] In this embodiment of the invention, the feature distance weighting correction is to use the abnormal features in each data level to weight and correct the corresponding basic feature distance, so as to obtain the corrected distance of each window data in the corresponding data level.

[0058] Specifically, the basic feature distance is corrected by a feature distance weighting using the following formula:

[0059] in, This represents the corrected base feature distance. Represents the distance to basic features. This represents the preset abnormal weight coefficient. This represents the abnormal characteristics in each data level.

[0060] Taking device-level data as an example, the basic feature distance is corrected using the following formula with feature distance weighting:

[0061] in, This represents the base feature distance after feature distance weighting correction of the device layer data. This represents the distance of the basic features of the device layer data. This represents the preset anomaly weighting coefficient for device-level data. This indicates abnormal characteristics in the device layer data.

[0062] Furthermore, the signal layer data, transport layer data, and database layer data, as described above, can be used to calculate the base feature distance after weighted correction of the feature distance: .

[0063] In this embodiment of the invention, identifying abnormal data windows in the window data based on the corrected distance includes: The normal base feature distance for each data level is calculated based on the normal window feature vector. Construct a normal distance sequence corresponding to the normal window feature vector based on the normal basic feature distance; Calculate the anomaly detection threshold based on the normal distance sequence; Abnormal data windows are identified using the aforementioned anomaly detection threshold and the corrected distance.

[0064] In this embodiment of the invention, for the base period The normal window feature vectors of each time window are used to construct the normal basic feature distance for each data level within each time window using the method described above for calculating the total corrected distance. Since the baseline window is normal and the abnormal features are zero, the normal basic feature vectors of each data level are summed to obtain the normal distance for that time window. These normal distances are then sorted according to the chronological order of the time windows to obtain the normal distance sequence. This can be represented as... : ,in, , These represent the normal distances for the first and second time windows, respectively. Indicates the first The normal distance of a time window.

[0065] Furthermore, the 99th percentile is used as the anomaly detection threshold: The anomaly detection threshold is obtained. If the total correction distance of the window data is greater than the anomaly detection threshold, the window data is determined to be an abnormal window; otherwise, it is a normal window. For example, for window data... The total corrected distance is obtained by summing the corrected distances for each data level. If its total correction distance If so, it is determined to be an abnormal data window.

[0066] S4. Perform threshold attribution based on the abnormal data window to obtain the average contribution ratio of each data level, and calculate the fault probability of each data level using the pre-built fault model.

[0067] In this embodiment of the invention, threshold attribution determines the main cause of anomalies in window data by setting a numerical limit. For example, from four candidate layers—signal layer, device layer, transmission layer, and database layer—those layers that contribute more than 25% on average in multiple anomaly windows are selected and placed into the main problem set to obtain the first problem set obtained by threshold attribution.

[0068] Specifically, the step of performing threshold attribution based on the abnormal data window to obtain the average contribution ratio of each data level includes: Calculate the total correction distance of the abnormal data window based on the correction distance; The contribution of each data level is calculated based on the window feature vector; The contribution ratio of the abnormal data window at the corresponding data level is calculated based on the total correction distance and the contribution degree. The average contribution ratio of each data level is calculated based on the single-window contribution ratio of each abnormal data window.

[0069] In this embodiment of the invention, the correction distance is the correction distance at different data levels within the abnormal data window. For example, the correction distance for device layer data within the abnormal data window is expressed as... The corrected distance of the signal layer data is expressed as The corrected distances for transport layer data and database layer data are respectively expressed as... , Then the total correction distance for:

[0070] Furthermore, the contribution of different data levels within the abnormal data window is calculated separately. Specifically, the contribution of each data level can be calculated using the steps described above for calculating the distance to the basic features. Taking the device layer as an example, the contribution of the device layer is calculated using the following formula:

[0071] in, Indicates the contribution of the device layer. Indicates the device layer. The device layer represents the first Each window feature vector Indicates the first Features Corresponding normal baseline characteristics Indicates the first Features The corresponding normal characteristic standard deviation.

[0072] Similarly, the contribution of the signal layer, transport layer, and database layer in each abnormal data window can be calculated. Next, for the... The total corrected distance of the anomalous data windows is used to calculate the single-window contribution ratio of each data level within the anomalous data window using the following formula:

[0073] in, , , , These represent the single-window contribution ratios of the signal layer, device layer, transport layer, and database layer, respectively. , , , These represent the contributions of the signal layer, device layer, transport layer, and database layer, respectively. This represents the total correction distance.

[0074] Furthermore, let the number of abnormal data windows be... For each data level The contribution ratio of each outlier data window at that data level is calculated individually, and the average value is taken to obtain the average contribution ratio of that data level.

[0075] in, , This indicates the total number of abnormal data windows. Indicates the first An abnormal data window at the data level The contribution ratio of a single window on the screen. Indicates data hierarchy The average contribution ratio.

[0076] In this embodiment of the invention, by calculating the average contribution ratio of different data levels in the abnormal data window, it is helpful to determine the main sources of problems that cause anomalies in the abnormal data window.

[0077] In this embodiment of the invention, the pre-built fault model can be a pre-trained multi-output gradient boosting tree model, which is an extension of the gradient boosting decision tree to multi-output regression tasks. Its basic principle is to sequentially construct multiple weak learners (decision trees), with each subsequent tree dedicated to correcting the prediction error of the preceding tree, and employing a gradient descent strategy for iterative optimization in the function space.

[0078] For multi-output scenarios, the multi-output gradient boosting tree model maintains an independent prediction function for each output dimension, but comprehensively considers the gradient information of all output dimensions when constructing the tree structure. In each iteration, the model simultaneously calculates the negative gradients of the four output dimensions (signal layer, device layer, transmission layer, and database layer), and determines the optimal splitting feature and splitting point based on the comprehensive gradient information.

[0079] The training process of the multi-output gradient boosting tree model: Training data comes from historical operation and maintenance records, including input features (four-level feature vectors extracted from each time window in history) and output labels (the distribution of true fault root causes determined based on historical fault maintenance records). The model uses a multi-output squared error loss function. in The number of training samples. For the true contribution matrix, For the model prediction matrix, Indicates the first The training data at the th th The actual output labels at each data level Indicates the first The training data at the th th The model predicts labels at each data level. This represents the total number of training data.

[0080] The training process includes an initialization phase, an iterative construction phase, and a termination condition determination. In the iterative construction phase, the prediction residuals of the current model are calculated, a new regression tree is constructed based on the residuals, the weight coefficients are determined through line search, and finally the new tree is added to the ensemble model.

[0081] Specifically, calculating the failure probability of each data level using a pre-built failure model includes: Obtain the multi-level feature vector of the abnormal data window; The multi-level feature vectors are passed along the decision tree splitting rules in the fault model to obtain the accumulated values ​​of the leaf nodes; The predicted values ​​of the leaf nodes are normalized using an activation function to obtain the fault probability of each data level.

[0082] In this embodiment of the invention, after the decision tree in the fault model receives the input multi-level feature vector, the multi-level feature vector first reaches the root node. The root node stores the splitting rules, which include which feature in the multi-level feature vector to use for judgment. The root node judges whether the feature after splitting should go to the left subtree or the right subtree in the decision tree. The entire judgment process is carried out recursively layer by layer until the current node is a leaf node (no child nodes) or the maximum depth limit is reached. The sum of the gradients of all samples covered by the leaf / (number of samples + regularization term) is obtained, which is the cumulative value of the node. The output values ​​of each decision tree leaf are added together to obtain the fault probability of each data level.

[0083] Among them, the multi-level feature vector represents the window feature vectors of four data levels in the abnormal data window: signal, device, transmission, and database. The signal layer features include the mean, variance, extreme values, slope, constant value ratio, and number of jumps of the sensor reading sequence; the device layer features include the sensor age, the object age, the number of historical failures, and the corresponding abnormality identifiers and device layer abnormality frequency; the transmission layer features include the mean network latency, the mean packet loss rate, the mean signal strength, and the corresponding abnormality identifiers and transmission layer abnormality frequency; the database layer features include the data integrity missing rate, the data consistency violation rate, the data entry delay, the data duplication rate, the database response time, the database connection utilization rate, the database storage utilization rate, and the corresponding abnormality identifiers and database layer abnormality frequency; the abnormal time series features include the longest continuous abnormality duration, the average abnormality duration, and the proportion of composite abnormalities.

[0084] Specifically, the pre-built fault model is input with a multi-level feature vector; the main processes include feature forward propagation (input features are passed along the decision tree splitting rules), leaf node cumulative value increment (accumulating the predicted values ​​of each tree), probability normalization (converting to a probability distribution using the softmax function), and main problem determination (determining the anomaly level based on probability thresholds). The final output is a four-dimensional probability distribution vector. ,satisfy .in, , , , This indicates the failure probability of the signal layer, device layer, transmission layer, and database layer.

[0085] In this embodiment of the invention, the abnormal data status at the overall device level is quantified into a clear responsibility ratio for each level (e.g., "device layer contributes 45%, transmission layer contributes 30%"). This allows maintenance personnel to directly pinpoint the source of the problem, completely changing the inefficient mode of relying on experience to troubleshoot step by step, and achieving precise guidance for fault tracing.

[0086] S5. Calculate the final fusion contribution and consistency index for each data level based on the average contribution ratio and the failure probability, and generate attribution results based on the final fusion contribution and the consistency index.

[0087] In this embodiment of the invention, the average contribution ratio is obtained by averaging the single-window contribution ratios of the above-mentioned multiple abnormal data windows according to the data level. The final fusion contribution is calculated based on the average contribution ratio of each data level and the failure probability. The consistency index is the consistency between the average contribution ratio and the failure probability, including complete consistency, complete inconsistency, and partial consistency.

[0088] Specifically, the step of calculating the final fusion contribution and consistency index for each data layer based on the average contribution ratio and the failure probability includes: The average contribution ratio and the failure probability are weighted and summed to obtain the final fusion contribution of each data level; Construct the intersection and union of the average contribution ratio and the failure probability; The consistency index between the average contribution ratio and the failure probability is calculated based on the intersection and the union.

[0089] In detail, the average contribution ratio of each data level and the 50% weight of the failure probability are weighted to obtain the final fusion contribution of the data level.

[0090] Furthermore, the average contribution ratio constitutes problem set 1: , , This represents the preset average contribution ratio threshold, and the failure probability constitutes the problem set 2. The failure probability threshold , , These represent the average contribution ratio and failure probability of each data level, respectively.

[0091] Specifically, the consistency index for each data level is calculated using the following formula:

[0092] in, Indicators of consistency This indicates the number of elements in the problem set.

[0093] In this embodiment of the invention, the "consistency index" of the data level is not calculated separately for each data level, but rather the degree of consistency between the threshold attribution module and the machine learning fault model module on the "main problem level set".

[0094] Among them, in the formula for calculating the consistency index above, This represents a set of problems consisting of data levels whose average contribution ratio is greater than a preset threshold. This represents the set of questions output by the machine learning model. and The intersection represents the data level that both problem sets consider to be problematic, and the union represents the data level that at least one of the two problem sets considers to be problematic.

[0095] Specifically, It is a value between 0 and 1: in The threshold attribution is completely consistent with the judgment of the fault model; The threshold attribution is consistent with the judgment part of the fault model; Threshold attribution is completely inconsistent with the judgment of the fault model.

[0096] For example, suppose there are four data levels:

[0097] They represent: Signal layer; Equipment layer; Transport layer; Database layer.

[0098] Example 1: completely consistent Threshold attribution module output:

[0099] Machine learning fault model output:

[0100] but:

[0101] The number of elements in the intersection is 2.

[0102]

[0103] The number of elements in the union is also 2.

[0104] therefore:

[0105] This indicates that the two modules are completely identical, and ultimately, the main problem level can be directly determined as follows:

[0106] This refers to a "combined fault at the device layer and the transport layer".

[0107] Example 2: Partial Consistency Assuming the threshold attribution is based on the average contribution ratio: the average contribution ratios of the signal layer, device layer, transport layer, and database layer are 0.18, 0.42, 0.31, and 0.09, respectively. If the preset average contribution ratio threshold is 0.25, then:

[0108] The fault probabilities output by the fault model are as follows: 0.12, 0.35, 0.189, and 0.35 for the signal layer, device layer, transmission layer, and database layer, respectively. If the fault probability threshold is also 0.25, then:

[0109] at this time:

[0110] The number of elements in the intersection is 1.

[0111]

[0112] The union has 3 elements.

[0113] therefore:

[0114] This indicates that the threshold attribution is consistent with the judgment part of the fault model, both considering the equipment layer... There's a problem, but it applies to the transport layer. and database layer There are disagreements.

[0115] The next step is to further filter based on the final contribution to the fusion. For example:

[0116]

[0117]

[0118] in, , , Representing the device layer Transport layer and database layer The corresponding final fusion contribution.

[0119] If a strict threshold of 0.3 is used for partial consistency, then only the device layer satisfies the following:

[0120] Therefore, the final problem level is:

[0121] This means "device hardware failure leading to data anomalies".

[0122] Example 3: Complete Conflict Threshold attribution module output:

[0123] Fault model output:

[0124] but:

[0125] The number of elements in the intersection is 0.

[0126]

[0127] The union has 2 elements.

[0128] therefore:

[0129] This indicates a complete conflict between the two modules. The logic in the documentation at this point is: Marked as a high-uncertainty scenario; Manual verification is recommended; If no human intervention is required, the level with the highest fusion contribution exceeding the specified threshold will be selected. If the highest fusion contribution degree is not high enough, output "uncertain".

[0130] The problem of "how to determine which specific data level", the key is not determined by alone. The role is only to judge how high the consistency degree of the two modules is.

[0131] Which specific level has a problem depends on: The set selected by the threshold attribution module ; The set selected by the machine learning module ; The intersection and union of the two; The final fusion contribution degree of each level .

[0132] In the embodiments of the present invention, based on value grading decision - if = 1, directly adopt the common result. If 0 < J < 1, take the union of the results and screen out the final problem set with a more strict threshold (0.3). If J = 0, mark it as a high-uncertainty scenario, and it is recommended to manually intervene for verification or conservatively select the level with the highest fusion contribution degree when there is no manual intervention. After completion, will be obtained, and then the corresponding conclusion is found from the preset mapping table (single-layer, double-layer, triple-layer, full-layer anomaly), and finally the attribution result described in natural language is output.

[0133] Among them, the preset mapping table is expressed as: Single-level anomaly: If , the conclusion is "pure business / environment problem or core sensor failure" If , the conclusion is "data anomaly caused by device hardware failure" If , the conclusion is "data anomaly caused by network transmission failure" If , the conclusion is "data anomaly caused by database processing failure" Double-level composite anomaly: If , the conclusion is "data anomaly caused by device layer problem" If , the conclusion is "data anomaly caused by transmission layer problem" If , the conclusion is "data anomaly caused by database layer problem" If , the conclusion is "device and transmission composite failure" If The conclusion is then "a combined device and database failure". like Therefore, the conclusion is "a combined transmission and database failure". Three-level composite anomaly: like Therefore, the conclusion is "a combined problem of equipment and transmission". like Therefore, the conclusion is "a combined device and database problem". like Therefore, the conclusion is "a combined problem of transmission and database". like The conclusion is then "a combined fault involving equipment, transmission, and database". Systematic anomalies across all levels: like The conclusion is "systematic complex failure".

[0134] Finally, maintenance suggestions are given based on the anomaly level: Recommendations for device-level issues: Check the sensor hardware status and replace the device if necessary; Recommendations for transport layer issues: Check network connection quality and optimize communication links; Database layer issues: Recommendations: Check database performance and optimize data processing flow; Recommendation for complex faults: Collaborate with multiple teams to troubleshoot and prioritize cases based on their contribution.

[0135] In this embodiment of the invention, by employing a parallel and fused decision-making process combining threshold rules and machine learning modules, the system significantly enhances its intelligent diagnostic capabilities for complex and latent faults while ensuring high reliability of attribution results. The system operates in parallel with a threshold analysis method based on explicit rules and a data-driven machine learning model, and performs weighted fusion and consistency checks on the independent conclusions of both. This approach combines the transparency and stability of rule-based methods with the flexibility and adaptability of machine learning methods, enabling the system to both explicitly handle common typical faults and intelligently identify unknown and complex patterns.

[0136] like Figure 2 The diagram shown is a functional block diagram of a sensor data quality problem attribution system provided in an embodiment of the present invention.

[0137] The sensor data quality problem attribution system 100 described in this invention can be installed in a processing device. Depending on the functions implemented, the sensor data quality problem attribution system 100 may include a data window partitioning module 101, a basic feature distance calculation module 102, an abnormal data window identification module 103, a threshold attribution and model calculation module 104, and an attribution result generation module 105. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0138] In this embodiment, the functions of each module / unit are as follows: The data window partitioning module 101 is used to acquire raw data of multiple data layers, partition the raw data of the data layers into data windows, and obtain window data. The basic feature distance calculation module 102 is used to calculate the data features of the window data, obtain the window feature vector, and calculate the basic feature distance between each window feature vector and the preset normal window feature vector. The abnormal data window identification module 103 is used to perform feature distance weighting correction on the basic feature distance to obtain the corrected distance of each window data, and to identify abnormal data windows in the window data according to the corrected distance. The threshold attribution and model calculation module 104 is used to perform threshold attribution based on the abnormal data window, obtain the average contribution ratio of each data level, and calculate the fault probability of each data level using a pre-built fault model. The attribution result generation module 105 calculates the final fusion contribution and consistency index of each data level based on the average contribution ratio and the failure probability, and generates attribution results based on the final fusion contribution and the consistency index.

[0139] The specific execution methods for the steps in each of the above modules are the same as the corresponding execution steps in the above sensor data quality problem attribution method.

[0140] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0141] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0142] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for attributing sensor data quality problems, characterized in that, include: Obtain raw data from multiple data layers, and divide the raw data into data windows to obtain window data; Calculate the data features of the window data to obtain window feature vectors, and calculate the basic feature distance between each window feature vector and a preset normal window feature vector; The basic feature distance is weighted and corrected to obtain the corrected distance for each window data, and abnormal data windows in the window data are identified based on the corrected distance. Threshold attribution is performed based on the abnormal data window to obtain the average contribution ratio of each data level, and the failure probability of each data level is calculated using a pre-built fault model. The final fusion contribution and consistency index of each data level are calculated based on the average contribution ratio and the failure probability, and attribution results are generated based on the final fusion contribution and the consistency index.

2. The sensor data quality problem attribution method as described in claim 1, characterized in that, The calculation of the data features of the window data to obtain the window feature vector includes: Calculate the continuous statistical characteristics of the signal layer data in the window data to obtain the signal layer feature sequence; Calculate the device characteristics of the device layer data in the window data and compare them with a preset device layer threshold to obtain the abnormal characteristics of the device layer data; Calculate the statistical characteristics of network latency, packet loss rate, and signal strength in the transport layer data within the window data, and identify abnormal features in the statistical characteristics; Calculate the data layer features corresponding to the database layer data in the window data and compare them with a preset data layer feature threshold to obtain abnormal features; Calculate the abnormal frequency features and time-series features corresponding to the abnormal features, and generate a window feature vector based on the abnormal frequency features, the time-series features and the signal layer feature sequence.

3. The sensor data quality problem attribution method as described in claim 2, characterized in that, The step of performing feature distance weighted correction on the basic feature distance to obtain the corrected distance for each window data includes: The basic feature distance is corrected using the following formula with feature distance weighting: in, This represents the corrected base feature distance. Represents the distance to basic features. This represents the preset abnormal weight coefficient. This represents the abnormal characteristics in each data level.

4. The sensor data quality problem attribution method as described in claim 1, characterized in that, The step of identifying abnormal data windows in the window data based on the corrected distance includes: The normal base feature distance for each data level is calculated based on the normal window feature vector. Construct a normal distance sequence corresponding to the normal window feature vector based on the normal basic feature distance; Calculate the anomaly detection threshold based on the normal distance sequence; Abnormal data windows are identified using the aforementioned anomaly detection threshold and the corrected distance.

5. The sensor data quality problem attribution method as described in claim 1, characterized in that, The step of performing threshold attribution based on the abnormal data window to obtain the average contribution ratio of each data level includes: Calculate the total correction distance of the abnormal data window based on the correction distance; The contribution of each data level is calculated based on the window feature vector; The contribution ratio of the abnormal data window at the corresponding data level is calculated based on the total correction distance and the contribution degree. The average contribution ratio of each data level is calculated based on the single-window contribution ratio of each abnormal data window.

6. The sensor data quality problem attribution method as described in claim 1, characterized in that, The calculation of the failure probability for each data level using a pre-built failure model includes: Obtain the multi-level feature vector of the abnormal data window; The multi-level feature vectors are passed along the decision tree splitting rules in the fault model to obtain the accumulated values ​​of the leaf nodes; The predicted values ​​of the leaf nodes are normalized using an activation function to obtain the fault probability of each data level.

7. The sensor data quality problem attribution method as described in claim 1, characterized in that, The step of calculating the final fusion contribution and consistency index for each data layer based on the average contribution ratio and the failure probability includes: The average contribution ratio and the failure probability are weighted and summed to obtain the final fusion contribution of each data level; Construct the intersection and union of the average contribution ratio and the failure probability; The consistency index between the average contribution ratio and the failure probability is calculated based on the intersection and the union.

8. A sensor data quality problem attribution system, characterized in that, include: The data window partitioning module is used to acquire raw data from multiple data layers, partition the raw data into data windows, and obtain window data. The basic feature distance calculation module is used to calculate the data features of the window data, obtain the window feature vector, and calculate the basic feature distance between each window feature vector and the preset normal window feature vector. An abnormal data window identification module is used to perform feature distance weighting correction on the basic feature distance to obtain the corrected distance of each window data, and to identify abnormal data windows in the window data according to the corrected distance; The threshold attribution and model calculation module is used to perform threshold attribution based on the abnormal data window, obtain the average contribution ratio of each data level, and calculate the failure probability of each data level using a pre-built fault model. The attribution result generation module calculates the final fusion contribution and consistency index of each data level based on the average contribution ratio and the failure probability, and generates attribution results based on the final fusion contribution and the consistency index.

9. A processing device, characterized in that, It includes at least one processor and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the method as described in any one of claims 1-7 by invoking the program instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the computer to perform the method as described in any one of claims 1-7.

Images