Data set quality analysis and evaluation method

Abstract

The invention belongs to the field of data quality analysis, and particularly discloses a data set quality analysis and evaluation method, which performs integrity evaluation on a current data set, starts data confidence analysis only after the number of data groups reaches the standard, ensures that confidence evaluation is established on the basis of data with a complete structure, avoids erroneous judgment caused by data missing, and improves the quality of the data set. Meanwhile, during data confidence analysis, dynamic comparison between current data and a local time sequence neighborhood is implemented by constructing an adjacent data set of each data group; meanwhile, a historical reference data set is combined, longitudinal comparison with a historical fluctuation range is carried out, and dual verification of local consistency and historical comparability is fused, so that the anomaly detection sensitivity and the evaluation accuracy are improved to a great extent; and finally, establishing a closed-loop feedback path with an acquisition end when availability judgment is carried out based on a data set integrity and confidence evaluation result, and realizing self-adaptive adjustment of an acquisition strategy according to a risk positioning result.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of data quality analysis, and specifically discloses a data set quality analysis and evaluation method. BACKGROUND

[0002] In the current digital era, data has become the core basis for business processes and management decisions, and is widely used in various fields. Data sources have shifted from traditional manual records to real-time collection by sensors, intelligent instruments and other devices. However, even if the data collected by high-precision instruments is of high quality, its quality may still be affected by various factors. In addition, modern data collection systems generally use continuous and batch acquisition of multiple sets of time series data. In this context, a single data point anomaly may trigger a chain of misjudgments. Therefore, after data collection is completed, the data set must be systematically evaluated for data quality.

[0003] There are technical solutions for data set quality analysis in the prior art. For example, a data set quality evaluation method, device and electronic equipment are provided in Chinese Patent No. CN116821203A. The data set is evaluated for quality based on basic evaluation indicators, and the basic evaluation result of the data set is obtained. In the case where the basic evaluation result indicates that the quality of the data set is pending, the data type of the data set is obtained. The data set is evaluated for quality based on specific evaluation indicators matching the data type, and the in-depth evaluation result of the data set is obtained.

[0004] For another example, a data set quality evaluation method, device, computer equipment and storage medium are provided in Chinese Patent No. CN113448955A. The data set is evaluated for quality based on the attribute characteristics of the data set in multiple evaluation dimensions, and the quality evaluation result of the data set in the multiple evaluation dimensions is obtained.

[0005] Although the above two solutions construct an analysis framework based on multi-dimensional indicators or hierarchical evaluation, there are still some key technical limitations. First, the structural integrity of the data set is not systematically analyzed. Data integrity is a prerequisite for quality evaluation, and its absence will lead to subsequent analysis based on incomplete samples, which is easy to cause evaluation bias or even misjudgment, especially in the presence of data loss or interruption in data collection. The evaluation result lacks representativeness and credibility.

[0006] Secondly, the quality evaluation of the data set mainly focuses on the static attributes of a single data point such as distribution, type and missing rate, and does not fully consider the essential characteristics of the data set as a time series continuous collection result. Modern data collection usually obtains multiple groups of related data in the form of time series, which has inherent time series dependence and local smoothness. However, the above method lacks modeling and analysis of the dynamic correlation of data within the local time series window, and cannot identify short-term mutations, step drifts or local abnormal fluctuations, resulting in insensitivity to local abnormalities and difficulty in distinguishing between normal fluctuations and real data degradation, thereby causing insufficient sensitivity of anomaly detection.

[0007] Thirdly, the evaluation result is only used for state labeling or alarm output, and no feedback channel is established with the data collection end, resulting in lack of active intervention and optimization ability for the collection process, limiting the adaptability and robustness in complex environments. SUMMARY

[0008] Therefore, one purpose of an embodiment of the present application is to provide a data set quality analysis and evaluation method, which effectively solves the problems existing in the prior art by evaluating the quality of the data set from the aspects of data integrity and data confidence.

[0009] The purpose of the present application can be achieved by the following technical solutions: a data set quality analysis and evaluation method, comprising the following steps: data integrity evaluation: counting the number of data groups of the current data set obtained by the data collection end, and comparing with a preset number threshold;

[0010] If the number of data groups meets the standard, the data confidence analysis step is executed;

[0011] If the number of data groups does not meet the standard, the distribution characteristics of the missing data group are identified to determine whether it is continuous missing or intermittent missing;

[0012] When it is determined to be continuous missing, an unavailable conclusion is directly generated;

[0013] When it is determined to be intermittent missing, local confidence analysis is performed based on the existing data groups;

[0014] Data confidence analysis: performing fluctuation analysis between adjacent data groups on the complete data set that meets the standard, and combining with comparison analysis with the historical data fluctuation range to determine whether there is a confidence risk;

[0015] Data availability decision: generating an availability determination conclusion according to the data integrity evaluation result and the data confidence analysis result, the conclusion including effective availability, partial availability or unavailability;

[0016] When it is determined to be effective availability, the current data set is retained, when it is partial availability, the effective data groups are retained and the invalid data groups are marked, and when it is unavailable, the current data set is discarded;

[0017] Collection optimization feedback: send re-collection instruction to the data collection end for part of the available and unavailable conclusions.

[0018] In combination with all the above technical solutions, the present application has the following positive effects: 1. The present application first performs integrity evaluation on the current data set collected by the data collection end, and only starts data confidence analysis after the number of data sets meets the standard, ensuring that the confidence evaluation is based on structurally complete data, avoiding misjudgment caused by data missing, improving analysis reliability, reducing invalid calculation overhead, and ensuring the accuracy and interpretability of the quality evaluation result.

[0019] 2. When performing data confidence analysis on the current data set, the present application constructs adjacent data sets of each data set to implement dynamic comparison of the current data and the local time sequence neighborhood; at the same time, longitudinal comparison with the historical fluctuation range is performed in combination with the historical reference data set, and dual verification of local consistency and historical comparability is fused to effectively identify local anomalies such as short-term mutation and drift, greatly improving the abnormal detection sensitivity and evaluation accuracy.

[0020] 3. When determining the availability based on the data set integrity and confidence evaluation result, the present application can dynamically trigger supplementary collection or parameter optimization instruction according to the risk positioning result by establishing a closed-loop feedback path with the collection end, realizing adaptive adjustment of the collection strategy, and effectively improving the data quality closed-loop control ability and the level of system self-optimization. BRIEF DESCRIPTION OF DRAWINGS

[0021] The present application is further illustrated by the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present application. For ordinary skilled persons in the art, other drawings can be obtained without creative labor on the basis of the following drawings.

[0022] Figure 1 The present application is further illustrated by the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present application. For ordinary skilled persons in the art, other drawings can be obtained without creative labor on the basis of the following drawings.

[0023] Figure 2 The present application is further illustrated by the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present application. For ordinary skilled persons in the art, other drawings can be obtained without creative labor on the basis of the following drawings.

[0024] Figure 3 The present application is further illustrated by the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present application. For ordinary skilled persons in the art, other drawings can be obtained without creative labor on the basis of the following drawings. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0026] See also Figure 1 As shown, the present invention proposes a data set quality analysis and evaluation method, comprising the following steps: data integrity evaluation: performing data group statistics on the current data set acquired by the data acquisition end, and comparing it with a preset quantity threshold;

[0027] If the number of data groups meets the requirements, perform data confidence analysis steps;

[0028] If the number of data groups does not meet the standard, the distribution characteristics of the missing data groups are identified and determined to be continuous missing or intermittent missing;

[0029] When it is determined that the continuity is missing, an unavailable conclusion is directly generated;

[0030] When it is determined to be discontinuous missing, local data confidence analysis is performed based on the existing data set;

[0031] As a preferred implementation of the above scheme, the distribution characteristics of the missing data group are identified and determined as continuous missing or discontinuous missing. Please refer to the following content: extract the identification numbers of all data groups from the current data set, and perform a set difference operation with the preset complete number sequence range to determine the number set of the missing data group.

[0032] It's important to understand that data collection is essentially the process of generating an ordered sequence of data in chronological or logical order. Each set of data is assigned a unique, increasing identification number, and ideally, the numbering sequence should be continuous and uninterrupted. When the number of data sets in a dataset falls below a preset threshold, it indicates missing data. By extracting the number set of the current dataset and performing a set difference operation with the theoretical complete number range, missing number sets can be accurately identified.

[0033] Arrange the number set of the missing data group in ascending order of values, then traverse the ordered sequence and divide the missing segments according to the numerical intervals between adjacent numbers. If the difference between adjacent missing numbers is greater than 1, it is regarded as the boundary of different missing segments, thereby dividing the missing number sequence into several sub-intervals with consecutive numbers, and each sub-interval constitutes a missing segment.

[0034] In the above, after sorting the missing numbers, the missing data are segmented according to the difference between adjacent missing numbers. In essence, this is a decomposition of the connected components of the missing set on the integer axis. Each missing segment represents a maximum continuous integer subinterval, reflecting the clustering of data loss in the sequence space.

[0035] The total number of missing segments and the length of each missing segment are counted, and the missing distribution characteristics are identified according to the following judgment criteria:

[0036] a) Continuous missing: the number of missing segments is 1, and the length of the segment is not less than 2;

[0037] b) Intermittent missing: the number of missing segments is not less than 2, or the number of missing segments is 1 but the length is 1.

[0038] In the above, the length of the missing segment is set to not less than 2 as the criterion for continuous missing, which excludes the interference of accidental single-point loss, and the number of missing segments is set to not less than 2 as the intermittent standard, which reflects the random characteristics of multiple and dispersion.

[0039] It should be noted that when the number of data groups does not meet the standard in the above integrity evaluation of the current data set, the data set is not directly determined to be invalid, but the distribution characteristics of the missing data groups are further analyzed, the purpose is to distinguish the causes and effects of missing, continuous missing usually reflects systematic failure, destroys the time sequence continuity of data, seriously affects the analysis reliability, and should be invalidated as a whole. While intermittent missing is often caused by random disturbances such as transient interference and packet loss, and if it is directly invalidated, it will lead to waste of valid data. Through local confidence analysis on such data sets, local anomalies can be identified and removed, and reliable data segments can be retained, realizing efficient utilization and quality classification and control of data resources.

[0040] Data confidence analysis: Perform fluctuation analysis between adjacent data groups on the complete data set that meets the standard, and determine whether there is a confidence risk by comparing and analyzing the fluctuation range with historical data.

[0041] As an optional implementation of the above scheme, the data confidence analysis process is as follows: based on the collection time stamp of each data group in the current data set, the neighborhood data group set corresponding to each data group is constructed in chronological order, that is, the target data group is taken as the center, and one or more adjacent data groups before and after it are selected to form a local time sequence window, forming a neighborhood data group set for comparative analysis.

[0042] For each monitoring index, the standard deviation of its value sequence in the neighborhood data group set is calculated, and compared with the preset index stability threshold. If the standard deviation of a certain monitoring index in the neighborhood data group set is higher than the index stability threshold, the arithmetic mean is used as the neighborhood reference value of the index, otherwise the median is used as the neighborhood reference value of the index.

[0043] The index stability threshold reflects the acceptable fluctuation degree of the monitoring index in the local time sequence neighborhood, is the decision boundary for distinguishing normal fluctuation and abnormal disturbance, and can be calculated by using the standard deviation distribution of the same index in the neighborhood window under the historical normal working condition, and the quantile is taken as the index stability threshold.

[0044] It should be noted that the standard deviation of the monitoring index in the neighborhood data set reflects the dispersion degree of the numerical sequence, the smaller the standard deviation, the more stable the fluctuation of the index in the local time sequence range, the more concentrated the data distribution, at this time, the arithmetic mean as the neighborhood reference value can more accurately represent the trend center, and the sensitivity and representativeness of the reference value are improved; the larger the standard deviation, the more significant the fluctuation or potential outliers, the more dispersed the data distribution, at this time, the median as the reference value has stronger robustness, can effectively suppress the influence of outliers, avoid reference deviation, and ensure the stability and reliability of subsequent comparative analysis.

[0045] The absolute deviation between the value of each monitoring index in the current data set and the corresponding neighborhood reference value is calculated, and the proportion of the absolute deviation relative to the neighborhood reference value is calculated as the fluctuation rate of each index in the neighborhood data set.

[0046] It should be further explained that in the time sequence sampling of the data set, the time interval between adjacent data sets is short, and has high time continuity and sequence correlation. Under normal working conditions, each data set and its neighborhood data should show a smooth change trend, and the fluctuation amplitude is small. When there is a significant fluctuation, it indicates that there is a confidence risk.

[0047] Based on the environmental parameters and the collection period of the current data set, a historical data set sequence with the same environmental configuration and period characteristics is retrieved from the historical reference database to construct a historical matching data set.

[0048] In the example applied to the above operation, the environmental parameters can be temperature, humidity, etc.

[0049] It should be understood that the historical reference database is retrieved based on the current collection environmental parameters and period characteristics, aiming to construct a context-matched historical data set to realize data comparability under consistent working conditions. Since the value of the monitoring index is significantly affected by environmental and time factors, if the full amount of historical data is directly used as a reference, working condition deviation will be introduced, resulting in inaccurate abnormality discrimination. Through context matching, the historical reference can reflect the typical behavior under the current conditions, and the accuracy and reliability of abnormality detection are improved.

[0050] The historical fluctuation range of each monitoring index is counted for the historical matching data set.

[0051] In the specific implementation of the above scheme, the mean ± k times the standard deviation in the historical matching data set can be used to define the fluctuation range.

[0052] The fluctuation of each data group and its original value are double-checked with the allowed fluctuation threshold and the historical fluctuation range, respectively.

[0053] If the fluctuation of an index exceeds the preset limit value or the original value of the index exceeds the corresponding historical fluctuation range, it is determined that the data group has a data confidence risk.

[0054] The fluctuation limit value of the monitoring index reflects the maximum relative change amplitude allowed in the local time neighborhood of the index, which is a key threshold for determining whether the data mutation exceeds the normal fluctuation range. The limit value can be determined according to the equipment operation specification of the collection terminal combined with the dynamic response parameters such as the maximum change rate provided by the manufacturer to meet the actual application requirements.

[0055] In the data confidence analysis, the present application comprehensively evaluates from the perspectives of neighborhood fluctuation and historical fluctuation: the neighborhood fluctuation reflects the dynamic consistency of the data in a short time window from the perspective of local time continuity; the historical fluctuation reflects the long-term change rule of the index under similar working conditions from the perspective of global statistical characteristics. The combination of the two realizes the fusion of local abnormal sensitivity and global trend robustness, improving the accuracy and reliability of data quality evaluation.

[0056] The local confidence analysis is as follows:

[0057] From all the missing data groups, extract the complete data subsequence that exists continuously, that is, the data segment that is preserved between the missing intervals and has continuous numbering or timestamp.

[0058] It can be understood that the local confidence analysis is for complete data subsequences rather than data segments containing missing data, because neighborhood comparison needs to rely on adjacent data to construct a local reference benchmark. If it is executed in the missing interval, the neighborhood data will be incomplete, the reference value will be distorted, and the analysis accuracy will be affected. Moreover, the complete data subsequence has temporal or logical continuity, which can truly reflect the dynamic behavior of the system in a short time window, and is the premise of evaluating local consistency. More importantly, even if the overall data set has missing data, there may still be high-quality continuous subsequences. By only performing confidence evaluation on complete subsequences, it is possible to identify and retain reliable data segments, and to achieve fine utilization of partially available data.

[0059] For each data group in the complete data subsequence, perform neighborhood data group set comparison and historical matching data set comparison. If the fluctuation of a data group corresponding to an index exceeds the preset allowed threshold or the original value of the index exceeds the corresponding historical fluctuation range, it is determined that the local confidence is not passed, otherwise it is determined that the local confidence is passed.

[0060] The above operation realizes re-evaluation of the credibility of the effective fragment in the non-complete data set, avoids denying the availability due to local data loss, and ensures the consistency of the evaluation standard by referring to the confidence analysis logic under the condition that the number of data groups meets the standard, thereby supporting subsequent reservation and utilization of high-confidence sub-sequences in the partially available data set and improving the data resource utilization rate.

[0061] Data availability decision: refer to Figure 2 As shown, the availability judgment conclusion is generated according to the data integrity evaluation result and the data confidence analysis result, and the conclusion includes effective availability, partial availability or unavailability.

[0062] Preferably, the data availability decision includes the following contents: when the number of data groups meets the standard and there is no confidence risk, the current data set is determined to be effectively available.

[0063] If the number of data groups meets the standard but there is a confidence risk, the identification numbers of all risk data groups are extracted and arranged in ascending order.

[0064] The minimum number and the maximum number are extracted from the sorted risk number sequence, and the risk span is defined as the difference between the maximum number and the minimum number.

[0065] It needs to be explained that the risk span represents the span of all risk data groups on the number axis, reflecting the distribution width.

[0066] The confidence risk aggregation degree is constructed based on the ratio relationship between the number of risk data groups and the risk span.

[0067] It needs to be further explained that the confidence risk aggregation degree is essentially a density index of risk data on the number axis, that is, the number of risk data groups in a unit span. When risk data groups appear continuously, their numbers are continuous, the risk span is small, and in the case of a fixed number, the aggregation degree value is high, reflecting the abnormal concentrated distribution; when risk data groups are distributed discretely, the number interval is large, the risk span is large, and the aggregation degree is low under the same number, indicating that the anomaly is dispersed. This index effectively distinguishes the spatial distribution characteristics of the anomaly.

[0068] The confidence risk proportion is calculated based on the number of risk data groups and the total number of data groups in the current data set.

[0069] The confidence risk proportion in the above is a measure of the relative size of risk data in the entire data set. Even if the risk is dispersed, if the overall proportion is too high, it may still affect the credibility of the overall analysis result, embodying the quality control thought of quantitative change causing qualitative change.

[0070] The product of the confidence risk aggregation degree and the confidence risk proportion is defined as a confidence risk index, and the confidence risk index is compared with a risk threshold value.

[0071] It should be explained that the confidence risk index is constructed by the product of the confidence risk aggregation degree and the confidence risk proportion, and is intended to comprehensively reflect the spatial concentration and overall influence size of the anomaly. The product form has a nonlinear amplification effect: when either dimension value is high, even if the other is moderate, the index can still rise; only when both are low can the index remain at a low level. This mechanism conforms to the quality determination logic that high risk needs to meet both local density and global significance.

[0072] The confidence risk threshold mentioned above reflects the risk tolerance boundary for determining whether the data set is unusable. Since the maximum value of the confidence risk aggregation degree is near 1, and the value of the confidence risk proportion is at most 1, this makes the values of the two under normal circumstances be in the range of 0 to 1. When multiplied, the confidence risk index has a nonlinear decay characteristic. Only when both are large does the confidence risk index approach a high level. When a single index is high, the product is still limited by the suppression of the other, and the overall value is relatively moderate. Therefore, the confidence risk threshold should not be set too high, otherwise it will lead to a missed judgment risk that most risk scenarios cannot trigger the determination.

[0073] When the number of data groups does not meet the standard and the local confidence passes, it is determined that the current data set is partially usable.

[0074] When the number of data groups does not meet the standard and the local confidence does not pass, it is determined that the current data set is unusable.

[0075] It can be understood that when the number of data groups meets the standard but there is a confidence risk, the present application does not use a simple binary determination such as directly classifying it as unusable or partially usable, but rather comprehensively considers the number and distribution characteristics of the risk data groups to construct a multi-dimensional evaluation model for hierarchical decision-making. This mechanism can effectively distinguish between isolated random anomalies and systematic data degradation, avoid over-discarding data sets due to local minor anomalies, improve data utilization efficiency, while ensuring the rigor of the determination in high-risk scenarios, and achieve the refinement of data usability evaluation.

[0076] It is further understood that when the present invention generates an availability judgment conclusion, if the number of data groups does not meet the standard but the local confidence is passed, it indicates that although there are missing data in the current data set, there are complete data subsequences with continuous numbers between the missing intervals, and after verification by neighborhood and historical comparison, its internal data consistency is good and the data is credible. Such subsequences contain valid information. If they are directly eliminated as a whole, it will lead to waste of useful data and reduce the system data utilization rate. Therefore, it is judged to be partially available, supporting the subsequent conditional use of credible subsequences. On the contrary, if the number of data groups does not meet the standard and the local confidence is not passed, it means that not only are there structural missing data, but also that significant abnormal fluctuations or deviations from historical patterns are still detected in the existing complete subsequences, indicating that the data quality problem has both integrity defects and content credibility failures, and it can no longer guarantee that any fragment meets the basic analysis requirements. At this time, it is judged to be unavailable to avoid misleading conclusions based on low-quality data and ensure the safety and reliability of data use.

[0077] When it is determined to be valid and available, the current data set is retained. When it is partially available, the valid data group is retained and the invalid data group is marked. When it is unavailable, the current data set is invalidated.

[0078] As a specific implementation of the above operation, see Figure 3 As shown, when part of the data is available, the valid data group is retained and the invalid data group is marked as follows: when the number of data groups meets the standard and is determined to be partially available, the risk data group is extracted from the current data set as the invalid data group for elimination, and the retained data group is recorded as the valid data group.

[0079] When the number of data groups does not meet the standard and is determined to be partially available, the data groups that do not participate in the local confidence analysis are extracted from the current data set as invalid data groups for elimination, and the data groups that participate in the local confidence analysis are taken as valid data groups.

[0080] It should be noted that if the number of data sets does not meet the requirement but is judged partially usable, this indicates that no anomalies were detected in the complete data subsequence after local confidence analysis, and the relevant data sets can be retained as valid data sets. However, data sets that did not participate in the confidence analysis are located in areas with incomplete boundaries and incomplete neighborhood information, making consistency verification impossible and their credibility unreliable. Marking them as invalid and removing them avoids the introduction of unverified and uncertain data, ensuring the dual reliability of the output results in terms of temporal continuity and data credibility, and improving the accuracy and security of subsequent analysis.

[0081] Collection optimization feedback: For some available and unavailable conclusions, re-collection instructions are sent to the data collection end.

[0082] Applying the optimized implementation of the above scheme, for some available and unavailable conclusions, sending the re-collection instruction to the data collection end includes the following content: when generating the unavailable conclusion, positioning the collection time interval when the data collection end collects the current data set and the associated collection equipment identifier.

[0083] Based on the equipment identifier, the operation record of the equipment in the above collection period is retrieved from the equipment operation log database.

[0084] By time alignment, it is identified from the operation record whether there is abnormal operation in the collection time interval.

[0085] Specifically, the actual operation parameters in the equipment operation record, such as working temperature, voltage, vibration amplitude, communication state, etc., are compared with the normal parameter range determined based on the equipment model and specifications. The reference parameter can be extracted from the equipment technical manual or manufacturer's specification. If a parameter deviates from the normal working interval, it is determined as abnormal operation behavior, and the parameter is marked as an abnormal feature.

[0086] According to the identified abnormal features, the corresponding sensor calibration parameter adjustment instruction is generated.

[0087] The generated calibration parameter adjustment instruction is sent to the data collection end to send the re-collection instruction.

[0088] It should be noted that when the data set is unavailable, it is usually caused by the lack of data continuity or the loss of confidence. Such problems are often caused by system-level exceptions of the collection terminal. If the root cause is not located and the re-collection is directly executed, the same fault may occur repeatedly, causing resource waste and unable to guarantee data quality. By analyzing the equipment operation state, identifying and correcting abnormal operation parameters, and implementing targeted equipment parameter optimization before re-collection, the subsequent collection can be based on stable system state and compliant configuration, effectively improving data integrity and reliability, avoiding the recurrence of similar defects, and realizing quality closed-loop control.

[0089] Applying the further optimized implementation of the above scheme, for some available and unavailable conclusions, sending the re-collection instruction to the data collection end also includes the following content: when generating the partially available conclusion, extracting the spatial position coordinates of the invalid data group and the missing data group.

[0090] Based on the spatial position coordinates, a supplementary collection area map is generated and sent to the data collection end.

[0091] Understandably, in spatial data collection, data quality is often closely tied to its physical or logical spatial location. Invalid or missing data sets directly correspond to blind spots or low-quality areas in space. Localized data loss can be estimated by interpolating adjacent data, but this approach is limited in accuracy. The most reliable method for data complementation is targeted supplemental acquisition near the missing location, driving the acquisition system to proactively optimize data coverage.

[0092] It should be pointed out that when some available judgment conclusions are generated, it is usually due to intermittent data loss or local confidence risk. Such problems are mostly caused by non-systematic factors such as instantaneous interference, communication packet loss or random environmental disturbances, reflecting occasional anomalies in the acquisition process. Given that such faults generally do not involve equipment parameter drift or system-level failures, there is no need for complex parameter tuning or mode reconstruction to avoid system state oscillation or configuration instability due to excessive intervention. At this time, by directly extracting the spatial position coordinates of failed and missing data groups, accurately locating areas with data quality defects, and generating targeted supplementary acquisition instructions, it is possible to selectively re-acquire only the problem areas while retaining existing high-quality data, effectively reducing redundant acquisition overhead, significantly improving data repair efficiency, optimizing resource utilization, and achieving fast and robust data integrity recovery.

[0093] In the innovative implementation of the above solution, the collection and optimization feedback also includes equipment health association:

[0094] The cumulative number of re-collection instructions triggered by the same collection device.

[0095] When the number of re-collection instructions exceeds the preset alarm threshold, an equipment maintenance instruction is generated.

[0096] The preset alarm thresholds mentioned above reflect the maximum reasonable frequency at which data collection equipment can trigger re-collection due to data quality issues within the statistical period. For specific settings, please refer to the equipment reliability indicators and maintenance cycle recommendations provided by the manufacturer.

[0097] This operation correlates abnormal behavior during data collection with device health monitoring. Frequently triggering recollection instructions indicates that the collection device has failed to obtain high-quality data after multiple attempts. This may indicate a potential hardware failure or reduced environmental adaptability. Promptly generating equipment maintenance instructions prevents the continuous influx of low-quality data into the analysis system, reducing the risk of misjudgment due to erroneous data. It also avoids repeated, ineffective collection tasks on faulty equipment, saving manpower, time, and energy costs.

[0098] The above embodiments may be implemented in whole or in part through software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0099] Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the examples disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. The various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality, without limitation. Depending upon the implementation, the techniques disclosed herein can each be fully implemented, in one or more specific purposes machines, enclosures, devices, systems or semiconductors, software, and / or various combinations thereof. Such examples are merely illustrative and can not be limited to any specific combination of hardware and / or software.

[0100] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist physically independently, or two or more modules can be integrated in one module.

[0101] The above descriptions are merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any modification or substitution within the technical scope disclosed by the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0102] Finally, the above descriptions are merely preferred embodiments of the present application, and are not intended to limit the present application, and any modification, equivalent substitution, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A data set quality analysis and evaluation method, characterized in that: The following steps are involved: Data integrity assessment: Count the number of data groups in the current data set obtained by the data acquisition end and compare it with the preset number threshold; If the number of data groups meets the requirements, perform data confidence analysis steps; If the number of data groups does not meet the standard, the distribution characteristics of the missing data groups are identified and determined to be continuous missing or intermittent missing; When it is determined that the continuity is missing, an unavailable conclusion is directly generated; When it is determined to be discontinuous missing, a local confidence analysis is performed based on the existing data set; Data confidence analysis: Perform fluctuation analysis on neighborhood data groups for complete datasets that meet the standards, and compare the fluctuation range with historical data to determine whether there is confidence risk. Data availability decision: Generates availability conclusions based on data integrity assessment results and data confidence analysis results. The conclusions include effectively available, partially available, or unavailable. If it is determined to be valid and available, the current data set will be retained. If it is partially available, the valid data set will be retained and the invalid data set will be marked. If it is unavailable, the current data set will be invalidated. Collection optimization feedback: Send re-collection instructions to the data collection end for some available and unavailable conclusions.

2. The data set quality analysis and evaluation method according to claim 1, wherein: The distribution characteristics of the missing data group are determined to be continuous missing or intermittent missing, as shown below: Extract the identification numbers of all data groups from the current data set, and perform set difference operation with the preset complete number sequence range to determine the number set of the missing data group; Arrange the number set of the missing data group in ascending order of value, then traverse the ordered sequence and divide the missing segments according to the numerical interval between adjacent numbers. If the difference between adjacent missing numbers is greater than 1, it is regarded as the boundary of different missing segments, thus dividing the missing number sequence into several consecutively numbered subintervals, each of which constitutes a missing segment; Count the total number of missing segments and the length of each missing segment, and identify the missing distribution characteristics based on the following criteria: a) Continuous missing: The number of missing segments is 1, and the length of the segment is not less than 2; b) Intermittent deletions: The number of missing segments is not less than 2, or the number of missing segments is 1 but the length is 1.

3. The data set quality analysis and evaluation method according to claim 1, wherein: The data confidence analysis is performed as follows: Construct the neighborhood data group set corresponding to each data group in chronological order based on the acquisition timestamp of each data group in the current data set; For each monitoring indicator, the standard deviation of its numerical sequence is calculated in the neighborhood data set and compared with the preset indicator stability threshold. If the standard deviation of a monitoring indicator in the neighborhood data set is higher than the indicator stability threshold, the arithmetic mean is used as the neighborhood reference value of the indicator; otherwise, the median is used as the neighborhood reference value of the indicator. For each monitoring indicator in each data set, the absolute deviation between its value and the corresponding neighborhood reference value is calculated, and the ratio of the absolute deviation to the neighborhood reference value is calculated as the volatility of each indicator in each data set relative to the neighborhood data set. Based on the environmental parameters and collection period of the current data set, a historical data set sequence with the same environmental configuration and period characteristics is retrieved from the historical reference database to construct a historical matching data set; Calculate the historical fluctuation range of each monitoring indicator based on the historical matching data set; Double-check the volatility of each indicator and its original value for each data group with the allowed volatility threshold and historical volatility range: If the volatility of an indicator in any data group exceeds the preset limit or the original value of the indicator exceeds the corresponding historical fluctuation range, the current data set is judged to have confidence risk and the data group is marked as a risky data group. Otherwise, the current data set is judged to have no confidence risk.

4. The data set quality analysis and evaluation method according to claim 3, wherein: The local confidence analysis process is as follows: Extract consecutive complete data subsequences from all missing data groups; Similarly, neighborhood data group comparison and historical matching data group comparison are performed on each data group in the complete data subsequence. If the volatility of a certain indicator corresponding to a data group exceeds the preset allowable threshold or the original value of the indicator exceeds the corresponding historical fluctuation range, the local confidence is judged to have failed. Otherwise, the local confidence is judged to have passed.

5. The method for analyzing and evaluating data set quality according to claim 4, wherein: Generating the availability determination conclusion based on the data integrity assessment results and the data confidence analysis results includes the following: When the number of data sets meets the requirements and there is no confidence risk, the current data set is considered valid and usable; If the number of data groups meets the standard but there is confidence risk, the identification numbers of all risk data groups are extracted and arranged in ascending order; Extract the minimum and maximum numbers from the sorted risk number sequence, and define the risk span as the difference between the maximum and minimum numbers; Construct the confidence risk aggregation based on the ratio of the number of risk data groups to the risk span; Calculate the confidence risk ratio based on the number of risk data groups and the total number of data groups in the current dataset; The product of the confidence risk concentration and the confidence risk ratio is defined as the confidence risk index. The confidence risk index is compared with the risk threshold. If the confidence risk index reaches the risk threshold, the current dataset is judged to be unusable. Otherwise, the current dataset is judged to be partially usable.

6. The method for analyzing and evaluating data set quality according to claim 5, wherein: Generating the availability determination conclusion based on the data integrity assessment results and the data confidence analysis results also includes the following: When the number of data sets does not meet the standard and the local confidence passes, it is determined that the current data set is partially available; When the number of data sets does not meet the requirements and the local confidence level fails, the current data set is deemed unusable.

7. The method for analyzing and evaluating data set quality according to claim 3, wherein: When the part is available, retain the valid data group and mark the invalid data group as follows: When the number of data groups reaches the standard and is determined to be partially available, the risk data groups are extracted from the current data set as invalid data groups for elimination, and the remaining data groups are recorded as valid data groups; When the number of data groups does not meet the standard and is determined to be partially available, the data groups that do not participate in the local confidence analysis are extracted from the current data set as invalid data groups for elimination, and the data groups that participate in the local confidence analysis are taken as valid data groups.

8. The method for analyzing and evaluating data set quality according to claim 1, wherein: The collection optimization feedback includes the following: When an unavailable conclusion is generated, the acquisition time interval and the associated acquisition device identifier when the data acquisition terminal acquires the current data set; Retrieving the operation record of the device within the above-mentioned collection period from the device operation log database based on the device identification; Through time alignment, we can identify whether there is abnormal operation in the acquisition time interval from the operation records; Generate targeted sensor calibration parameter adjustment instructions based on the identified abnormal features; The generated calibration parameter adjustment instruction is sent to the data acquisition end as a re-collection instruction.

9. The method for analyzing and evaluating data set quality according to claim 1, wherein: The collection optimization feedback also includes the following: Extracting the spatial location coordinates of the failed data group and the missing data group when generating partially available conclusions; A supplementary acquisition area map is generated based on the spatial position coordinates and sent to the data acquisition end.

10. The method for analyzing and evaluating data set quality according to claim 1, wherein: The collection optimization feedback also includes device health association: The cumulative number of re-collection instructions triggered by the same collection device; When the number of re-collection instructions exceeds the preset alarm threshold, an equipment maintenance instruction is generated.

Images