Time series data acquisition monitoring method, device and equipment and storage medium

Abstract

The invention provides a time sequence data collection monitoring method, device and equipment and a storage medium, and the method comprises the steps: obtaining real-time collection time sequence data, dividing the real-time collection time sequence data into data windows of a preset size, obtaining a plurality of to-be-evaluated data windows, and carrying out the evaluation of the to-be-evaluated data windows; performing integrity evaluation, consistency evaluation, timeliness evaluation and validity evaluation on the time sequence data in each to-be-evaluated data window to obtain a window data evaluation result of each to-be-evaluated data window, and for any to-be-evaluated data window, matching a corresponding correction strategy based on the window data evaluation result to obtain a correction result of the to-be-evaluated data window. Correcting the time sequence data in the to-be-evaluated data window according to a corresponding correction strategy; according to the method, multi-dimensional quality evaluation is performed by dividing the data windows, and the correction strategy is dynamically matched, so that the data quality problem can be detected and repaired in real time, and the credibility and the application value of data acquisition are effectively improved by comprehensively evaluating the time sequence data quality in real time and matching the differential correction strategy.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a time series data acquisition and monitoring method, device, equipment and storage medium. Background Art

[0002] With the rapid development of the Internet of Things and big data technologies, the time series data generated by various sensors and devices has shown explosive growth. These time series data have significant temporal continuity characteristics, and their collection quality directly affects the subsequent data analysis and application effects.

[0003] In practical application scenarios, due to factors such as sensor failures, network transmission delays, and equipment performance fluctuations, the time series data collection process often encounters problems such as missing data, duplicate collection, timestamp errors, and numerical anomalies. Existing data processing methods often use offline batch processing, which makes it difficult to promptly detect and correct data quality issues, resulting in reduced data credibility. This is especially true in areas with high real-time requirements, where the timeliness and accuracy of data collection directly impact the security and reliability of system operations. Existing time series data quality assessment methods typically focus only on single-dimensional data features and lack a comprehensive data assessment. Furthermore, general data repair methods result in poor data repair results, limiting the application value of time series data in scenarios such as real-time monitoring and predictive analysis. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a time series data acquisition and monitoring method, device, equipment and storage medium to solve the above technical problems.

[0005] The present application provides a time series data acquisition and monitoring method, which includes: acquiring real-time acquired time series data, and dividing the real-time acquired time series data into data windows of preset sizes to obtain multiple data windows to be evaluated; performing integrity assessment, consistency assessment, timeliness assessment, and effectiveness assessment on the time series data in each data window to be evaluated, respectively, to obtain window data evaluation results for each data window to be evaluated; for any of the data windows to be evaluated, matching a corresponding correction strategy based on the window data evaluation result, and correcting the time series data in the data window to be evaluated according to the corresponding correction strategy.

[0006] In one embodiment of the present application, performing integrity assessment on the time series data in each data window to be evaluated includes: calculating the expected number of data collections in the data window to be evaluated based on the time span of the data window to be evaluated and a preset collection interval; calculating the collection intervals between adjacent data in the data window to be evaluated to obtain multiple collection interval data, and determining the median of the multiple collection interval data as the basic time interval; counting the number of missing points in the data window to be evaluated, and counting the number of special value points containing invalid values ​​in the data window to be evaluated, the number of missing points is determined based on the number of collection interval data in the multiple collection interval data whose ratio to the basic time interval is greater than or equal to a first judgment threshold; calculating the integrity index based on the expected number of data collections, the number of missing points and the number of special value points, and determining the integrity index as the integrity assessment result.

[0007] In one embodiment of the present application, performing consistency evaluation on the time series data in each data window to be evaluated includes: calculating the expected number of data collection in the data window to be evaluated based on the time span of the data window to be evaluated and the preset collection interval, and obtaining the number of actual collected data in the data window to be evaluated; calculating the collection interval between each adjacent data in the data window to be evaluated to obtain multiple collection interval data, and determining the median of the multiple collection interval data as the basic time interval; determining the over-dense point data in the data window to be evaluated based on the number of collection interval data in the multiple collection interval data in the data window to be evaluated whose ratio to the basic time interval is less than or equal to a second judgment threshold; counting the number of over-dense point data in the data window to be evaluated, calculating the consistency index based on the expected number of data collection, the number of over-dense point data and the number of actual collected data, and determining the consistency index as the consistency evaluation result.

[0008] In one embodiment of the present application, performing timeliness evaluation on the time series data in each data window to be evaluated includes: calculating the expected number of data collections in the data window to be evaluated based on the time span of the data window to be evaluated and the preset collection interval, and obtaining the number of actually collected data in the data window to be evaluated; calculating the collection intervals between each adjacent data in the data window to be evaluated to obtain multiple collection interval data, and determining the median of the multiple collection interval data as the basic time interval; determining the data in the data window to be evaluated whose ratio of the collection interval data of the previous adjacent data to the basic time interval is greater than or equal to a first judgment threshold and whose ratio of the collection interval data of the next adjacent data to the basic time interval is less than or equal to a second judgment threshold as delayed data; counting the number of delayed data in the data window to be evaluated, and calculating the timeliness index based on the expected number of collections, the number of delay points and the number of actually collected data, and determining the timeliness index as the timeliness evaluation result.

[0009] In one embodiment of the present application, performing validity evaluation on the time series data in each data window to be evaluated includes: calculating the expected number of data collections in the data window to be evaluated based on a preset time span and a preset collection interval of the data window to be evaluated, and obtaining the actual collected data and the number of actually collected data in the data window to be evaluated; constructing a point value sequence, an adjacent value difference sequence, a speed sequence, and a speed change sequence based on the actually collected data in the data window to be evaluated; determining the number of abnormal point values ​​in the point value sequence that do not meet the value range constraint, the number of abnormal value differences in the adjacent value difference sequence that do not meet the value change constraint, the number of abnormal speed values ​​in the speed sequence that do not meet the speed constraint, and the number of abnormal speed change values ​​in the speed change sequence that do not meet the speed change constraint, respectively, the value range constraint, the value change constraint, the speed constraint, and the speed change constraint are all determined based on the median and the absolute median difference in the corresponding sequence; calculating a validity index based on the number of abnormal point values, the number of abnormal value differences, the number of abnormal speed values, the number of abnormal speed change values, the expected number of collections, and the number of actually collected data, and determining the validity index as the validity evaluation result.

[0010] In one embodiment of the present application, the correction strategy corresponding to the window data evaluation result matching includes: if the values ​​of the integrity evaluation result, consistency evaluation result, timeliness evaluation result and effectiveness evaluation result in the window data evaluation result are all less than the preset correction evaluation threshold, then the correction strategy is no correction, and the evaluation result is output; if at least one of the values ​​of the integrity evaluation result, consistency evaluation result, timeliness evaluation result and effectiveness evaluation result in the window data evaluation result is greater than or equal to the preset correction evaluation threshold, then the evaluation attribute type corresponding to the evaluation result with a value greater than or equal to the preset correction evaluation threshold is determined as the type to be corrected, and the correction steps corresponding to the type to be corrected are matched for correction, and the evaluation attribute types include integrity, consistency, timeliness and effectiveness.

[0011] In one embodiment of the present application, the correction step corresponding to the type to be corrected includes: if the type to be corrected is integrity, the time series data in the data window to be evaluated is corrected by linear interpolation; if the type to be corrected is consistency, the correction is performed by resetting the timestamp of the dense point data to the sum of the timestamp of the previous adjacent data and the basic time interval; if the type to be corrected is timeliness, the collection interval data of the delayed data and the adjacent data is corrected according to the basic time interval; if the type to be corrected is validity, the outliers that do not meet the constraints are corrected to the target value range determined based on the median and the absolute median difference, so as to correct the time series data in the data window to be evaluated.

[0012] An embodiment of the present application also provides a time series data acquisition and monitoring device, which includes: a data preprocessing module, used to obtain real-time acquired time series data, and divide the real-time acquired time series data into data windows of preset sizes to obtain multiple data windows to be evaluated; a data evaluation module, used to perform integrity evaluation, consistency evaluation, timeliness evaluation and effectiveness evaluation on the time series data in each data window to be evaluated, and obtain window data evaluation results of each data window to be evaluated; a data correction module, used to match a corresponding correction strategy for any of the data windows to be evaluated based on the window data evaluation result, and correct the time series data in the data window to be evaluated according to the corresponding correction strategy.

[0013] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the timing data acquisition and monitoring method as described in any one of the above embodiments.

[0014] An embodiment of the present application further provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the time series data acquisition and monitoring method as described in any one of the above embodiments.

[0015] The present application provides a time series data acquisition and monitoring method, apparatus, device and storage medium, which acquires real-time acquired time series data and divides the real-time acquired time series data into data windows of preset sizes to obtain multiple data windows to be evaluated, and performs integrity assessment, consistency assessment, timeliness assessment and effectiveness assessment on the time series data in each data window to be evaluated to obtain window data evaluation results of each data window to be evaluated. For any data window to be evaluated, a corresponding correction strategy is matched based on the window data evaluation result, and the time series data in the data window to be evaluated is corrected according to the corresponding correction strategy; the present application divides the data window for multi-dimensional quality assessment and dynamically matches the correction strategy, which can detect and repair data quality problems in real time, and has the function of effectively improving the credibility and application value of data acquisition by comprehensively evaluating the quality of time series data in real time and matching differentiated correction strategies.

[0016] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, serving to explain the principles of the present application. It is obvious that the drawings described below are merely some embodiments of the present application, and a person of ordinary skill in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0018] Figure 1 is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application;

[0019] Figure 2 This is a flow chart of a time series data acquisition and monitoring method shown in an exemplary embodiment of the present application;

[0020] Figure 3 This is a flowchart of a specific time series data acquisition and monitoring method evaluation step shown in an exemplary embodiment of the present application;

[0021] Figure 4 This is a flowchart of a specific time series data acquisition and monitoring method correction step shown in an exemplary embodiment of the present application;

[0022] Figure 5 This is a schematic diagram of a time series data acquisition and monitoring device shown in an exemplary embodiment of the present application;

[0023] Figure 6 It is a structural diagram of a computer system of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will describe the embodiments of the present application with reference to the accompanying drawings and specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for the purpose of illustrating the present application and are not intended to limit the scope of protection of the present application.

[0025] The illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present application. Therefore, the illustrations only show components related to the present application and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0026] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0027] The term "and / or" used in this application describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0028] Figure 1 It is a schematic diagram of an exemplary system architecture shown in an exemplary embodiment of the present application.

[0029] Reference Figure 1 As shown, the system architecture may include a time series database 110 and a computer device 120. The computer device 120 obtains the real-time collected time series data cached in the time series database 110, and divides the real-time collected time series data into data windows of preset time spans to obtain multiple data windows to be evaluated. The computer device 120 performs integrity assessment, consistency assessment, timeliness assessment, and effectiveness assessment on the time series data in each data window to be evaluated, respectively, to obtain window data evaluation results for each data window to be evaluated. For any data window to be evaluated, a corresponding correction strategy is matched based on the window data evaluation results, and the time series data in the data window to be evaluated is corrected according to the corresponding correction strategy. The above-mentioned computer device 130 refers to a computing power support device for carrying a program implementation environment for executing a time series data acquisition and monitoring method, including but not limited to microcomputers, server clusters, industrial computers, and virtual machines; the above-mentioned time series database 110 is used to cache time series data generated by various sensors and devices during the implementation of projects such as the Internet of Things and big data.

[0030] Schematically, the computer device 120 obtains the real-time collected time series data cached in the time series database 110, and divides the real-time collected time series data into data windows of preset sizes to obtain multiple data windows to be evaluated, and performs integrity assessment, consistency assessment, timeliness assessment and effectiveness assessment on the time series data in each data window to be evaluated to obtain the window data evaluation results of each data window to be evaluated. For any data window to be evaluated, the corresponding correction strategy is matched based on the window data evaluation result, and the time series data in the data window to be evaluated is corrected according to the corresponding correction strategy; the present application divides the data window for multi-dimensional quality assessment and dynamically matches the correction strategy, which can detect and repair data quality problems in real time, and has the function of effectively improving the credibility and application value of data acquisition by comprehensively evaluating the quality of time series data in real time and matching differentiated correction strategies.

[0031] Figure 2 This is a flowchart of a time series data acquisition and monitoring method shown in an exemplary embodiment of the present application. The time series data acquisition and monitoring method can be Figure 1 The implementation environment can be implemented in other implementation environments, and the above implementation environment is not specifically limited here. Figure 2 As shown, the flowchart of the time series data acquisition and monitoring method includes at least steps S210 to S230, which are described in detail as follows:

[0032] In step S210 , real-time collected time series data is acquired, and the real-time collected time series data is divided into data windows of preset time spans to obtain a plurality of data windows to be evaluated.

[0033] In one embodiment of the present application, data windowing refers to segmenting a continuous data stream into segments based on time length or data volume. This can be achieved using a fixed-length sliding window or an adaptive window mechanism to ensure that each window contains sufficient data samples for quality assessment. Specifically, the real-time collected time series data is first allocated to processing windows of predetermined time spans, and each window undergoes independent quality testing.

[0034] In step S220 , integrity assessment, consistency assessment, timeliness assessment, and effectiveness assessment are performed on the time series data in each data window to be assessed, respectively, to obtain a window data assessment result for each data window to be assessed.

[0035] In one embodiment of the present application, performing integrity assessment on the time series data in each data window to be evaluated includes calculating the expected number of data collections in the data window to be evaluated based on the time span of the data window to be evaluated and the preset collection interval, calculating the collection interval between each adjacent data in the data window to be evaluated to obtain multiple collection interval data, and determining the median of the multiple collection interval data as the basic time interval, counting the number of missing points in the data window to be evaluated, and counting the number of special value points containing invalid values ​​in the data window to be evaluated, the number of missing points is determined based on the number of collection interval data in the multiple collection interval data whose ratio to the basic time interval is greater than or equal to a first judgment threshold, calculating the integrity index based on the expected number of data collections, the number of missing points and the number of special value points, and determining the integrity index as the integrity assessment result.

[0036] The time span refers to the length of time covered by the data window, which can be implemented with a fixed duration or dynamically adjusted, and is used to define the time range of data evaluation. The preset collection interval refers to a pre-set data collection period, such as 1 second or 5 minutes, and is used to calculate the expected number of data collections. The expected number of data collections refers to the total amount of data that should theoretically be collected within the data window to be evaluated. It can be calculated by dividing the time span by the preset collection interval and rounding up, and is used as a theoretical benchmark for measuring data integrity. The base time interval refers to the median of adjacent data collection intervals, which can be determined by sorting the collection interval data and taking the middle value. It is used to eliminate the impact of accidental fluctuations on interval calculations. The number of missing points refers to the number of abnormal intervals where the actual collection interval exceeds the base time interval by a certain percentage. It can be calculated by comparing the ratio of the collection interval to the base time interval with a first judgment threshold to identify data missing due to network interruption or equipment failure. The number of special value points refers to the number of invalid data points that contain null values, zero values, or exceed the range in the data window to be evaluated. It can be calculated by traversing the data and comparing it with the preset valid range to reflect abnormal data content.

[0037] Specifically, when calculating the integrity index, the theoretical total amount of data that should be collected is first determined by the time span and the preset collection interval. Then, the adjacent data intervals of the data window to be evaluated are traversed, and accidental fluctuations are filtered out using the median as the benchmark. When an interval exceeds a certain proportion of the benchmark interval, it is considered to have data missing, and the number of missing points is accumulated. At the same time, the number of invalid values ​​in the window is counted as the number of special value points. Finally, based on the theoretical data volume, missing points, and special value points, an assessment result that quantitatively reflects the data integrity is generated. One feasible calculation method for the integrity index can be expressed as the number of missing points + the number of special value points / the expected number of data collections. It can also be expressed as the ratio of the sum of the number of missing points and the number of special value points minus 1 to the expected number of data collections.

[0038] By introducing the basic time interval, it is possible to adaptively identify real missing points and avoid misjudgments caused by short device delays. At the same time, combined with statistics of special value points, a comprehensive assessment of data content quality is achieved, improving the robustness of the assessment.

[0039] In one embodiment of the present application, performing consistency evaluation on the time series data in each data window to be evaluated includes calculating the expected number of data collections in the data window to be evaluated based on the preset time span and preset collection interval of the data window to be evaluated, and obtaining the number of actual collected data in the data window to be evaluated, calculating the collection interval between each adjacent data in the data window to be evaluated to obtain multiple collection interval data, and determining the median of the multiple collection interval data as the basic time interval, determining the over-dense point data in the data window to be evaluated based on the number of collection interval data in the multiple collection interval data in the data window to be evaluated whose ratio to the basic time interval is less than or equal to a second determination threshold, counting the number of over-dense point data in the data window to be evaluated, calculating the consistency index based on the expected number of data collections, the number of over-dense point data and the number of actual collected data, and determining the consistency index as the consistency evaluation result.

[0040] Among them, overly dense point data refers to data points where the interval between adjacent data collection is significantly smaller than the basic time interval. Specifically, the interval ratio can be compared with the second judgment threshold to identify it, and it is used to detect situations where data collection is too dense. In a specific implementation environment, the above-mentioned second judgment threshold can be selected as 0.5.

[0041] Specifically, first, the expected number of data collections that should be collected within the window is calculated based on the preset collection interval, and the number of actually collected data is counted at the same time. By calculating the time intervals of adjacent data, the median is taken as the basic time interval to eliminate the interference of outliers. For each collection interval, the ratio between it and the basic time interval is calculated. When the ratio is lower than the second judgment threshold, it is determined that there is an over-dense collection phenomenon at that location. The comprehensive calculation of the number of over-dense points, the expected number of collections, and the actual number of collections finally generates an evaluation index reflecting the consistency of data collection. In some feasible environments, the calculation process of the above consistency index includes determining it by calculating the ratio of twice the number of over-dense point data to the sum of the number of actual collected data and the number of expected data collections. It can also be expressed by subtracting 1 from the ratio of twice the number of over-dense point data to the sum of the number of actual collected data and the number of expected data collections.

[0042] This application can identify data overcrowding caused by equipment failure or network anomalies in real time, preventing duplicate data from wasting subsequent storage and analysis resources. For example, in industrial sensor scenarios, it can promptly detect redundant data caused by signal interference and trigger an alarm, preventing abnormal data from entering the analysis model and causing biased results. Furthermore, the overcrowding point detection mechanism based on a dynamic benchmark improves adaptability to different acquisition environments and addresses the failure of traditional fixed threshold methods in variable-frequency acquisition scenarios.

[0043] In one embodiment of the present application, performing timeliness evaluation on the time series data in each data window to be evaluated includes calculating the expected number of data collections in the data window to be evaluated based on a preset time span and a preset collection interval of the data window to be evaluated, and obtaining the number of actually collected data in the data window to be evaluated, calculating the collection interval between each adjacent data in the data window to be evaluated to obtain multiple collection interval data, and determining the median of the multiple collection interval data as the basic time interval, determining the data in the data window to be evaluated whose ratio of the collection interval data with the previous adjacent data to the basic time interval is greater than or equal to a first determination threshold and whose ratio of the collection interval data with the next adjacent data to the basic time interval is less than or equal to a second determination threshold as delayed data, counting the number of delayed data in the data window to be evaluated, and calculating the timeliness index based on the expected number of collections, the number of delay points and the number of actually collected data, and determining the timeliness index as the timeliness evaluation result.

[0044] In one embodiment of the present application, delayed data refers to data points that simultaneously meet the conditions of too large a forward interval and too small a backward interval. Specifically, the data can be judged by setting a first judgment threshold (e.g., 2 times) and a second judgment threshold (e.g., 0.5 times) to identify abnormal points where data arrival time is misaligned due to network delays.

[0045] Specifically, first, the expected number of data collections that should be collected within the window is calculated based on the preset collection interval, and the number of actual collected data is counted at the same time. By counting the timestamp sequence of the data points that actually arrive, the time difference set of each two consecutive data is calculated, and the median is taken as the basic time interval of the window. For each data point, if the interval with the previous data exceeds 2 times the basic time interval (for example, it reaches 2 seconds when the basic time interval is 1 second), and the interval with the next data is less than 0.5 times the basic time interval (for example, 0.5 seconds), it is determined to be a timestamp mismatch point caused by transmission delay. Finally, the timeliness score is calculated by the formula of the ratio of twice the number of delay points to the numerical sum of the number of actual collected data and the expected number of data collections. It can also be expressed by 1 minus the ratio of twice the number of delay points to the numerical sum of the number of actual collected data and the expected number of data collections.

[0046] This application can accurately identify data points with time stamp misalignment due to transmission delays, providing a reliable basis for subsequent time stamp correction or data rearrangement. Especially in industrial Internet of Things scenarios, it can effectively solve the timing disorder problem caused by network jitter and ensure the time series accuracy of data points in subsequent data analysis.

[0047] In one embodiment of the present application, performing validity evaluation on the time series data in each data window to be evaluated includes calculating the expected number of data collections in the data window to be evaluated based on a preset time span and a preset collection interval of the data window to be evaluated, and obtaining the actual collected data and the number of actually collected data in the data window to be evaluated, respectively constructing a point value sequence, an adjacent value difference sequence, a speed sequence, and a speed change sequence based on the actually collected data in the data window to be evaluated, respectively determining the number of abnormal point values ​​in the point value sequence that do not meet the value range constraint, the number of abnormal value differences in the adjacent value difference sequence that do not meet the value change constraint, the number of abnormal speed values ​​in the speed sequence that do not meet the speed constraint, and the number of abnormal speed change values ​​in the speed change sequence that do not meet the speed change constraint, respectively, the value range constraint, the value change constraint, the speed constraint, and the speed change constraint are all determined based on the median and the absolute median difference in the corresponding sequence, calculating a validity index based on the number of abnormal point values, the number of abnormal value differences, the number of abnormal speed values, the number of abnormal speed change values, the expected number of collections, and the number of actually collected data, and determining the validity index as the validity evaluation result.

[0048] In one embodiment of the present application, a point value sequence refers to a sequence formed by arranging the values ​​of each acquisition point in the data window to be evaluated in chronological order, which can be achieved by sorting the values ​​corresponding to the acquisition timestamps, and is used to detect abnormal point values ​​that exceed a reasonable range; an adjacent value difference sequence refers to a sequence composed of the value difference between two adjacent acquisition points, which can be achieved by calculating the difference between the values ​​of two consecutive acquisition points, and is used to identify mutation data that does not conform to the expected change trend; a speed sequence refers to a sequence composed of the amplitude of the value change per unit time, which can be achieved by dividing the adjacent value difference by the corresponding time interval, and is used to discover abnormal speeds that do not conform to physical laws or equipment characteristics; a speed change sequence refers to a sequence composed of adjacent changes in speed values, which can be achieved by calculating the difference between adjacent speed values, and is used to capture acceleration abnormalities.

[0049] Among them, the value range constraint, value change constraint, speed constraint and speed change constraint are all dynamically determined based on the median and the absolute median difference. Specifically, the constraint relationship can be expressed by the absolute value of the difference between the median of the data to be judged and the sequence data in the point value sequence, adjacent value difference sequence, speed sequence and speed change sequence being greater than the product of the preset deviation from the median multiple and the absolute median difference of the sequence.

[0050] Specifically, during the effectiveness evaluation process, four feature sequences are first constructed based on the actual collected data. Anomalies are detected from four dimensions: static range of values, dynamic change amplitude, change rate, and rate volatility. The constraints of each sequence are represented by the median and the absolute median difference. For each sequence, data points that exceed the constraints are marked as anomalies. After counting the number of anomalies in each sequence, the expected number of collections and the actual number of collections are combined to obtain a comprehensive effectiveness index through weighted or proportional calculation. This index reflects the credibility of the data in multiple dimensions within the data window. When the index is lower than the threshold, the correction process is triggered. In some specific embodiments, the effectiveness score is calculated by calculating the number of abnormal point values, the number of abnormal value differences, and the number of abnormal speed values, and the ratio of half of the number of abnormal speed change values ​​to the sum of the number of actual collected data and the expected number of data collections. It can also be represented by 1 minus the ratio of half of the number of abnormal point values, the number of abnormal value differences, and the number of abnormal speed values ​​to the sum of the number of actual collected data and the expected number of data collections.

[0051] This application can identify multi-dimensional anomalies in time series data in real time, including complex situations such as sudden changes in values, unreasonable speeds, and abnormal fluctuations. It effectively addresses data distortion caused by sensor drift, signal interference, or transmission errors. By using dynamic constraints and multi-sequence joint evaluation, it reduces the false positive rate while maintaining detection sensitivity, providing a high-quality data foundation for subsequent data analysis and decision-making.

[0052] In step S230 , for any data window to be evaluated, a corresponding correction strategy is matched based on the window data evaluation result, and the time series data in the data window to be evaluated is corrected according to the corresponding correction strategy.

[0053] In one embodiment of the present application, if the values ​​of the integrity assessment result, consistency assessment result, timeliness assessment result and effectiveness assessment result in the window data assessment result are all less than the preset correction assessment threshold, the correction strategy is no correction, and the assessment result is output.

[0054] In one embodiment of the present application, if at least one of the values ​​of the integrity assessment result, consistency assessment result, timeliness assessment result and effectiveness assessment result in the window data assessment result is greater than or equal to a preset correction assessment threshold, the assessment attribute type corresponding to the assessment result whose value is greater than or equal to the preset correction assessment threshold is determined as the type to be corrected, and the correction steps corresponding to the type to be corrected are matched for correction. The assessment attribute types include integrity, consistency, timeliness and effectiveness.

[0055] Among them, the preset correction assessment threshold refers to a pre-set critical value used to determine whether data correction is needed, which can be implemented by numerical comparison; the type to be corrected refers to the classification of data quality problems corresponding to the assessment results that exceed the threshold, which can be implemented by type marking, for example, integrity anomalies correspond to missing data repair, consistency anomalies correspond to over-dense data processing, etc.; correction steps refer to specific data adjustment methods taken for different types of quality problems, which can be implemented by strategy mapping tables, such as establishing a correspondence between types and interpolation algorithms and timestamp adjustment rules.

[0056] Specifically, when the integrity, consistency, timeliness and effectiveness evaluation results of the data window do not exceed the preset thresholds, it indicates that the data quality meets the requirements. At this time, there is no need to perform correction operations, and only the evaluation results need to be output for use in subsequent processes. When any evaluation result exceeds the threshold, the system automatically identifies the corresponding quality defect type, such as insufficient integrity or delayed timeliness, and selects the corresponding processing steps based on the preset correction strategy library. For example, integrity anomalies trigger the missing value interpolation process, and consistency anomalies trigger the timestamp calibration process. Through the classification correction mechanism, different types of data quality problems can be solved in a targeted manner, avoiding the accumulation of errors caused by a single correction method.

[0057] In one embodiment of the present application, if the correction type is integrity, the time series data in the data window to be evaluated is corrected using linear interpolation. Linear interpolation refers to filling in missing data using the relationship between the values ​​and timestamps of known adjacent data points. Specifically, this can be achieved by calculating the intermediate value of the values ​​of adjacent time points based on the time ratio to restore the continuity of the missing data.

[0058] In one embodiment of the present application, if the correction type is consistency, the correction is performed by resetting the timestamp of the overcrowded data point to the sum of the timestamp of the previous adjacent data point and the base time interval. Timestamp resetting refers to adjusting the collection time of the overcrowded data point. Specifically, this can be achieved by updating the timestamp of the overcrowded point to the sum of the timestamp of the previous data point and the base time interval. This is used to eliminate timestamp accumulation caused by abnormal collection frequency.

[0059] In one embodiment of the present application, if the type to be corrected is timeliness, the collection interval data of the delayed data and the adjacent data is corrected according to the basic time interval. Specifically, it refers to adjusting the interval distribution of the delayed data according to the normal collection interval in the data window. This can be achieved by reallocating the timestamps before and after the delayed data according to the basic time interval, which is used to repair the time series break caused by the delay.

[0060] In one embodiment of the present application, if the type to be corrected is validity, the time series data in the data window to be evaluated is corrected by correcting the outliers that do not meet the constraints to within the target value range determined based on the median and the absolute median difference. The target value range refers to the reasonable interval of data calculated based on the median and the absolute median difference. Specifically, it can be achieved by multiplying the absolute value of the difference between the data to be judged and the median of the sequence data in the point value sequence, adjacent value difference sequence, speed sequence, and speed change sequence by 0.5 multiplied by the product of a preset deviation from the median multiple and the absolute median difference of the sequence, so as to adjust the outliers that exceed the reasonable range to the valid interval.

[0061] This application can implement precise corrections for different data quality defects. For example, in sensor network data collection, resetting the timestamp of overly dense data can avoid time series overlap, linear interpolation can effectively fill the occasional omissions caused by network transmission, and outlier correction can eliminate numerical jumps caused by transient sensor failures, thereby improving the availability of time series data in real-time monitoring scenarios and providing high-quality input for subsequent data analysis.

[0062] Figure 3 This is a flowchart of a specific time series data acquisition and monitoring method evaluation link shown in an exemplary embodiment of the present application, with reference to Figure 3 As shown, the integrity, consistency, timeliness and effectiveness of the real-time collected time series data are evaluated, and after the real-time collected time series data is divided into multiple data windows to be evaluated according to the set window size, for the evaluation of integrity, consistency and timeliness, the time interval characteristics are first calculated, that is, the basic time interval in the aforementioned embodiment. The integrity index is calculated by counting the number of missing points and the number of special value points, and combining them with the expected number of data collection, that is, the expected number of data collection in the aforementioned embodiment. The consistency index is obtained by calculating the number of over-dense points and combining them with the expected number of data collection. The consistency index is obtained by calculating the number of delayed points and combining them with the expected number of data collection. For the effectiveness evaluation, the median deviation multiple k value is first preset, and the number of data points that violate the value range constraint, the number of data points that violate the value change constraint, the number of data points that violate the speed constraint, and the number of data points that violate the speed change constraint are calculated, and the effectiveness index is calculated in combination with the expected number of data collection.

[0063] Figure 4 This is a flowchart of a specific time series data acquisition and monitoring method correction link shown in an exemplary embodiment of the present application, with reference to Figure 4As shown, after obtaining the window data evaluation results of each data window to be evaluated, the evaluation results of integrity, consistency, timeliness and effectiveness are matched according to the window data evaluation results to see whether they are less than the preset correction evaluation threshold. If the values ​​of the integrity evaluation results, consistency evaluation results, timeliness evaluation results and effectiveness evaluation results in the window data evaluation results are all less than the preset correction evaluation threshold, no correction is made, and the evaluation results are output. If at least one of the values ​​of the integrity evaluation results, consistency evaluation results, timeliness evaluation results and effectiveness evaluation results in the window data evaluation results is greater than or equal to the preset correction evaluation threshold, the evaluation attribute type corresponding to the evaluation result whose value is greater than or equal to the preset correction evaluation threshold is determined as the type to be corrected, and the correction step corresponding to the type to be corrected is matched for correction.

[0064] The present application provides a time series data acquisition and monitoring method, apparatus, device and storage medium, which acquires real-time acquired time series data and divides the real-time acquired time series data into data windows of preset sizes to obtain multiple data windows to be evaluated, and performs integrity assessment, consistency assessment, timeliness assessment and effectiveness assessment on the time series data in each data window to be evaluated to obtain window data evaluation results of each data window to be evaluated. For any data window to be evaluated, a corresponding correction strategy is matched based on the window data evaluation result, and the time series data in the data window to be evaluated is corrected according to the corresponding correction strategy; the present application divides the data window for multi-dimensional quality assessment and dynamically matches the correction strategy, which can detect and repair data quality problems in real time, and has the function of effectively improving the credibility and application value of data acquisition by comprehensively evaluating the quality of time series data in real time and matching differentiated correction strategies.

[0065] The following describes an embodiment of the device of the present application, which can be used to execute the time series data acquisition and monitoring method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the time series data acquisition and monitoring method in the above embodiment of the present application.

[0066] Figure 5 This is a schematic diagram of a time series data acquisition and monitoring device shown in an exemplary embodiment of the present application. The device can be applied to Figure 2 The method implementation process shown in the figure can be based on Figure 1 The present invention is executed in the implementation environment shown in , and may also be applicable to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the apparatus is applicable.

[0067] like Figure 5 As shown, the exemplary time series data acquisition and monitoring device includes: a data preprocessing module 501, a data evaluation module 502 and a data correction module 503.

[0068] Among them, the data preprocessing module 501 is used to obtain real-time collected time series data and divide the real-time collected time series data into data windows of preset sizes to obtain multiple data windows to be evaluated; the data evaluation module 502 is used to perform integrity evaluation, consistency evaluation, timeliness evaluation and effectiveness evaluation on the time series data in each data window to be evaluated, and obtain the window data evaluation results of each data window to be evaluated; the data correction module 503 is used to match the corresponding correction strategy for any data window to be evaluated based on the window data evaluation result, and correct the time series data in the data window to be evaluated according to the corresponding correction strategy.

[0069] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the electronic device implements the timing data acquisition and monitoring method provided in the above-mentioned embodiments.

[0070] Figure 6 This is a schematic diagram of the structure of a computer system of an electronic device according to an exemplary embodiment of the present application. Figure 6 The computer system 600 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0071] like Figure 6 As shown, computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in read-only memory (ROM) 602 or the program loaded from the storage portion into random access memory (RAM) 603, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in RAM 603. CPU 601, ROM 602 and RAM 603 are connected to each other via a bus. I / O interface 605 is also connected to bus 604, wherein I / O interface 605 refers to an input / output (Input / Output) interface.

[0072] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 610 as needed so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0073] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from a removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the various functions defined in the system of the present application are executed.

[0074] The computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0075] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0076] In the corresponding drawings of the above embodiments, connecting lines can represent the connection relationship between various components to represent more constituent signal paths (constituent_signalpath) and / or one or more ends of some lines have arrows to indicate the main information flow direction. The connecting lines serve as an identifier and are not a limitation to the scheme itself. Instead, the use of these lines in combination with one or more exemplary embodiments helps to connect circuits or logic units more easily. Any represented signal (determined by design requirements or preferences) may actually include one or more signals that can be transmitted in any direction and can be implemented with any appropriate type of signal scheme.

[0077] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0078] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.

[0079] An embodiment of the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the time series data acquisition and monitoring method as described in any one of the above embodiments.

[0080] Although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0081] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0082] The present application can be used in a wide variety of general-purpose or specialized computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above.

[0083] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0084] The above content of this application is only a preferred exemplary embodiment of this application and is not intended to limit the implementation scheme of this application. Ordinary technicians in this field can easily make corresponding changes or modifications based on the main ideas and spirit of this application. Therefore, the scope of protection of this application should be the scope of protection required by the claims.

Claims

1. A time series data acquisition and monitoring method, characterized in that: The time series data acquisition and monitoring method includes: Acquire real-time collected time series data, and divide the real-time collected time series data into data windows of preset time spans to obtain multiple data windows to be evaluated; Perform integrity assessment, consistency assessment, timeliness assessment, and effectiveness assessment on the time series data in each data window to be evaluated, and obtain the window data assessment results of each data window to be evaluated; For any of the data windows to be evaluated, a corresponding correction strategy is matched based on the window data evaluation result, and the time series data in the data window to be evaluated is corrected according to the corresponding correction strategy.

2. The time series data acquisition and monitoring method according to claim 1, characterized in that: Performing integrity assessment on the time series data in each data window to be evaluated includes: Calculating the expected number of data collections for the data window to be evaluated based on the time span of the data window to be evaluated and a preset collection interval; Calculating the collection intervals between adjacent data in the data window to be evaluated to obtain a plurality of collection interval data, and determining the median of the plurality of collection interval data as the basic time interval; Counting the number of missing points in the data window to be evaluated, and counting the number of special value points containing invalid values ​​in the data window to be evaluated, wherein the number of missing points is determined based on the number of acquisition interval data whose ratio to the basic time interval is greater than or equal to a first determination threshold among the multiple acquisition interval data; An integrity index is calculated according to the expected number of data collections, the number of missing points and the number of special value points, and the integrity index is determined as an integrity assessment result.

3. The time series data acquisition and monitoring method according to claim 1, characterized in that: Performing consistency evaluation on the time series data in each data window to be evaluated includes: Calculating the expected number of data collections for the data window to be evaluated based on the preset time span and the preset collection interval of the data window to be evaluated, and obtaining the number of data actually collected in the data window to be evaluated; Calculating the collection intervals between adjacent data in the data window to be evaluated to obtain a plurality of collection interval data, and determining the median of the plurality of collection interval data as the basic time interval; Determine the overcrowded point data in the data window to be evaluated according to the number of the collection interval data whose ratio to the basic time interval is less than or equal to the second determination threshold among the multiple collection interval data in the data window to be evaluated; The number of overcrowded point data in the data window to be evaluated is counted, a consistency index is calculated according to the expected number of data collection, the number of overcrowded point data and the number of actually collected data, and the consistency index is determined as a consistency evaluation result.

4. The time series data acquisition and monitoring method according to claim 1, characterized in that: The timeliness evaluation of the time series data in each data window to be evaluated includes: Calculating the expected number of data collections for the data window to be evaluated based on the preset time span and the preset collection interval of the data window to be evaluated, and obtaining the number of data actually collected in the data window to be evaluated; Calculating the collection intervals between adjacent data in the data window to be evaluated to obtain a plurality of collection interval data, and determining the median of the plurality of collection interval data as the basic time interval; Determine, in the data window to be evaluated, data whose ratio of the collection interval data of the previous adjacent data to the basic time interval is greater than or equal to the first determination threshold, and whose ratio of the collection interval data of the next adjacent data to the basic time interval is less than or equal to the second determination threshold as delayed data; The amount of delayed data in the data window to be evaluated is counted, and a timeliness index is calculated based on the expected number of collections, the number of delay points, and the number of actually collected data, and the timeliness index is determined as a timeliness evaluation result.

5. The time series data acquisition and monitoring method according to claim 1, characterized in that: Performing validity evaluation on the time series data in each data window to be evaluated includes: Calculating the expected number of data collections for the data window to be evaluated according to the preset time span and the preset collection interval of the data window to be evaluated, and obtaining the actual collected data and the number of the actual collected data in the data window to be evaluated; According to the actual collected data in the data window to be evaluated, a point value sequence, an adjacent value difference sequence, a speed sequence and a speed change sequence are respectively constructed; Determine respectively the number of abnormal point values ​​in the point value sequence that do not satisfy the value range constraint, the number of abnormal value differences in the adjacent value difference sequence that do not satisfy the value change constraint, the number of abnormal speed values ​​in the speed sequence that do not satisfy the speed constraint, and the number of abnormal speed change values ​​in the speed change sequence that do not satisfy the speed change constraint, wherein the value range constraint, value change constraint, speed constraint, and speed change constraint are all determined based on the median and absolute median difference in the corresponding sequences; The effectiveness index is calculated based on the number of abnormal point values, the number of abnormal numerical differences, the number of abnormal speed values, the number of abnormal speed change values, the expected number of collections and the number of actual collected data, and the effectiveness index is determined as the effectiveness evaluation result.

6. The time series data acquisition and monitoring method according to claim 1, characterized in that: The corresponding correction strategies based on the window data evaluation results include: If the values ​​of the integrity assessment result, consistency assessment result, timeliness assessment result and effectiveness assessment result in the window data assessment result are all less than the preset correction assessment threshold, the correction strategy is no correction, and the assessment result is output; If at least one of the values ​​of the integrity assessment result, consistency assessment result, timeliness assessment result and effectiveness assessment result in the window data assessment result is greater than or equal to the preset correction assessment threshold, the assessment attribute type corresponding to the assessment result whose value is greater than or equal to the preset correction assessment threshold is determined as the type to be corrected, and the correction steps corresponding to the type to be corrected are matched for correction. The assessment attribute types include integrity, consistency, timeliness and effectiveness.

7. The time series data acquisition and monitoring method according to any one of claims 1 to 6, characterized in that: Matching the correction steps corresponding to the type to be corrected includes: If the type to be corrected is integrity, the time series data in the data window to be evaluated is corrected by linear interpolation; If the type to be corrected is consistency, the correction is performed by resetting the timestamp of the over-dense point data to the sum of the timestamp of the previous adjacent data and the basic time interval; If the type to be corrected is time-sensitive, the collection interval data of the delayed data and adjacent data is corrected according to the basic time interval; If the type to be corrected is validity, the time series data in the data window to be evaluated is corrected by correcting the outliers that do not meet the constraints to be within the target value range determined based on the median and the absolute median difference.

8. A time series data acquisition and monitoring device, characterized in that: The time series data acquisition and monitoring device comprises: A data preprocessing module is used to obtain real-time collected time series data and divide the real-time collected time series data into data windows of preset sizes to obtain multiple data windows to be evaluated; The data evaluation module is used to perform integrity evaluation, consistency evaluation, timeliness evaluation and effectiveness evaluation on the time series data in each data window to be evaluated, and obtain the window data evaluation results of each data window to be evaluated; The data correction module is used to match a corresponding correction strategy to any of the data windows to be evaluated based on the window data evaluation result, and correct the time series data in the data window to be evaluated according to the corresponding correction strategy.

9. An electronic device, characterized in that: It includes a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute the computer program stored in the memory to implement the time series data acquisition and monitoring method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is used to enable a computer to execute the time series data acquisition and monitoring method according to any one of claims 1 to 7.

Images