Automatic inspection method and system for monitoring time sequence data

Abstract

The invention discloses an automatic inspection method and system for monitoring time sequence data, and belongs to the technical field of intelligent monitoring, and the method comprises the steps: employing a sliding time window technology, and extracting continuous data fragments from a preprocessed time sequence data set according to a preset step length, and taking the continuous data fragments as to-be-detected data; calling a predefined anomaly detection algorithm from a method library, and performing multi-dimensional analysis on the extracted data fragments to identify an anomaly condition in the data; after abnormal data is detected, the system automatically records the type, occurrence time and characteristics of the abnormal data, and then continues to detect the next data fragment; and for data without abnormity, the system directly continues to detect the next data fragment until all data are detected. According to the method, the accuracy and the reliability of data processing are remarkably improved, and the requirement of manual intervention is reduced; and the time consumption is less, and the requirements of large-scale real-time data processing can be met, so that the high quality of monitoring data and the high accuracy of monitoring work are ensured.

Description

Technical Field

[0001] The present invention mainly relates to the field of intelligent monitoring technology, and specifically to an automatic inspection method and system for monitoring time series data. Background Art

[0002] Intelligent monitoring is inseparable from detection equipment. In actual applications, the data collected by these devices are not always accurate and reliable. Abnormal data may appear due to equipment failure, external interference or other special reasons. These abnormal data usually manifest as missing data, data duplication, abnormal data deviation, or no change for a long time. The existence of abnormal data will not only affect the accuracy of the monitoring system, but also may lead to erroneous analysis results, thus having a negative impact on the decision-making process. Therefore, timely and accurate detection and processing of abnormal data is crucial to ensure the effectiveness and reliability of the monitoring system.

[0003] Currently, most monitoring systems rely on simple threshold judgments or manual intervention to identify abnormal data. This approach is time-consuming and error-prone, and it is difficult to meet the needs of large-scale real-time data processing. Especially in complex environments such as smart construction sites, there are many types of monitoring data and huge amounts of data, and traditional anomaly detection methods are often difficult to cope with. In addition, how to effectively classify and store the detected abnormal data and generate visual analysis reports is also a major challenge facing the current monitoring system. Summary of the invention

[0004] The technical solution of the present invention aims at the technical problem that the existing technical solutions are too single, and provides a solution that is significantly different from the existing technologies. It mainly provides an automatic inspection method and system for monitoring time series data, which is used to solve the technical problem that the traditional anomaly detection method proposed in the above background technology is time-consuming and error-prone, and is difficult to meet the needs of large-scale real-time data processing.

[0005] The technical solution adopted by the present invention to solve the above technical problems is:

[0006] An automatic inspection method for monitoring time series data includes the following steps:

[0007] (1) Using the sliding time window technology, continuous data segments are extracted from the preprocessed time series data set according to the preset step size as the data to be detected;

[0008] (2) Calling a predefined anomaly detection algorithm from the method library to perform multi-dimensional analysis on the extracted data fragments to identify anomalies in the data;

[0009] (3) After detecting abnormal data, the system automatically records the type, occurrence time and characteristics of the abnormal data, and then continues to detect the next data segment; for data without abnormalities, the system directly continues to detect the next data segment until all data are detected.

[0010] Furthermore, the method also includes step (4) that after all data are detected, the system automatically generates an inspection report based on the results of the abnormality detection.

[0011] Furthermore, the steps of obtaining the preprocessed time series data set are:

[0012] A. The system obtains a time series data set within a set time range from the monitoring device and preprocesses it. The preprocessing includes: data cleaning, deduplication, and formatting operations.

[0013] Specifically, the specific steps of preprocessing include:

[0014] A1. Data cleaning: including checking the integrity of each record and removing empty values ​​or data with incorrect formats;

[0015] A2. Redundant data processing: By removing duplicate data points, ensure that each device has only one valid record at each time point;

[0016] A3. Data formatting: Convert data from different sources into a unified format.

[0017] Specifically, in step (1), the width of the time window is dynamically set according to the characteristics of the time series data and the monitoring requirements. The sliding time window obtains the corresponding data according to the preset step size. The step size is fixed or variable to meet the needs of different monitoring scenarios.

[0018] Specifically, in step (2), the anomaly detection includes:

[0019] Offline anomaly detection: For each data segment, the number of data is counted. When the number of data is 0, it is determined that the device is offline abnormally in the corresponding time period;

[0020] Low value anomaly detection: When all data values ​​in a data segment are lower than the preset threshold a, it is judged as a low value anomaly;

[0021] Constant value anomaly detection: Calculate the variance of data in a data segment. When the variance is less than or equal to the preset threshold b, it is determined to be a constant value anomaly.

[0022] Evaluate data integrity rate: Ensure that there is only one data record per minute. The calculation formula is: data integrity rate = number of all valid data / total number of minutes in the detection period;

[0023] Determine data accuracy: Calculate the ratio of all data to the number of abnormal data. The calculation formula is: Data accuracy = [number of all valid data - (number of constant value abnormal data + number of low value abnormal data)] / number of all valid data.

[0024] Specifically, step (3) includes:

[0025] C1. Recording of abnormal data: When an abnormality is detected in a data segment, the system first records the specific characteristics of the abnormal data, including the type of abnormality, the time period in which it occurred, and basic information about the device;

[0026] C2. Data storage and database update: When the system stores abnormal information in the database, it first deletes the relevant records that already exist in the database, and then inserts the new abnormal detection results.

[0027] Specifically, step (4) includes:

[0028] D1, summarize the anomaly detection results, including all marked anomaly data and their characteristics;

[0029] D2. Classify the abnormal data into predefined categories and calculate the spatial and temporal distribution characteristics of the abnormal data, including the frequency, time and classification information of the abnormal occurrence;

[0030] D3. Generate a structured inspection report, which includes statistical tables, charts and related analysis of abnormal data;

[0031] D4. Store the generated inspection report in a document format and export the report to a local file to facilitate distribution and archiving of the report.

[0032] More specifically, step D2 includes:

[0033] D2.1. Identification and classification of abnormal devices: Filter the status field of the device. The system identifies all devices other than those with a status of "normal" as abnormal objects, and selects and stores the devices with abnormal status fields.

[0034] D2.2. Statistics of abnormality rate and problem rate: The system performs statistical analysis on abnormal devices according to different dimensions. The statistical process includes calculating the total number of devices, number of abnormalities, abnormality rate and problem rate under each dimension.

[0035] D2.3. Generation of abnormality rate chart: The system visualizes the statistical results of abnormality rate and problem rate into charts to intuitively show the distribution of abnormal situations;

[0036] D2.4. Statistics of abnormal types: The system classifies and counts different types of abnormalities in detail, calculates the number, percentage and problem rate of each type of abnormality and records them;

[0037] D2.5. Saving results: The system saves all statistical results to the specified file so that detailed exception reports can be generated later.

[0038] The present invention also provides an automatic inspection system for monitoring time series data, comprising a processor and a memory, wherein a computer program is stored in the memory, and when the computer program runs in the processor, the automatic inspection method for monitoring time series data is executed.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) The present invention uses sliding time window technology to extract continuous data segments as the data to be detected from the preprocessed time series data set according to the preset step size, and then uses a multi-dimensional anomaly detection algorithm to perform multi-dimensional analysis on the extracted data segments (including offline anomaly detection, low-value anomaly detection, constant-value anomaly detection, etc.), which can automatically identify abnormal situations in the data to achieve automatic inspection of time series data. The present invention integrates sliding time window technology and multi-dimensional anomaly detection algorithm, significantly improves the accuracy and reliability of data processing, reduces the need for manual intervention; and is less time-consuming, which can meet the needs of large-scale real-time data processing.

[0041] (2) In the optimization scheme of the present invention, the system can effectively classify and store the detected abnormal data according to the spatial and temporal distribution characteristics based on the results of abnormality detection, and then automatically generate a visual analysis report (i.e., inspection report), providing managers with more comprehensive decision-making support and effectively improving the efficiency and intelligence level of monitoring work.

[0042] (3) In the optimization scheme of the present invention, the acquired original data set is preprocessed by data cleaning, deduplication and formatting operations to ensure the consistency and integrity of the monitoring data, thereby providing a more accurate and reliable technical means for equipment data monitoring.

[0043] In summary, the present invention significantly improves the accuracy and reliability of data processing and reduces the need for manual intervention by integrating sliding time window technology and multi-dimensional anomaly detection algorithms; at the same time, the inspection reports automatically generated by the system provide managers with more comprehensive decision-making support, effectively improving the efficiency and intelligence level of monitoring work. In addition, the present invention also has data cleaning, deduplication and formatting operation functions, ensuring the consistency and integrity of monitoring data, thereby providing a more accurate and reliable technical means for equipment data monitoring.

[0044] The present invention will be explained in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a step diagram of the automatic inspection method in the present invention;

[0046] Figure 2 The figure is a specific flow chart of the automatic inspection method in the present invention. DETAILED DESCRIPTION

[0047] To facilitate understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings, but the present invention can be implemented in different forms and is not limited to the embodiments described in the text. On the contrary, these embodiments are provided to make the content disclosed in the present invention more thorough and comprehensive.

[0048] Please refer to the attached Figure 1 , an automatic inspection method for monitoring time series data, comprising the following steps:

[0049] S1: The system obtains the time series data set within the set time range from the monitoring equipment and preprocesses it. The preprocessing steps include: data cleaning, deduplication, and formatting operations to ensure the consistency and accuracy of the data.

[0050] Please refer to the attached Figure 2 , the step specifically includes:

[0051] S1-1. Database connection

[0052] The system first connects to the time series database through the database access module. Through the database connection, the system can efficiently obtain the time series data within the specified time range.

[0053] S1-2. Data Query

[0054] After connecting to the database, the system queries relevant data according to the set time range. The main contents of the query include project number, equipment number, equipment brand, equipment model, and monitoring time series data (such as water level and rainfall, etc.). SQL query statements are used to extract these data from the corresponding database tables.

[0055] sql = "select * from" + table_name + "where xmbh = '" + project_data [1] + "' and device_no = '" + sbbh [0] + "' and (col_time between '" + argument [' start time '] + "' and '" + argument [' end time '] + "')"

[0056] S1-3. Data cleaning

[0057] After data acquisition is completed, the system cleans the raw data. The purpose of cleaning is to remove invalid records and duplicate data to ensure the integrity and accuracy of the data. The cleaning process includes checking the integrity of each record and removing null values ​​or data with incorrect formats. The data cleaning part is mainly completed by calling custom functions.

[0058] S1-4. Redundant data processing

[0059] The system deduplicates the acquired data to ensure that each device has only one valid record at each time point. By eliminating duplicate monitoring data, the system can avoid the impact of data redundancy on subsequent analysis.

[0060] S1-5. Data formatting

[0061] In order to ensure the smooth progress of subsequent data analysis, the system converts data from different sources into a unified format. This includes unifying all timestamps into a standard format and converting numerical data into a unified unit. The data formatting process ensures the comparability of the data in the analysis.

[0062] bgtime_object = time.strptime(argument['start time'],'%Y-%m-%d%H:%M:%S')

[0063] entime_object = time.strptime(argument['end time'],'%Y-%m-%d%H:%M:%S')

[0064] S1-6. Result storage

[0065] After preprocessing is completed, the processed data is stored in a predefined structure to provide data support for subsequent anomaly detection and report generation. The system will store the processed data in the specified file path and record the status information related to the monitoring equipment.

[0066] S2: Use the sliding time window technology to extract multiple continuous data segments from the time series data set according to the preset step size as the data to be detected. The width and step size of the time window can be dynamically adjusted according to the characteristics of the data and monitoring requirements.

[0067] Please refer to the attached Figure 2 During the processing, the system uses the sliding time window technology to extract all data fragments within a time window at regular intervals for subsequent analysis. The width and step size of the sliding time window can be flexibly adjusted to meet different detection requirements. The width of the time window is dynamically set according to the characteristics of the time series data and the monitoring requirements, and the step size can be adjusted during the setting.

[0068] In the following code, the step size of the time window is set to one hour (3600 seconds), and the system processes the data in each time window one by one.

[0069]

[0070] In each time window, the system extracts all data within that time period.

[0071] S3: Please refer to the attached Figure 2 , the system calls the pre-defined anomaly detection algorithm from the method library and performs multi-dimensional analysis on the extracted data fragments to identify anomalies in the data. Specifically, it includes:

[0072] (1) Offline anomaly detection

[0073] For a certain device, the system uses 60 minutes as a time window to detect whether there is data record in the time window. If there is no data in the window, the 60 minutes are considered offline abnormal. The system accumulates the offline time in all detection time periods and uses it to calculate the online rate of the device.

[0074] (2) Low value anomaly detection

[0075] For a certain device, the system uses 60 minutes as a time window and detects the data within the window. If all data values ​​in the window are lower than the preset threshold (such as 10), it is judged as a low value anomaly. The system records the detected low value anomaly data and writes the relevant information into the device's notes.

[0076] (3) Constant value anomaly detection

[0077] The system also uses 60 minutes as a time window and calculates the variance of the data within the time window. If the variance is less than or equal to the preset threshold, it is judged as a constant value abnormality. This usually indicates that the device failed to record the changing data normally during the time period (such as equipment failure or data collection abnormality). The system will record the data with constant value abnormality and write the relevant information into the device's notes information.

[0078] In the following code, the system determines whether there is a constant value anomaly by calculating the variance of the pm10 data. If the variance is less than or equal to the preset threshold, it is marked as a constant value anomaly.

[0079] if len(pm_10)>=3:#The detection standard of constant value anomaly requires more than 3 data

[0080] window_var = np.var(pm_10)

[0081] if window_var == argument['variance threshold']:

[0082] hengzhi_time_stamp=True

[0083] result_hzyc = 'Constant value abnormal device number: {}, variance: {}, window value: {}'.format(sbbh,float(window_var),pm_10).

[0084] (4) Calculation of data integrity rate

[0085] The data integrity rate indicates the proportion of data recorded by the device every minute during the entire detection period. For a certain device, under normal circumstances, there is and can only be 1 data per minute during the detection period. If there is more than 1 data in 1 minute, only 1 data is counted, and if there is no data in 1 minute, it is recorded as 0. The number of all valid data / the total number of minutes in the detection period = the data integrity rate of the device.

[0086] (5) Data accuracy calculation

[0087] Data accuracy refers to the proportion of all valid data that does not contain abnormal data (constant value abnormalities and low value abnormalities). The system calculates data accuracy using the following formula: For a certain device, [total number of valid data - (number of constant value abnormal data + number of low value abnormal data)] / total number of valid data = data accuracy.

[0088] S4: Please refer to the attached Figure 2 After detecting abnormal data, the system automatically records the type, occurrence time and characteristics of the abnormal data, stores it as evidence, and then continues to detect the next data segment; for data without abnormalities, the system directly continues to detect the next data segment until all data are detected.

[0089] S4-1. Recording of abnormal data

[0090] When the system detects an anomaly (such as a low value anomaly, a constant value anomaly, or an offline anomaly) in a data segment of a certain device, the system will first record the specific characteristics of the abnormal data. This includes the type of anomaly (such as low value, constant value, offline), the time period in which it occurred, and relevant information about the device (such as the device number, etc.). This information will be stored in the device's notes for subsequent query and analysis.

[0091] S4-2. Data storage and database update

[0092] Once an anomaly is detected, the system will store the anomaly information in the database. To ensure data consistency and avoid redundancy, the system will first delete the existing related records in the database and then insert the new anomaly detection results.

[0093] S5: Please refer to the attached Figure 2 After all data are detected, the system automatically generates an inspection report based on the results of abnormal detection. The report includes the classification, distribution and impact analysis of abnormal data for users to review and make decisions.

[0094] Specifically include:

[0095] S5-5.1 Initialization and formatting of reports

[0096] The system will first initialize the report document and set the document format, including title, paragraph style, font and size, etc. The report title and generation date will be automatically generated according to the current time.

[0097] S5-5.2 Report writing

[0098] The system automatically generates the report content based on the previous abnormal data analysis results. The report includes multiple parts, such as statistical analysis by device and a list of abnormal devices. Each part contains detailed statistical tables and charts to help users understand the distribution and impact of abnormal data.

[0099] S5-5.3 Generation and insertion of statistical charts

[0100] In order to help users understand the data more intuitively, the system will generate various statistical charts and insert them into the report. The chart generation process is based on the previous abnormal data analysis and is implemented through the functions in the CounterPack module.

[0101] S5-5.4 Generation of detailed statistical tables

[0102] The system inserts the previously generated statistical tables into the corresponding positions of the report. These tables include statistical analysis by equipment type, region, construction unit, equipment brand and other dimensions, helping users to fully understand the distribution and impact of abnormal data.

[0103] S5-5.5 Saving and exporting reports

[0104] After completing the report writing and inserting charts, the system will save the generated report document to the specified file path for user reference and archiving.

[0105] An automatic inspection system for monitoring time series data includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is run in the processor, the automatic inspection method for monitoring time series data is executed. The storage of abnormal data is managed by a database management system, and supports multi-user query and analysis.

[0106] The above is an exemplary description of the present invention in combination with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as such non-substantial improvements are made by adopting the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. An automatic inspection method for monitoring time series data, characterized in that: The steps include: (1) Using the sliding time window technology, continuous data segments are extracted from the preprocessed time series data set according to the preset step size as the data to be detected; (2) Calling a predefined anomaly detection algorithm from the method library to perform multi-dimensional analysis on the extracted data fragments to identify anomalies in the data; (3) After detecting abnormal data, the system automatically records the type, occurrence time and characteristics of the abnormal data, and then continues to detect the next data segment; for data without abnormalities, the system directly continues to detect the next data segment until all data are detected.

2. The automatic inspection method for monitoring time series data according to claim 1, characterized in that: The method also includes step (4) that after all data are detected, the system automatically generates an inspection report based on the results of the abnormality detection.

3. The automatic inspection method for monitoring time series data according to claim 1, characterized in that: The steps for obtaining the preprocessed time series data set are: A. The system obtains a time series data set within a set time range from the monitoring device and preprocesses it. The preprocessing includes: data cleaning, deduplication, and formatting operations.

4. The automatic inspection method for monitoring time series data according to claim 3 is characterized in that: The specific steps of preprocessing include: A1. Data cleaning: including checking the integrity of each record and removing empty values ​​or data with incorrect formats; A2. Redundant data processing: By removing duplicate data points, ensure that each device has only one valid record at each time point; A3. Data formatting: Convert data from different sources into a unified format.

5. The automatic inspection method for monitoring time series data according to claim 1, characterized in that: In step (1), the width of the time window is dynamically set according to the characteristics of the time series data and the monitoring requirements. The sliding time window obtains the corresponding data according to the preset step size. The step size is fixed or variable to meet the needs of different monitoring scenarios.

6. The automatic inspection method for monitoring time series data according to claim 1, characterized in that: In step (2), the anomaly detection includes: Offline anomaly detection: For each data segment, the number of data is counted. When the number of data is 0, it is determined that the device is offline abnormally in the corresponding time period; Low value anomaly detection: When all data values ​​in a data segment are lower than the preset threshold a, it is judged as a low value anomaly; Constant value anomaly detection: Calculate the variance of data in a data segment. When the variance is less than or equal to the preset threshold b, it is determined to be a constant value anomaly. Evaluate data integrity rate: Ensure that there is only one data record per minute. The calculation formula is: data integrity rate = number of all valid data / total number of minutes in the detection period; Determine data accuracy: Calculate the ratio of all data to the number of abnormal data. The calculation formula is: Data accuracy = [number of all valid data - (number of constant value abnormal data + number of low value abnormal data)] / number of all valid data.

7. The automatic inspection method for monitoring time series data according to claim 1, characterized in that: Step (3) includes: C1. Recording of abnormal data: When an abnormality is detected in a data segment, the system first records the specific characteristics of the abnormal data, including the type of abnormality, the time period in which it occurred, and basic information about the device; C2. Data storage and database update: When the system stores abnormal information in the database, it first deletes the relevant records that already exist in the database, and then inserts the new abnormal detection results.

8. The automatic inspection method for monitoring time series data according to claim 2, characterized in that: Step (4) specifically includes: D1, summarize the anomaly detection results, including all marked anomaly data and their characteristics; D2. Classify the abnormal data into predefined categories and calculate the spatial and temporal distribution characteristics of the abnormal data, including the frequency, time and classification information of the abnormal occurrence; D3. Generate a structured inspection report, which includes statistical tables, charts and related analysis of abnormal data; D4. Store the generated inspection report in a document format and export the report to a local file to facilitate distribution and archiving of the report.

9. The automatic inspection method for monitoring time series data according to claim 8, characterized in that: Step D2 includes: D2.

1. Identification and classification of abnormal devices: Filter the status field of the device. The system identifies all devices other than those with a status of "normal" as abnormal objects, and selects and stores the devices with abnormal status fields. D2.

2. Statistics of abnormality rate and problem rate: The system performs statistical analysis on abnormal devices according to different dimensions. The statistical process includes calculating the total number of devices, number of abnormalities, abnormality rate and problem rate under each dimension. D2.

3. Generation of abnormality rate chart: The system visualizes the statistical results of abnormality rate and problem rate into charts to intuitively show the distribution of abnormal situations; D2.

4. Statistics of abnormal types: The system classifies and counts different types of abnormalities in detail, calculates the number, percentage and problem rate of each type of abnormality and records them; D2.

5. Saving results: The system saves all statistical results to the specified file so that detailed exception reports can be generated later.

10. An automatic inspection system for monitoring time series data, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program runs in the processor, the method for automatically inspecting time series data according to any one of claims 1 to 9 is executed.

Images