A method and electronic device for monitoring captured data

Abstract

This application discloses a method and electronic device for monitoring captured data, relating to the field of traffic technology. The method involves: acquiring a first time series consisting of at least one set of captured data within a set time range; applying a target exponential smoothing model to predict data based on part or all of the first time series, obtaining a second time series consisting of predicted data for at least one target object within the set time range; the target time type being different time periods on weekdays, holidays, or weekends; calculating a target variance based on the first and second time series; the target variance describing the closeness between the first and second time series; and determining that the captured data in the first time series is abnormal if the first and second time series do not meet the variance conditions defined by the target variance or do not meet a target threshold. This method allows for accurate monitoring of captured data, enabling timely handling of abnormal captured data.

Description

Technical Field

[0001] This application relates to the field of traffic data flow monitoring technology, and in particular to a method and electronic device for monitoring captured data. Background Technology

[0002] With the development of technology, the processes involving data access and governance in various fields are becoming increasingly complex. For example, after the public security industry receives the captured data from surveillance equipment, it needs to have it parsed by different algorithm vendors. The parsed results are then aggregated and combined with police report data for integrated analysis.

[0003] For example, there are numerous vehicle checkpoints, community access control systems, facial recognition cameras, and video surveillance devices used by public security and social construction, distributed across various locations. For instance, community surveillance data needs to be aggregated at community platforms in different districts and counties, undergo district-level facial recognition, vehicle analysis, and video structured analysis before being uploaded to the processing platform. Problems at any stage can lead to data loss, and the numerous steps in the process make troubleshooting extremely complex. Therefore, this places high demands on the capabilities of maintenance personnel.

[0004] Therefore, there is an urgent need for a method to monitor abnormalities in captured data, so as to quickly find the cause of the abnormality or the abnormal capture device and deal with it in a timely manner after the captured data becomes abnormal. Summary of the Invention

[0005] An exemplary embodiment of this application provides a method and electronic device for monitoring captured data, so as to handle abnormal captured data in a timely manner.

[0006] According to a first aspect of an exemplary embodiment, a method for monitoring captured data is provided, comprising:

[0007] Acquire a first time series consisting of at least one set of snapshot data within a set time range; wherein each set of snapshot data includes the quantity of at least one target object;

[0008] Based on part or all of the first time series, a target exponential smoothing model is applied to make predictions, resulting in a second time series consisting of the prediction data of at least one target object within a set time range; wherein, the training samples used to train the target exponential smoothing model indicate the target time type consistent with the set time range; the target time type is different time periods of weekdays, different time periods of holidays, or different time periods of weekends; the parameters of the target exponential smoothing model include a seasonality index and a smoothing coefficient.

[0009] The target variance is calculated based on the first time series and the second time series; whereby the target variance is used to describe the similarity between the first time series and the second time series.

[0010] If the first time series and the second time series do not meet the variance condition formed by the target variance, or do not meet the target threshold, then the captured data in the first time series is determined to be abnormal; wherein, the target threshold is determined based on the changing pattern of the captured data in the first time series.

[0011] According to a second aspect of an exemplary embodiment, an electronic device is provided, including a processor and a memory;

[0012] The memory is configured to execute:

[0013] Store at least one set of snapshot data within a specified time range.

[0014] The processor is configured to execute:

[0015] Acquire a first time series consisting of at least one set of snapshot data within a set time range; wherein each set of snapshot data includes the quantity of at least one target object;

[0016] Based on part or all of the first time series, a target exponential smoothing model is applied to make predictions, resulting in a second time series consisting of the prediction data of at least one target object within a set time range; wherein, the training samples used to train the target exponential smoothing model indicate the target time type consistent with the set time range; the target time type is different time periods of weekdays, different time periods of holidays, or different time periods of weekends; the parameters of the target exponential smoothing model include a seasonality index and a smoothing coefficient.

[0017] The target variance is calculated based on the first time series and the second time series; whereby the target variance is used to describe the similarity between the first time series and the second time series.

[0018] If the first time series and the second time series do not meet the variance condition formed by the target variance, or do not meet the target threshold, then the captured data in the first time series is determined to be abnormal; wherein, the target threshold is determined based on the changing pattern of the captured data in the first time series.

[0019] According to a third aspect of an exemplary embodiment, a device for monitoring captured data is provided, comprising:

[0020] The data transmission unit is used to: acquire a first time series consisting of at least one set of snapshot data within a set time range; wherein each set of snapshot data includes the quantity of at least one target object;

[0021] The data processing unit is used to: based on part or all of the first time series, apply a target exponential smoothing model to make predictions, and obtain a second time series consisting of the prediction data of at least one target object within a set time range; wherein the training samples used to train the target exponential smoothing model indicate the target time type consistent with the set time range; the target time type is different time periods of weekdays, different time periods of holidays, or different time periods of weekends; the parameters of the target exponential smoothing model include a seasonality index and a smoothing coefficient.

[0022] The data processing unit is further configured to: calculate a target variance based on the first time series and the second time series; wherein the target variance is used to describe the degree of similarity between the first time series and the second time series;

[0023] The data processing unit is further configured to: if the first time series and the second time series do not meet the variance condition formed by the target variance, or do not meet the target threshold, then determine that the source of the captured data in the first time series is an abnormal capturing device; wherein, the target threshold is determined based on the changing pattern of the captured data in the first time series.

[0024] According to a fourth aspect of an exemplary embodiment, a computer storage medium is provided, which stores computer program instructions that, when executed on a computer, cause the computer to perform the method for determining anomalies in a snapshot device as described in the first aspect.

[0025] In this embodiment, a first time series consisting of at least one set of captured data within a set time range is first obtained. Then, based on part or all of the first time series, a target exponential smoothing model is applied for prediction to obtain a second time series consisting of predicted data for at least one target object within the set time range. Finally, a target variance is calculated based on the first and second time series. If the first and second time series do not meet the variance conditions for the target variance or do not meet the target threshold, the captured data in the first time series is determined to be abnormal. In the process of applying the target exponential smoothing model for prediction, the different characteristics of captured data of different time types are considered. Therefore, different training samples can be obtained for captured data of different time types, and different target exponential smoothing models can be trained. Furthermore, removing the trend term from the target exponential smoothing model while retaining the seasonality index and smoothing coefficient can improve model accuracy and make the predicted data more accurate. Moreover, in the process of comparing captured data and predicted data, using a dual standard of variance comparison and threshold comparison can improve accuracy. The more accurate the comparison result, the more accurate the monitoring of the captured data. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 An exemplary flowchart illustrates a method for monitoring captured data provided in an embodiment of this application;

[0028] Figure 2 An exemplary flowchart illustrates a method for obtaining snapshot information provided in an embodiment of this application;

[0029] Figure 3 An exemplary flowchart illustrates a method for determining predicted data at each moment within a set time range, as provided in an embodiment of this application.

[0030] Figure 4 An exemplary flowchart illustrates a method for training a target exponential smoothing model according to an embodiment of this application;

[0031] Figure 5 An exemplary flowchart of a method for determining a target threshold provided in an embodiment of this application is shown;

[0032] Figure 6 An exemplary flowchart illustrates a method for determining whether a first time series and a second time series do not meet a target variance condition, provided in an embodiment of this application.

[0033] Figure 7 An exemplary schematic diagram illustrates the self-learning process of a target exponential smoothing model provided in an embodiment of this application;

[0034] Figure 8 An exemplary illustration shows a flowchart of a data monitoring and capture process provided in an embodiment of this application;

[0035] Figure 9 An exemplary schematic diagram of a data capture monitoring device provided in an embodiment of this application is shown.

[0036] Figure 10 An exemplary schematic diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0038] For ease of understanding, the terms used in the embodiments of this application are explained below:

[0039] (1) The Holt-Winters method is a time series analysis and forecasting method. This method is applicable to non-stationary series containing linear trends and periodic fluctuations. The Holt-Winters model is an exponential smoothing model that uses exponential smoothing to continuously adapt the model parameters to changes in the non-stationary series and to make short-term predictions of future trends. In this embodiment, the target exponential smoothing model refers to the improved Holt-Winters model.

[0040] In related technologies, when captured data needs to be uploaded to a processing platform, problems at any stage can lead to data loss. The numerous process steps also make troubleshooting extremely complex.

[0041] To address this, this application provides a method for monitoring captured data. This method considers the periodic and seasonal characteristics of captured data. For example, traffic flow is high during weekday commuting hours, while traffic flow and facial captures are lower during daytime non-commuting hours, and the number of captures at night is significantly lower than during the day. The data trends on Mondays and Fridays differ from those on Tuesdays, Wednesdays, and Thursdays; Monday mornings see relatively high traffic and pedestrian flow, while Friday afternoons have long and heavy evening rush hours; weekends are less affected by commuting times, and their data trends differ significantly from weekdays. During holidays, data flow is inconsistent with weekends due to travel restrictions. Therefore, the exponential smoothing model is improved by removing the trend term. When using the improved exponential smoothing model (target exponential smoothing model) for prediction, the periodic and seasonal characteristics of captured data are considered, meaning that the different periodic patterns of weekdays, weekends, holidays, before and after holidays, and different times of day are integrated. Furthermore, when comparing predicted and actual captured data, a dual-channel serial comparator strategy (variance comparison and threshold comparison) is used, improving the accuracy of captured data monitoring.

[0042] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application.

[0043] refer to Figure 1 The flowchart shown illustrates a method for monitoring captured data, and explains the technical solution provided in the embodiments of this application.

[0044] S101: Obtain the first time series consisting of at least one set of snapshot data within a set time range.

[0045] S102: Based on part or all of the first time series, a target exponential smoothing model is applied to make predictions, resulting in a second time series consisting of the prediction data of at least one target object within a set time range.

[0046] S103: Calculate the target variance based on the first time series and the second time series.

[0047] S104: If the first time series and the second time series do not meet the variance condition formed by the target variance, or do not meet the target threshold, then the source of the capture data in the first time series is determined to be an abnormal capture device.

[0048] The target threshold is determined based on the changing patterns of the captured data in the first time series.

[0049] In this embodiment, a first time series consisting of at least one set of captured data within a set time range is first obtained. Then, based on part or all of the first time series, a target exponential smoothing model is applied for prediction to obtain a second time series consisting of predicted data for at least one target object within the set time range. Finally, a target variance is calculated based on the first and second time series. If the first and second time series do not meet the variance conditions for the target variance or do not meet the target threshold, the captured data in the first time series is determined to be abnormal. In the process of applying the target exponential smoothing model for prediction, the different characteristics of captured data of different time types are considered. Therefore, different training samples can be obtained for captured data of different time types, and different target exponential smoothing models can be trained. Furthermore, removing the trend term from the target exponential smoothing model while retaining the seasonality index and smoothing coefficient can improve model accuracy and make the predicted data more accurate. Moreover, in the process of comparing captured data and predicted data, using a dual standard of variance comparison and threshold comparison can improve accuracy. The more accurate the comparison result, the more accurate the monitoring of the captured data.

[0050] Regarding S101, considering the periodicity and seasonality of the captured data, the set time range can be divided according to different time types. Time types include different time periods on weekdays, different time periods on holidays, or different time periods on weekends. For example, if the set time range is the morning rush hour from 7:00 to 9:00 on weekdays within a month, then a first time series consisting of at least one set of captured devices within the set time range can be obtained, wherein each set of captured data includes at least one number of target objects. Optionally, the target objects can be pedestrians or vehicles; in this embodiment, vehicles can be used as an example.

[0051] In addition, the methods for collecting the captured data are also different. Figure 2 This is a flowchart illustrating a method for acquiring snapshot information provided in an embodiment of this application. Figure 2 It should include at least the following steps:

[0052] S201: Determine the target scene information of at least one set of capture data.

[0053] Among them, captured data, as a type of data stream, can have an overall transmission architecture that includes a collection end, an aggregation end, a server end, an alarm end, and a client end. The collection end is implemented by developing corresponding collectors. Therefore, the collection end can develop corresponding clients for different collection scenarios. The aggregation end can uniformly aggregate and sort the data collected by all clients.

[0054] Optionally, different data collection scenarios can correspond to different scenario information. In this embodiment, the target scenario information may refer to scenario information in the digital government field or scenario information in the public security system, etc.

[0055] S202: Based on the pre-defined correspondence between scenarios and data acquisition methods, determine the target data acquisition method for the target scenario information.

[0056] The client can include db_agent, kafka_agent, and es_agent, and the data collection methods can be respectively: collecting the number of records in the relational database table, the real-time offset of each topic and group_id in the Kafka middleware, and the real-time data volume of each index in Elasticsearch. Therefore, the client can be matched according to the target scenario information, and the corresponding collection method of the client can be used as the target data collection method.

[0057] S203: Use the target acquisition method to obtain at least one set of capture information.

[0058] In the above embodiments, a first time series consisting of actual snapshot data within a set time range was obtained.

[0059] In step S102, based on part or all of the first time series, a target exponential smoothing model is applied to make predictions, resulting in a second time series consisting of the prediction data of at least one target object within a set time range.

[0060] Specifically, the target time type indicated by the training samples used to train the target exponential smoothing model is consistent with the set time range; the target time type is different time periods on weekdays, different time periods on holidays, or different time periods on weekends.

[0061] During the training of the target exponential smoothing model, different target exponential smoothing models can be trained for different time types. For example, if the time range is set to the morning rush hour of weekdays from 7:00 to 9:00 within a month, then the target exponential smoothing model applied here is also the model within this time range. In addition, since the time type is taken into account, the trend term in the target exponential smoothing model may affect the prediction results. Therefore, in this embodiment, the trend term is removed, and the parameters of the target exponential smoothing model include the seasonality index and the smoothing coefficient.

[0062] Optional, see Figure 3 The flowchart illustrates a method for determining the predicted data at each moment within a given time range. Figure 3 It should include at least the following steps:

[0063] S301: For the first moment, the residual data of the first moment is calculated based on the periodic data and the snapshot data of the first moment.

[0064] In this data set, the captured data is represented by X(), the residual data by a(), the periodic data by s(), and the predicted data by y(). The residual data at the first moment is a. t =α(X) t -s t-k )+(1-α)a t-k The periodic data at the first moment is s. t =γ(X) t -a t )+(1-γ)s t-k α and γ are set constant parameters, and k is the step size of the sliding process, which can be set by the user. The periodic data at the first moment is used to describe the periodic change pattern of the captured data at the first moment.

[0065] S302: Perform a moving average process on the residual data to obtain the residual data at the second time step.

[0066] Among them, the prediction of residual data is the prediction of the sequence (a t-m a t-m+1 , ...a t-2 a t-1 A moving average process is used to predict periodic data based on the sequence (s). t-mk s t-(m-1)k ,...s t-k This involves performing a moving average process on the residual data. Therefore, a moving average process can be performed on the residual data to obtain the residual data at the second time step.

[0067] S303: Calculate the predicted data for the second time step using the residual data and the periodic data at the second time step.

[0068] Optionally, the predicted data for the second time step is:

[0069] Y t+h =a t +s t-k+(h-1)modk .

[0070] Where h represents the difference between the second time point and the first time point; the larger h is, the higher the predicted value Y. t+h The larger the error, the better. Here, the captured data is a discrete time series statistically analyzed minute by minute. If we need to predict the number of captures in the next minute, then h = 1. Therefore, if we want to monitor the captured data in real time, we can set h = 1, update the input sequence whenever new captured data is available, and then predict the data for the next minute: Y t+1 =a t +s t-k .

[0071] The training process of the target exponential smoothing model will be explained next. See [link to relevant documentation]. Figure 4 The flowchart illustrates a method for training a target exponential smoothing model. Figure 4 It should include at least the following steps:

[0072] S401: Obtain training samples.

[0073] Since different target exponential smoothing models can be trained from training samples under different time types (weekdays, holidays, weekends, or specific time periods), when obtaining training samples during the model training process, the training samples are historical capture sequences composed of capture data within a historical time range, and the historical time range indicates the target time type.

[0074] In addition, to improve training accuracy, data processing can be performed after obtaining the initial training samples, such as removing outliers. The steps for removing outliers mainly include:

[0075] Noise and missing values ​​are removed, and obviously anomalous data (such as log tables showing an increasing trend, instances of zero values, or negative data volumes) are deleted. If the input sequence contains anomalous data, these anomalous data will be included in the periodic data. A method for removing anomalous data is to remove data whose trend falls between the historical maximum and minimum periodic data.

[0076] In addition, the time granularity should not be lower than the acquisition period. For example, one minute can be used as a time granularity, and the average value of the period should be taken within each time granularity. Here, null values ​​need to be processed, and the average value of all non-null values ​​within the granularity interval after noise removal should be taken.

[0077] Furthermore, when constructing the target exponential smoothing model, the dependence on data trend changes was considered, integrating the different periodic patterns of weekdays, weekends, holidays, the periods before and after holidays, and different times of day (commuting hours, rest periods). For example, for vehicle capture data, traffic flow is higher during commuting hours than at other times, and the data volume is higher during commuting hours on holidays than during weekday commuting hours. Therefore, the following dimensions were added to the time series features: whether it is a holiday, weekday, weekend, data type, timestamp, data volume, and whether it is abnormal. To improve retrieval speed and rapid statistics, time series data can also be used as a data storage engine.

[0078] Furthermore, to more accurately validate the model's performance, the test set needs to be split. For example, it can be split chronologically, with earlier data used for training the model and more recent data used for validation / testing. For instance, if training a model with data from the past 5 years, the most recent year's data can be used as the validation set, and the data from the previous 5 years can be used as the training set.

[0079] Given that snapshot data is significantly affected by weekdays, weekends, and holidays, and that the periodicity varies across different hours each day, the model is trained on data with the same type of labels. These labels include: whether it is a weekday, day of the week, whether it is a holiday, whether it is the day before or after a holiday, and the hour (distinguishing between different time periods each day, such as rush hour, daytime, and nighttime). This allows for more accurate model selection based on time labels when making predictions on data of the same type.

[0080] S402: Input the training samples into the initial exponential smoothing model for training.

[0081] The initial exponential smoothing model is also an improved Holt-Winters model, that is, a Holt-Winters model with the trend term removed. Applying the improved Holt-Winters model for modeling and forecasting can provide accurate time series forecast results.

[0082] To predict the amount of data captured in the next minute in real time, and to improve prediction accuracy, at least two months' worth of data can be used each time. The trend data in the predictor of the target exponential smoothing model represents the overall trend of the time series. However, if analyzed on a daily basis, the actual daily total amount of captured data does not change significantly, and the number of captures is large, without a clear trend. Therefore, the Holt-Winters model was modified by removing the trend term (trend data), simplifying the Holt-Winters model, resulting in the improved Holt-Winters model.

[0083] Furthermore, if the sequence contains periodic sharp increases and decreases, the model will retain these trends in the calculated periodic data, thus accurately predicting these trends without generating false alarms. Using data from the past year, the training and test sets were cross-validated in a 7:3 ratio, and the training data was used for algorithm fitting analysis.

[0084] S403: Until the training termination condition is met, the target exponential smoothing model is obtained.

[0085] During model training, based on feedback on prediction results, data points where user tasks are predicted incorrectly are considered noise points and automatically removed. Data points considered correct are used as training data by default. The model is fitted and trained daily, and iteratively upgraded. Simultaneously, model versions are saved, and an accuracy reference value for each model is calculated based on user feedback. If the number of incorrect predictions exceeds the average daily value, the model is rolled back to the previous version with higher accuracy.

[0086] Regarding S103, to compare the actual captured data with the predicted data and thus determine whether the actual captured data is abnormal, threshold comparison and variance comparison can be used. Only captured data that meets both criteria is considered normal, thus improving accuracy. Specifically, a dual-channel serial comparator can be designed and integrated into the algorithm, using this comparator to implement the comparison process.

[0087] Regarding variance, potential outliers are filtered out based on the dispersion of the prediction error curve. The variance of a sequence represents the degree of dispersion of that sequence; the larger the variance, the greater the fluctuation of the sequence. If the variance of a prediction error sequence is relatively large, then it is reasonable to consider a relatively larger alarm threshold for the prediction error.

[0088] Therefore, the target variance can be calculated based on the first time series and the second time series. The target variance is used to describe the similarity between the first time series and the second time series.

[0089] Optionally, the formula for calculating the target variance is as follows:

[0090]

[0091] Where n represents the amount of data in the second time series. For t i Real-time snapshot data, For t i Predicted data for each moment.

[0092] Regarding S104, if the first time series and the second time series do not meet the variance condition formed by the target variance, or do not meet the target threshold, then the captured data in the first time series is determined to be abnormal, and the source of the abnormal captured data in the first time series can be determined as abnormal capture equipment.

[0093] The process involves filtering out potential outliers based on whether the absolute value of the error exceeds a certain threshold. When using a variance comparator for filtering, the alarm threshold increases with the volatility of the error sequence. However, when the input data is relatively small, the variance of the error sequence is small, and the alarm threshold is also small, easily leading to false alarms. Therefore, a threshold comparison method based on the absolute value of the error was designed for filtering.

[0094] Optionally, the target threshold is determined based on the changing patterns of the captured data in the first time series; see details below. Figure 5 , Figure 5 This is a flowchart of a method for determining a target threshold provided in an embodiment of this application. Figure 5 It should include at least the following steps:

[0095] S501: Calculate the rate of change of two sets of snapshot data within a certain time interval.

[0096] Wherein, the certain time interval can be k, for example, the rate of change. For t n Real-time snapshot data.

[0097] S502: Determine the rate of change that meets the set rate of change condition as the target threshold.

[0098] The target threshold can be the largest of the various rates of change, for example, max. In practical applications, meeting the target threshold means that the rate of change is less than the target threshold.

[0099] in addition, Figure 6 This application provides a flowchart of a method for determining whether a first time series and a second time series do not meet a target variance condition. Figure 6 It should include at least the following steps:

[0100] S601: Calculate the average value of each snapshot data included in the first time series.

[0101] The formula for calculating the average value of each snapshot data included in the first time series is as follows:

[0102]

[0103] Where n is the number of captured data points, which is also the number of predicted data points.

[0104] S602: Calculate the first sum of the first differences between the first number of predicted data in the second time series and the mean.

[0105] The formula for calculating the first sum is as follows:

[0106]

[0107] S603: Calculate the second sum of the second differences between the second number of predicted data in the second time series and the average value.

[0108] The formula for calculating the second sum is as follows:

[0109]

[0110] S604: If the difference between the first sum and the second sum is greater than the product of the target variance and the set coefficient, then the first time series and the second time series are determined not to meet the target variance condition.

[0111] The formula for determining whether the first and second time series meet the target variance condition is as follows:

[0112]

[0113] Where w is a set coefficient.

[0114] This application provides a model and early warning mechanism for real-time monitoring and warning of captured data based on periodic changes. Based on the analysis of changes in the data flow of captured data, an improved exponential smoothing model is constructed to predict captured data. The difference between the predicted data and the captured data is then compared to monitor the captured data. This allows for the analysis of abnormal captured data or the fault analysis of the captured device from which the abnormal captured data originates. Real-time and accurate alarms for abnormal captured data streams are achieved. Furthermore, public safety scenarios have high requirements for the real-time performance of captured data, enabling the detection of abnormal faults within minutes.

[0115] Optional, Figure 7 This is a schematic diagram illustrating the self-learning process of a target exponential smoothing model provided in an embodiment of this application. Figure 7 As can be seen, the model can be further trained based on the comparison between the predicted data and the actual data, or the model version can be selected based on the prediction results, thereby improving the training accuracy of the model.

[0116] In order to improve the technical solution of this application, Figure 8 This is a schematic diagram illustrating a process for monitoring and capturing data, provided as an embodiment of this application. Figure 8As can be seen, in this embodiment of the application, a real-time early warning mechanism is established for the entire process of time-series data acquisition, model building, data prediction, and fault alarm. Based on the analysis of the periodicity, seasonality, and real-time nature of the captured data, a prediction model is constructed, and two types of monitoring mechanisms (variance and threshold) are designed to achieve accurate early warning of faults.

[0117] Furthermore, the method described in this application embodiment can also be applied to other fields, and the captured data can be other types of data. For example, in the field of digital government, it is necessary to connect with different units or departments. After the data is accessed, there will be corresponding ETL (Extract-Transform-Load) tools to perform operations such as cleaning, standardization, tagging, and correlation analysis. The task monitoring built into the ETL tool itself can only monitor the success or failure of task execution, and there are cases where the task is executed successfully but the data is not updated. In addition, it will also be distributed in resource repositories, theme repositories, or business repositories. Moreover, the governance scenarios of theme repositories and business repositories are mostly separate processing flows, and there is no awareness of whether the original repository is synchronized successfully. The data governance process is very long and complex, with thousands of governance tasks. It is difficult to detect if data is lost or not updated. Often, the cause is found by backward investigation after the problem is encountered, which can easily reduce customer experience and system failure. Therefore, a separate data monitoring mechanism is needed to monitor all aspects of data governance in a unified manner.

[0118] like Figure 9 As shown, based on the same inventive concept, this application provides a monitoring device for captured data, including a data transmission unit 91 and a data processing unit 92.

[0119] The data transmission unit 91 is used to: acquire a first time series consisting of at least one set of snapshot data within a set time range; wherein each set of snapshot data includes the quantity of at least one target object;

[0120] The data processing unit 92 is used to: based on part or all of the first time series, apply a target exponential smoothing model to make predictions, and obtain a second time series consisting of the prediction data of at least one target object within a set time range; wherein the training samples used to train the target exponential smoothing model indicate the target time type consistent with the set time range; the target time type is different time periods of weekdays, different time periods of holidays, or different time periods of weekends; the parameters of the target exponential smoothing model include a seasonality index and a smoothing coefficient.

[0121] The data processing unit 92 is further configured to: calculate the target variance based on the first time series and the second time series; wherein the target variance is used to describe the degree of similarity between the first time series and the second time series;

[0122] Data processing unit 92 is further configured to: if the first time series and the second time series do not meet the variance condition formed by the target variance, or do not meet the target threshold, determine that the source of the captured data in the first time series is an abnormal capturing device; wherein, the target threshold is determined based on the changing pattern of the captured data in the first time series.

[0123] In an optional implementation, the data processing unit 92 is further configured to obtain prediction data for each moment within a set time range in the following manner:

[0124] For the first moment, the residual data for the first moment is calculated based on the periodic data and the captured data for the first moment; among them, the periodic data for the first moment is used to describe the periodic change pattern of the captured data for the first moment.

[0125] Perform a moving average on the residual data to obtain the residual data at the second time step;

[0126] Using the residual data and periodic data at the second time step, the predicted data for the second time step is calculated.

[0127] In an optional implementation, the data processing unit 92 is further configured to:

[0128] Obtain training samples; where training samples are historical capture sequences consisting of capture data within a historical time range, and the historical time range indicates the target time type;

[0129] The training samples are input into the initial exponential smoothing model for training.

[0130] The training continues until the end condition is met, at which point the target exponential smoothing model is obtained.

[0131] In an optional implementation, the data processing unit 92 is further configured to determine that the first time series and the second time series do not meet the target variance condition by means of:

[0132] Calculate the average value of each snapshot data included in the first time series;

[0133] Calculate the first sum of the first differences between the first number of predicted data points and the average value in the second time series;

[0134] Calculate the second sum of the second differences between the second number of predicted data points in the second time series and the average value;

[0135] If the difference between the first sum and the second sum is greater than the product of the target variance and the set coefficient, then the first time series and the second time series are determined not to meet the target variance condition.

[0136] In an optional implementation, the data processing unit 92 is further configured to determine the target threshold in the following manner:

[0137] Calculate the rate of change of the two sets of captured data within a certain time interval;

[0138] The rate of change that satisfies the set rate of change condition is the target threshold.

[0139] In an optional implementation, the data processing unit 92 is further configured to:

[0140] Determine at least one set of target scene information from the captured data;

[0141] Based on the pre-defined correspondence between scenarios and data acquisition methods, the target data acquisition method for the target scenario information is determined.

[0142] At least one set of capture information is obtained using the target acquisition method.

[0143] Since this device is the same as the device in the method of this application embodiment, and the principle of the device in solving the problem is similar to that of the method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described again.

[0144] like Figure 10 As shown, based on the same inventive concept, this application provides an electronic device including a processor 101 and a memory 102.

[0145] Memory 102 is configured to execute:

[0146] Store at least one set of snapshot data within a specified time range.

[0147] Processor 101 is configured to execute:

[0148] Acquire a first time series consisting of at least one set of snapshot data within a set time range; wherein each set of snapshot data includes the quantity of at least one target object;

[0149] Based on part or all of the first time series, a target exponential smoothing model is applied to make predictions, resulting in a second time series consisting of the prediction data of at least one target object within a set time range; wherein, the training samples used to train the target exponential smoothing model indicate the target time type consistent with the set time range; the target time type is different time periods of weekdays, different time periods of holidays, or different time periods of weekends; the parameters of the target exponential smoothing model include a seasonality index and a smoothing coefficient.

[0150] The target variance is calculated based on the first time series and the second time series; whereby the target variance is used to describe the similarity between the first time series and the second time series.

[0151] If the first time series and the second time series do not meet the variance condition formed by the target variance, or do not meet the target threshold, then the source of the captured data in the first time series is determined to be an abnormal capture device; wherein, the target threshold is determined based on the changing pattern of the captured data in the first time series.

[0152] In an alternative implementation, processor 101 is further configured to obtain prediction data for each moment within a defined time range in the following manner:

[0153] For the first moment, the residual data for the first moment is calculated based on the periodic data and the captured data for the first moment; among them, the periodic data for the first moment is used to describe the periodic change pattern of the captured data for the first moment.

[0154] Perform a moving average on the residual data to obtain the residual data at the second time step;

[0155] Using the residual data and periodic data at the second time step, the predicted data for the second time step is calculated.

[0156] In an alternative implementation, the 101 processor is also configured to perform:

[0157] Obtain training samples; where training samples are historical capture sequences consisting of capture data within a historical time range, and the historical time range indicates the target time type;

[0158] The training samples are input into the initial exponential smoothing model for training.

[0159] The training continues until the end condition is met, at which point the target exponential smoothing model is obtained.

[0160] In an alternative implementation, processor 101 is further configured to determine that the first time series and the second time series do not meet the target variance condition by:

[0161] Calculate the average value of each snapshot data included in the first time series;

[0162] Calculate the first sum of the first differences between the first number of predicted data points and the average value in the second time series;

[0163] Calculate the second sum of the second differences between the second number of predicted data points in the second time series and the average value;

[0164] If the difference between the first sum and the second sum is greater than the product of the target variance and the set coefficient, then the first time series and the second time series are determined not to meet the target variance condition.

[0165] In an alternative implementation, the processor 101 is further configured to determine the target threshold in the following manner:

[0166] Calculate the rate of change of the two sets of captured data within a certain time interval;

[0167] The rate of change that satisfies the set rate of change condition is the target threshold.

[0168] In an alternative implementation, the processor 101 is further configured to:

[0169] Determine at least one set of target scene information from the captured data;

[0170] Based on the pre-defined correspondence between scenarios and data acquisition methods, the target data acquisition method for the target scenario information is determined.

[0171] At least one set of capture information is obtained using the target acquisition method.

[0172] This application also provides a computer storage medium storing computer program instructions. When the instructions are executed on the computer, the computer performs the steps of the above-described method for monitoring captured data.

[0173] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0175] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0176] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0177] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for monitoring captured data, characterized in that, include: Acquire at least one set of snapshot data within a set time range; wherein each set of snapshot data includes at least one target object, and the snapshot data of each target object constitutes a first time series; Based on part or all of the first time series, a target exponential smoothing model is applied for prediction to obtain a second time series consisting of the predicted data of each of the at least one target object within the set time range; wherein, the target time type indicated by the training samples used to train the target exponential smoothing model is consistent with the set time range; the target time type is different time periods of weekdays, different time periods of holidays, or different time periods of weekends; the parameters of the target exponential smoothing model include a seasonality index and a smoothing coefficient. The target variance is calculated based on the first time series and the second time series; wherein the target variance is used to describe the similarity between the first time series and the second time series; If the first time series and the second time series do not meet the variance condition formed by the target variance, or do not meet the target threshold, then the captured data in the first time series is determined to be abnormal; wherein, the target threshold is determined based on the changing pattern of the captured data in the first time series. The method further includes: The first time series and the second time series are determined not to meet the variance condition formed by the target variance in the following manner: Calculate the average value of each captured data in the first time series; calculate the first sum of the first differences between each of the first number of predicted data in the second time series and the average value; calculate the second sum of the second differences between each of the second number of predicted data in the second time series and the average value; if the difference between the first sum and the second sum is greater than the product of the target variance and the set coefficient, then determine that the first time series and the second time series do not meet the target variance condition. The first and second time series are determined not to meet the target threshold in the following manner: The rate of change between the first time series and the second time series is determined to be inconsistent with the target threshold.

2. The method according to claim 1, characterized in that, The prediction data for each moment within the set time range is obtained in the following way: For the first moment, the residual data for the first moment is calculated based on the periodic data and the captured data for the first moment; wherein, the periodic data for the first moment is used to describe the periodic change pattern of the captured data for the first moment. Perform a moving average process on the residual data to obtain the residual data at the second time step; Using the residual data and periodic data at the second time point, the predicted data for the second time point is calculated.

3. The method according to claim 1, characterized in that, The method further includes: Obtain training samples; wherein, the training samples are historical capture sequences composed of capture data within a historical time range, and the historical time range indicates the target time type; The training samples are input into the initial exponential smoothing model for training. The training continues until the end condition is met, at which point the target exponential smoothing model is obtained.

4. The method according to claim 1, characterized in that, The method further includes determining the target threshold in the following manner: Calculate the rate of change of the two sets of captured data within a certain time interval; The rate of change that satisfies the set rate of change condition is the target threshold.

5. The method according to claim 1, characterized in that, The method further includes: Determine the target scene information of the at least one set of captured data; Based on the pre-defined correspondence between scenarios and data acquisition methods, the target data acquisition method for the target scenario information is determined; The target data acquisition method described above is used to obtain at least one set of capture information.

6. The method according to any one of claims 1 to 5, characterized in that, After determining that the captured data in the first time series is abnormal, the method further includes: The source of the captured data in the first time series was determined to be an abnormal capture device.

7. An electronic device, characterized in that, Including processor and memory; The memory is configured to execute: Store at least one set of snapshot data within a specified time range. The processor is configured to execute: Acquire at least one set of snapshot data within a set time range; wherein each set of snapshot data includes at least one target object, and the snapshot data of each target object constitutes a first time series; Based on part or all of the first time series, a target exponential smoothing model is applied for prediction to obtain a second time series consisting of the predicted data of each of the at least one target object within the set time range; wherein, the target time type indicated by the training samples used to train the target exponential smoothing model is consistent with the set time range; the target time type is different time periods of weekdays, different time periods of holidays, or different time periods of weekends; the parameters of the target exponential smoothing model include a seasonality index and a smoothing coefficient. The target variance is calculated based on the first time series and the second time series; wherein the target variance is used to describe the similarity between the first time series and the second time series; If the first time series and the second time series do not meet the variance condition formed by the target variance, or do not meet the target threshold, then the captured data in the first time series is determined to be abnormal; wherein, the target threshold is determined based on the changing pattern of the captured data in the first time series. The processor is also configured to execute: The first time series and the second time series are determined not to meet the variance condition formed by the target variance in the following manner: Calculate the average value of each captured data in the first time series; calculate the first sum of the first differences between each of the first number of predicted data in the second time series and the average value; calculate the second sum of the second differences between each of the second number of predicted data in the second time series and the average value; if the difference between the first sum and the second sum is greater than the product of the target variance and the set coefficient, then determine that the first time series and the second time series do not meet the target variance condition. The first and second time series are determined not to meet the target threshold in the following manner: The rate of change between the first time series and the second time series is determined to be inconsistent with the target threshold.

8. The electronic device according to claim 7, characterized in that, The processor is also configured to obtain prediction data for each moment within the defined time range in the following manner: For the first moment, the residual data for the first moment is calculated based on the periodic data and the captured data for the first moment; wherein, the periodic data for the first moment is used to describe the periodic change pattern of the captured data for the first moment. Perform a moving average process on the residual data to obtain the residual data at the second time step; Using the residual data and periodic data at the second time point, the predicted data for the second time point is calculated.

9. The electronic device according to claim 7, characterized in that, The processor is also configured to execute: Obtain training samples; wherein, the training samples are historical capture sequences composed of capture data within a historical time range, and the historical time range indicates the target time type; The training samples are input into the initial exponential smoothing model for training. The training continues until the end condition is met, at which point the target exponential smoothing model is obtained.

Images