Snapshot data monitoring method and electronic equipment

Abstract

The invention discloses a monitoring method for snapshot data and electronic equipment, and relates to the technical field of traffic. The method comprises the following steps: acquiring a first time sequence formed by at least one group of snapshot data in a set time range; based on part or all of the first time series, applying a target exponential smoothing model to perform prediction to obtain a second time series formed by respective prediction data of at least one target object in a set time range; the target time type is different time periods of workdays, different time periods of holidays or different time periods of weekends; calculating a target variance according to the first time sequence and the second time sequence; the target variance is used for describing the closeness degree of the first time sequence and the second time sequence; and if the first time sequence and the second time sequence do not meet a variance condition formed by the target variance or do not meet a target threshold, determining that the snapshot data in the first time sequence is abnormal. The snapshot data can be accurately monitored, so that the snapshot data can be timely processed when being abnormal.

Description

Technical Field

[0001] The present application relates to the technical field of traffic data flow monitoring, and in particular to a monitoring method and electronic equipment for captured data. Background Art

[0002] With the development of science and technology, the processes involved in data access management in various fields are becoming more and more complicated. For example, after the public security industry receives the captured data from the monitoring equipment, it needs to be analyzed by different algorithm manufacturers, and the results of the analysis are aggregated and then combined with the police data for fusion analysis.

[0003] For example, there are many vehicle checkpoints, community access control, face capture equipment, video surveillance equipment, etc. for public security and social construction, and they are distributed in different places. For example, community capture data needs to be aggregated on the community platforms of various districts and counties, and then uploaded to the processing platform after district-level face analysis, vehicle analysis, and video structured analysis. Problems in any link may lead to data loss, and the large number of process links makes it very complicated to troubleshoot problems. Therefore, the ability of operation and maintenance personnel is very high.

[0004] Therefore, there is an urgent need for a method that can monitor abnormal conditions in captured data so that after the captured data is abnormal, the cause of the abnormality or the abnormal capture device can be quickly found and handled in a timely manner. Summary of the invention

[0005] In an exemplary embodiment of the present application, a method for monitoring captured data and an electronic device are provided to monitor the captured data so as to process the captured data in a timely manner when the captured data is abnormal.

[0006] According to a first aspect of an exemplary embodiment, a method for monitoring captured data is provided, comprising:

[0007] Acquire a first time series consisting of at least one group of snapshot data within a set time range; wherein each group of snapshot data includes the number of at least one target object;

[0008] Based on part or all of the first time series, a target exponential smoothing model is applied to perform prediction to obtain a second time series consisting of prediction data of at least one target object within a set time range; wherein the target time type indicated by the training sample used to train the target exponential smoothing model is consistent with the set time range; the target time type is different time periods on weekdays, different time periods on holidays, or different time periods on weekends; and the parameters of the target exponential smoothing model include a seasonal index and a smoothing coefficient;

[0009] Calculating a target variance according to the first time series and the second time series; wherein the target variance is used to describe the degree of closeness between the first time series and the second time series;

[0010] If the first time series and the second time series do not satisfy the variance condition of the target variance, or do not satisfy the target threshold, the captured data in the first time series is determined to be abnormal; wherein the target threshold is determined according to the change rule of the captured data in the first time series.

[0011] According to a second aspect of the exemplary implementation, there is provided an electronic device, including a processor and a memory;

[0012] Memory, configured to execute:

[0013] Store at least one set of snapshot data within a set time range

[0014] A processor configured to execute:

[0015] Acquire a first time series consisting of at least one group of snapshot data within a set time range; wherein each group of snapshot data includes the number of at least one target object;

[0016] Based on part or all of the first time series, a target exponential smoothing model is applied to perform prediction to obtain a second time series consisting of prediction data of at least one target object within a set time range; wherein the target time type indicated by the training sample used to train the target exponential smoothing model is consistent with the set time range; the target time type is different time periods on weekdays, different time periods on holidays, or different time periods on weekends; and the parameters of the target exponential smoothing model include a seasonal index and a smoothing coefficient;

[0017] Calculating a target variance according to the first time series and the second time series; wherein the target variance is used to describe the degree of closeness between the first time series and the second time series;

[0018] If the first time series and the second time series do not satisfy the variance condition of the target variance, or do not satisfy the target threshold, the captured data in the first time series is determined to be abnormal; wherein the target threshold is determined according to the change rule of the captured data in the first time series.

[0019] According to a third aspect of an exemplary embodiment, a monitoring device for captured data is provided, including:

[0020] The data transmission unit is used to: obtain a first time series consisting of at least one group of snapshot data within a set time range; wherein each group of snapshot data includes the number of at least one target object;

[0021] The data processing unit is used to: apply the target exponential smoothing model to perform prediction based on part or all of the first time series, and obtain a second time series consisting of prediction data of at least one target object in a set time range; wherein the target time type indicated by the training sample used to train the target exponential smoothing model is consistent with the set time range; the target time type is different time periods on weekdays, different time periods on holidays, or different time periods on weekends; the parameters of the target exponential smoothing model include a seasonal index and a smoothing coefficient;

[0022] The data processing unit is further used to: calculate a target variance according to the first time series and the second time series; wherein the target variance is used to describe the degree of proximity between the first time series and the second time series;

[0023] The data processing unit is further used to: if the first time series and the second time series do not meet the variance condition of the target variance, or do not meet the target threshold, determine that the source capture device of the captured data in the first time series is abnormal; wherein the target threshold is determined according to the change law of the captured data in the first time series

[0024] According to a fourth aspect of the exemplary implementation, a computer storage medium is provided, in which computer program instructions are stored. When the instructions are executed on a computer, the computer executes the method for determining abnormality of a capture device as described in the first aspect.

[0025] In the embodiment of the present application, a first time series consisting of at least one group of captured data within a set time range is first obtained, and then based on part or all of the first time series, a target exponential smoothing model is applied for prediction to obtain a second time series consisting of the predicted data of at least one target object within the set time range, and finally the target variance is calculated according to the first time series and the second time series. If the first time series and the second time series do not meet the variance condition of the target variance, or do not meet the target threshold, the captured data in the first time series is determined to be abnormal. In the process of applying the target exponential smoothing model for prediction, the characteristics of the captured data of different time types are different. Therefore, different training samples can be obtained for the captured data of different time types and different target exponential smoothing models can be trained. In addition, removing the trend term in the target exponential smoothing model and retaining the seasonal index and smoothing coefficient can improve the model accuracy and make the predicted data more accurate. In addition, in the process of comparing the captured data and the predicted data, the dual standards of variance comparison and threshold comparison can be used to improve the accuracy. The more accurate the comparison result is, the more accurate the monitoring of the captured data is. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 A flowchart of a method for monitoring captured data provided by an embodiment of the present application is exemplified;

[0028] Figure 2 A flowchart of a method for obtaining snapshot information provided by an embodiment of the present application is exemplarily shown;

[0029] Figure 3 A flowchart of a method for determining prediction data at each moment within a set time range provided by an embodiment of the present application is exemplarily shown;

[0030] Figure 4 A method flow chart of a training process of a target exponential smoothing model provided in an embodiment of the present application is exemplarily shown;

[0031] Figure 5 A flow chart of a method for determining a target threshold value provided by an embodiment of the present application is exemplarily shown;

[0032] Figure 6 A flowchart of a method for determining that a first time series and a second time series do not meet a target variance condition provided by an embodiment of the present application is exemplarily shown;

[0033] Figure 7 A schematic diagram of a self-learning process of a target exponential smoothing model provided in an embodiment of the present application is exemplified;

[0034] Figure 8 The following is a schematic diagram of a process for monitoring and capturing data provided by an embodiment of the present application;

[0035] Fig. 9 The following is a schematic diagram showing the structure of a monitoring device for capturing data provided by an embodiment of the present application;

[0036] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is exemplarily shown. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0038] For ease of understanding, the terms involved in the embodiments of the present application are explained below:

[0039] (1) The Holt-Winters method is a time series analysis and prediction method. This method is applicable to non-stationary series containing linear trends and periodic fluctuations. The Holt-Winters model is an exponential smoothing model that uses exponential smoothing to continuously adapt model parameters to changes in non-stationary series and make short-term predictions of future trends. In the embodiments of the present application, the target exponential smoothing model refers to the improved Holt-Winters model.

[0040] In the related technology, when the captured data needs to be uploaded to the processing platform, problems in each link will lead to data loss, and there are many process links, which makes it very complicated to troubleshoot after the problem occurs.

[0041] To this end, the embodiment of the present application provides a monitoring method for captured data, in which the periodicity and seasonality of the captured data are taken into account, such as the large traffic volume during the commuting time on weekdays, the small traffic volume and face capture during the non-commuting time during the day, and the number of captures at night is much less than that during the day. The data trends on Monday and Friday are different from those on Tuesday, Wednesday, and Thursday; the traffic volume and passenger volume on Monday morning are relatively large, and the peak time on Friday afternoon is long and the volume is large; while the weekend is less affected by the commuting time, and the data trend is significantly different from that on weekdays. During holidays, the data volume is affected by the return trip to the hometown, and the weekend is inconsistent. Therefore, the exponential smoothing model is improved, the trend item is removed, and when the improved exponential smoothing model (target exponential smoothing model) is applied for prediction, the periodicity and seasonality of the captured data are taken into account, that is, the different laws of periodicity of working days, weekends, holidays, before and after holidays, and different times of the day are integrated. In addition, when the predicted captured data and the actual captured data are compared, a dual-channel serial comparator strategy (variance comparison and threshold comparison) is involved, which improves the accuracy of monitoring the captured data.

[0042] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiment of the present application.

[0043] refer to Figure 1 A flowchart of a method for monitoring captured data is shown to illustrate the technical solution provided in an embodiment of the present application.

[0044] S101: Acquire a first time series consisting of at least one set of snapshot data within a set time range.

[0045] S102: Based on part or all of the first time series, a target exponential smoothing model is applied to perform prediction to obtain a second time series consisting of prediction data of at least one target object within a set time range.

[0046] S103: Calculate the target variance according to the first time series and the second time series.

[0047] S104: If the first time series and the second time series do not satisfy the variance condition of the target variance, or do not satisfy the target threshold, it is determined that the source capture device of the capture data in the first time series is abnormal.

[0048] The target threshold is determined according to the change rule of the captured data in the first time series.

[0049] In the embodiment of the present application, a first time series consisting of at least one group of captured data within a set time range is first obtained, and then based on part or all of the first time series, a target exponential smoothing model is applied for prediction to obtain a second time series consisting of the predicted data of at least one target object within the set time range, and finally the target variance is calculated according to the first time series and the second time series. If the first time series and the second time series do not meet the variance condition of the target variance, or do not meet the target threshold, the captured data in the first time series is determined to be abnormal. In the process of applying the target exponential smoothing model for prediction, the characteristics of the captured data of different time types are different. Therefore, different training samples can be obtained for the captured data of different time types and different target exponential smoothing models can be trained. In addition, removing the trend term in the target exponential smoothing model and retaining the seasonal index and smoothing coefficient can improve the model accuracy and make the predicted data more accurate. In addition, in the process of comparing the captured data and the predicted data, the dual standards of variance comparison and threshold comparison can be used to improve the accuracy. The more accurate the comparison result is, the more accurate the monitoring of the captured data is.

[0050] Regarding S101, considering the periodicity and seasonality of the captured data, the set time range here can be divided according to different time types. Time types include different time periods on weekdays, different time periods on holidays, or different time periods on weekends. For example, if the time range is set to 7:00-9:00 during the morning rush hour on weekdays within a month, a first time series consisting of at least one group of capture devices within the set time range can be obtained, wherein each group of capture data includes at least one number of each target object. Optionally, the target object can be a pedestrian or a vehicle. In the embodiment of the present application, a vehicle can be used for example.

[0051] In addition, the capture data is collected in different ways. Figure 2 A flowchart of a method for obtaining snapshot information provided in an embodiment of the present application, Figure 2 At least the following steps are included:

[0052] S201: Determine target scene information of at least one set of captured data.

[0053] Among them, the snapshot data is a type of data stream, and the overall transmission architecture of the data stream can be divided into the collection end, aggregation end, service end, alarm end and client end. The collection end is realized by developing the corresponding collector. Therefore, the collection end can develop corresponding clients for different collection scenarios. The aggregation end can uniformly aggregate and sort the data collected by all clients.

[0054] Optionally, different collection scenarios may correspond to different scene information. In the embodiment of the present application, the target scene information may refer to scene information in the digital government field or scene information in the public security system.

[0055] S202: Determine a target data collection method for target scene information based on a correspondence between a preset scene and a data collection method.

[0056] Among them, the client can include db_agent, kafka_agent and es_agent, and the collection methods include collecting the number of records in the relational database table, the real-time offset of each topic and group_id of the kafka middleware, and the real-time data volume of each index of es. Therefore, the client can be matched according to the target scenario information, and the collection method corresponding to the corresponding client is applied as the target data collection method.

[0057] S203: Obtain at least one set of snapshot information by applying a target acquisition method.

[0058] In the above embodiment, a first time series consisting of actual captured data within a set time range is obtained.

[0059] In S102, based on part or all of the first time series, a target exponential smoothing model is applied to perform prediction to obtain a second time series consisting of prediction data of at least one target object within a set time range.

[0060] Specifically, the target time type indicated by the training samples used to train the target exponential smoothing model is consistent with the set time range; the target time type is different time periods on weekdays, different time periods on holidays, or different time periods on weekends.

[0061] In the training process of the target exponential smoothing model, different target exponential smoothing models can be trained for different time types. For example, if the time range is set to 7:00-9:00 during the morning rush hour of working days within a month, the target exponential smoothing model applied here is also a model within this time range. In addition, since the time type is taken into account, the trend term in the target exponential smoothing model may affect the prediction result. Therefore, in the embodiment of the present application, the trend term is removed, and the parameters of the target exponential smoothing model include the seasonal index and the smoothing coefficient.

[0062] Optional, see Figure 3 , showing a flow chart of a method for determining forecast data at each moment within a set time range, Figure 3 At least the following steps are included:

[0063] S301: for a first moment, according to the periodic data at the first moment and the captured data at the first moment, calculate and obtain the residual data at the first moment.

[0064] Among them, the captured data is represented by X(), the residual data is represented by a(), the periodic data is represented by s(), and the predicted data is represented by y(). The residual data at the first moment is a t =α(X t -s t-k )+(1-α)a t-k , the periodic data at the first moment is s t =γ(X t -a t )+(1-γ)s t-k , α and γ are set constant parameters respectively, and k is the step size of sliding processing, which can be set by yourself. The periodic data at the first moment is used to describe the periodic change law of the captured data at the first moment.

[0065] S302: Perform a sliding average process on the residual data to obtain the residual data at the second moment.

[0066] Among them, the prediction of residual data is the sequence (a t-m , a t-m+1 , ...a t-2 , a t-1 ) is a sliding average processing process. The prediction of periodic data is a series (s t-mk ,s t-(m-1)k ,...s t-k ). Therefore, a sliding average process can be performed on the residual data to obtain the residual data at the second moment.

[0067] S303: Apply the residual data at the second moment and the periodic data at the second moment to calculate the predicted data at the second moment.

[0068] Optionally, the forecast data at the second moment is:

[0069] Y t+h =a t +s t-k+(h-1)modk .

[0070] Among them, h represents the difference between the second moment and the first moment. The larger h is, the better the predicted value Y is. t+h The larger the error is, the greater the error is. Here, the snapshot data is a discrete time series counted by minute. If you need to predict the snapshot amount of the next minute, then h = 1. Therefore, if you want to monitor the snapshot data in real time, you can set h = 1. Whenever there is new snapshot data, update the input sequence and then predict the data for the next minute: Y t+1 =a t +s t-k .

[0071] Next, the training process of the target exponential smoothing model is described. Figure 4 , showing a method flow chart of the training process of a target exponential smoothing model, Figure 4 At least the following steps are included:

[0072] S401: Obtain training samples.

[0073] Among them, since training samples under different time types (weekdays, holidays or weekends, or specific time periods) can train different target exponential smoothing models, when obtaining training samples in the model training process, the training samples are historical snapshot sequences composed of snapshot data within the historical time range, and the historical time range indicates the target time type.

[0074] In addition, in order to improve the accuracy of training, after obtaining the initial training samples, data processing can be performed first, such as removing abnormal data. The steps for removing abnormal data mainly include:

[0075] Remove noise and missing values, and delete obviously abnormal data (such as log tables with an increasing trend, the number of 0, and negative data). If the input sequence contains abnormal data, these abnormal data will be calculated into the periodic data. The method for removing abnormal data can be to remove data with a change trend between the historical maximum period data and the historical minimum period data.

[0076] In addition, the time granularity cannot be lower than the collection period. For example, one minute is a time granularity, and the average value of this period is taken in each time granularity. Here, the null values ​​need to be processed, and the average value of all non-null values ​​in the granularity interval after noise removal is taken.

[0077] In addition, when constructing the target exponential smoothing model, the dependencies of data situation changes are taken into account, integrating the different periodic patterns of weekdays, weekends, holidays, before and after holidays, and different times of the day (commuting time, rest time). For example, for the data of passing vehicles, the traffic volume during commuting time is greater than other time periods, and the amount of data during the working hours of holidays is greater than that during the working hours of weekdays. Therefore, the following dimensions are added to the time series features: whether it is a holiday, weekday, weekend, data type, timestamp, data volume, whether it is abnormal, etc. In order to improve the retrieval speed and fast statistics, the time series can also be used as a data storage engine.

[0078] In addition, if you want to verify the model effect more accurately, you need to split the test set. For example, you can split it according to time, with the earlier period of data used for training the model and the more recent period of data used for verifying / testing the model effect. For example, if you use the data from the past five years to train the model, you can set aside the data from the most recent year as the verification set, and the data from the previous five years as the training set.

[0079] Since the snapshot data has a greater impact on weekdays, weekends, and holidays, and the periodicity of different hours of each day is different, the model is trained for data with the same type of labels. The type labels include: whether it is a weekday, the day of the week, whether it is a holiday, whether it is the day before or after the holiday, and the hour (the hour is used as the granularity to distinguish different time periods of each day such as rush hour, daytime, and nighttime). When the model prediction is performed on the data with the same type, the corresponding model is selected according to the time label, which is more accurate.

[0080] S402: Input the training samples into the initial exponential smoothing model for training.

[0081] The initial exponential smoothing model is also an improved Holt-Winters model, that is, a Holt-Winters model without the trend term. Applying the improved Holt-Winters model for modeling and forecasting can provide accurate time series forecasting results.

[0082] In order to predict the amount of snapshot data in the next minute in real time and to improve the prediction effect, at least two months of data can be used each time. The trend data of the predictor in the target exponential smoothing model represents the overall change trend of the time series. If analyzed on a daily basis, the total amount of snapshot data per day does not change much, and the number of snapshots is large, with no obvious change trend. Therefore, the Holt-Winters model was modified by removing the trend item (trend part of the data) and simplifying the Holt-Winters model, which is the improved Holt-Winters model.

[0083] In addition, if the sequence contains periodic steep increase and steep decrease points, the periodic data calculated by the model will retain these steep increase and steep decrease trends, so these trends can be accurately predicted without false positives. Taking the data of the past year, the training set and test set data are cross-validated in a ratio of 7:3, and the training data is used for algorithm fitting analysis.

[0084] S403: until the training end condition is met, a target exponential smoothing model is obtained.

[0085] During the model training process, based on the feedback of the prediction results, the data points with incorrect predictions for user tasks are considered as noise points, and this part of the data is automatically removed. The correct data is considered as the training data by default, and the training model is fitted and upgraded iteratively every day. At the same time, the model version is saved, and the accuracy reference value of each model is calculated based on the user feedback. If the number of errors in the predicted results every day exceeds the average, the model will be rolled back to the previous model version with high accuracy.

[0086] In S103, in order to compare the actual captured data with the predicted data, and then determine whether the actual captured data is abnormal. In this process, threshold comparison and variance comparison can be used, and only the captured data that meets the requirements are considered normal, which can improve the accuracy. The specific implementation process can be to design a dual-channel serial comparator, integrate it into the algorithm, and use the dual-channel serial comparator to implement the comparison process.

[0087] For variance, possible outliers are filtered out according to the degree of dispersion of the forecast error curve. The variance of a sequence indicates the degree of dispersion of the sequence. The larger the variance, the greater the volatility of the sequence. If the variance of a forecast error sequence is relatively large, it is reasonable to consider that the alarm threshold of the forecast error is relatively large.

[0088] Therefore, the target variance can be calculated based on the first time series and the second time series, where the target variance is used to describe the degree of closeness between the first time series and the second time series.

[0089] Optionally, the target variance is calculated as follows:

[0090]

[0091] Where n represents the amount of data in the second time series, t i The snapshot data at the moment, t i Forecast data at the moment.

[0092] In S104, if the first time series and the second time series do not satisfy the variance condition of the target variance, or do not satisfy the target threshold, it is determined that the captured data in the first time series is abnormal, and then it can be determined that the source of the abnormal captured data in the first time series is abnormal.

[0093] Among them, possible abnormal points are filtered out according to whether the absolute value of the error exceeds a certain threshold. When using the variance comparator for filtering, the alarm threshold increases as the fluctuation of the error sequence increases. However, when the input data is relatively small, the variance of the error sequence is relatively small, and the alarm threshold is also small, which is prone to false alarms. Therefore, a threshold comparison based on the absolute value of the error is designed for filtering.

[0094] Optionally, the target threshold is determined according to the change pattern of the captured data in the first time series. For details, see Figure 5 , Figure 5 A flow chart of a method for determining a target threshold provided in an embodiment of the present application, Figure 5 At least the following steps are included:

[0095] S501: Calculate the change rates of two groups of captured data within a certain time interval respectively.

[0096] Among them, the certain time interval can be k, for example, the change rate is t n Snapshot data at the moment.

[0097] S502: Determine the change rate that meets the set change rate condition as the target threshold.

[0098] The target threshold can be the maximum of the various change rates, for example, max In actual application, satisfying the target threshold means that the rate of change is less than the target threshold.

[0099] in addition, Figure 6 A flow chart of a method for determining that a first time series and a second time series do not meet a target variance condition provided in an embodiment of the present application, Figure 6 At least the following steps are included:

[0100] S601: Calculate the average value of each captured data included in the first time series.

[0101] The calculation formula for the average value of each captured data included in the first time series is as follows:

[0102]

[0103] Among them, n is the number of captured data, which is also the number of predicted data.

[0104] S602: Calculate a first sum of first differences between each of the first number of predicted data in the second time series and the average value.

[0105] The calculation formula for the first sum is as follows:

[0106]

[0107] S603: Calculate a second sum of second differences between each of the second number of predicted data in the second time series and the average value.

[0108] The calculation formula for the second sum is as follows:

[0109]

[0110] S604: If the difference between the first sum and the second sum is greater than the product of the target variance and the set coefficient, it is determined that the first time series and the second time series do not meet the target variance condition.

[0111] The formula for determining that the first time series and the second time series do not meet the target variance condition is as follows:

[0112]

[0113] Among them, w is the setting coefficient.

[0114] The embodiment of the present application provides a model and early warning mechanism for real-time snapshot data monitoring and early warning based on periodic changes. Based on the analysis of the changes in the data flow of the snapshot data, the exponential smoothing model is improved, and a target exponential smoothing model is constructed to predict the snapshot data, and then compare the difference between the predicted data and the snapshot data to achieve monitoring of the snapshot data, and then analyze the abnormal snapshot data, or perform fault analysis on the source snapshot device of the abnormal snapshot data. Real-time and accurate alarms for abnormal situations in the snapshot data flow are achieved. In addition, public safety scenarios have relatively high requirements for the real-time nature of snapshot data, and abnormal faults can be discovered at the minute level.

[0115] Optional, Figure 7 A schematic diagram of a self-learning process of a target exponential smoothing model provided in an embodiment of the present application, through Figure 7 It can be seen that the model can be further trained based on the comparison results of the predicted data and the actual data, or the model version can be determined based on the model's prediction results to improve the model training accuracy.

[0116] In order to make the technical solution of this application more perfect, Figure 8 A schematic diagram of a process for monitoring and capturing data provided by an embodiment of the present application. Figure 8It can be seen that the embodiment of the present application constructs a prediction model through a real-time warning mechanism for the entire process of time series data from data collection, model construction, data prediction, and fault alarm, based on the analysis of the periodicity, seasonality, and real-time nature of the captured data, and designs two types of monitoring mechanisms (variance and threshold) to achieve accurate early warning of faults.

[0117] In addition, the method in the embodiment of the present application can also be applied to other fields, and the captured data can also be other data. For example, in the field of digital government, it is necessary to connect with different units or departments. After the data is accessed, there will be corresponding ETL (Extract-Transform-Load) tools for cleaning, standardization, labeling, association analysis and other operations. The ETL tool itself carries out task monitoring, which can only monitor the status of the success or failure of the task execution. There is a situation where the task is successfully executed but the data is not updated. In addition, it will also be distributed to resource libraries, theme libraries or business libraries. And the governance scenarios for theme libraries and business libraries are mostly separate processing flows, and there is no perception of whether the synchronization of the original library is successful or not. The data governance process is very long and complex, with as many as thousands of governance tasks. It is difficult to find that one type of data is lost or not updated. It is often necessary to reverse the cause after encountering a problem, which is easy to reduce customer experience and system failures. Therefore, a separate data monitoring mechanism is needed to uniformly monitor all aspects of data governance.

[0118] like Fig. 9 As shown, based on the same inventive concept, an embodiment of the present application provides a monitoring device for captured data, including a data transmission unit 91 and a data processing unit 92 .

[0119] The data transmission unit 91 is used to: obtain a first time series consisting of at least one group of snapshot data within a set time range; wherein each group of snapshot data includes the number of at least one target object;

[0120] The data processing unit 92 is used to: apply the target exponential smoothing model to perform prediction based on part or all of the first time series, and obtain a second time series consisting of prediction data of at least one target object in a set time range; wherein the target time type indicated by the training sample used to train the target exponential smoothing model is consistent with the set time range; the target time type is different time periods on weekdays, different time periods on holidays, or different time periods on weekends; the parameters of the target exponential smoothing model include a seasonal index and a smoothing coefficient;

[0121] The data processing unit 92 is further used to: calculate a target variance according to the first time series and the second time series; wherein the target variance is used to describe the degree of proximity between the first time series and the second time series;

[0122] The data processing unit 92 is further used to: if the first time series and the second time series do not meet the variance condition of the target variance, or do not meet the target threshold, determine that the source capture device of the captured data in the first time series is abnormal; wherein the target threshold is determined according to the change law of the captured data in the first time series

[0123] In an optional implementation, the data processing unit 92 is further configured to obtain the forecast data at each moment within the set time range in the following manner:

[0124] For the first moment, the residual data of the first moment is calculated according to the periodic data of the first moment and the captured data of the first moment; wherein the periodic data of the first moment is data used to describe the periodic change law of the captured data of the first moment;

[0125] Perform a sliding average process on the residual data to obtain the residual data at the second moment;

[0126] The residual data at the second moment and the periodic data at the second moment are applied to calculate the predicted data at the second moment.

[0127] In an optional implementation, the data processing unit 92 is further configured to:

[0128] Acquire training samples; wherein the training samples are historical snapshot sequences composed of snapshot data within a historical time range, and the historical time range indicates a target time type;

[0129] Input the training samples into the initial exponential smoothing model for training;

[0130] Until the training end conditions are met, the target exponential smoothing model is obtained.

[0131] In an optional implementation, the data processing unit 92 is further configured to determine that the first time series and the second time series do not meet the target variance condition in the following manner:

[0132] Calculate the average value of each captured data included in the first time series;

[0133] Calculate a first sum of first differences between each of the first number of predicted data in the second time series and the average value;

[0134] Calculate a second sum of second differences between each of the second number of predicted data in the second time series and the average value;

[0135] If the difference between the first sum and the second sum is greater than the product of the target variance and the set coefficient, it is determined that the first time series and the second time series do not meet the target variance condition.

[0136] In an optional implementation, the data processing unit 92 is further configured to determine the target threshold value in the following manner:

[0137] Calculate the change rates of two groups of captured data within a certain time interval respectively;

[0138] The change rate that meets the set change rate condition is determined as the target threshold.

[0139] In an optional implementation, the data processing unit 92 is further configured to:

[0140] Determine target scene information of at least one set of captured data;

[0141] Determine the target data collection method of the target scene information based on the correspondence between the preset scene and the data collection method;

[0142] Apply the target collection method to obtain at least one set of snapshot information.

[0143] Since the device is the device in the method in the embodiment of the present application, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0144] like Fig.10 As shown, based on the same inventive concept, an embodiment of the present application provides an electronic device, including a processor 101 and a memory 102.

[0145] The memory 102 is configured to execute:

[0146] Store at least one set of snapshot data within a set time range

[0147] The processor 101 is configured to execute:

[0148] Acquire a first time series consisting of at least one group of snapshot data within a set time range; wherein each group of snapshot data includes the number of at least one target object;

[0149] Based on part or all of the first time series, a target exponential smoothing model is applied to perform prediction to obtain a second time series consisting of prediction data of at least one target object within a set time range; wherein the target time type indicated by the training sample used to train the target exponential smoothing model is consistent with the set time range; the target time type is different time periods on weekdays, different time periods on holidays, or different time periods on weekends; and the parameters of the target exponential smoothing model include a seasonal index and a smoothing coefficient;

[0150] Calculating a target variance according to the first time series and the second time series; wherein the target variance is used to describe the degree of closeness between the first time series and the second time series;

[0151] If the first time series and the second time series do not meet the variance condition of the target variance, or do not meet the target threshold, it is determined that the source capture device of the capture data in the first time series is abnormal; wherein the target threshold is determined according to the change law of the capture data in the first time series.

[0152] In an optional implementation, the processor 101 is further configured to execute, obtaining the prediction data at each moment within the set time range in the following manner:

[0153] For the first moment, the residual data of the first moment is calculated according to the periodic data of the first moment and the captured data of the first moment; wherein the periodic data of the first moment is data used to describe the periodic change law of the captured data of the first moment;

[0154] Perform a sliding average process on the residual data to obtain the residual data at the second moment;

[0155] The residual data at the second moment and the periodic data at the second moment are applied to calculate the predicted data at the second moment.

[0156] In an optional implementation, the processor 101 is further configured to execute:

[0157] Acquire training samples; wherein the training samples are historical snapshot sequences composed of snapshot data within a historical time range, and the historical time range indicates a target time type;

[0158] Input the training samples into the initial exponential smoothing model for training;

[0159] Until the training end conditions are met, the target exponential smoothing model is obtained.

[0160] In an optional implementation, the processor 101 is further configured to perform the following method to determine that the first time series and the second time series do not meet the target variance condition:

[0161] Calculate the average value of each captured data included in the first time series;

[0162] Calculate a first sum of first differences between each of the first number of predicted data in the second time series and the average value;

[0163] Calculate a second sum of second differences between each of the second number of predicted data in the second time series and the average value;

[0164] If the difference between the first sum and the second sum is greater than the product of the target variance and the set coefficient, it is determined that the first time series and the second time series do not meet the target variance condition.

[0165] In an optional implementation, the processor 101 is further configured to determine the target threshold in the following manner:

[0166] Calculate the change rates of two groups of captured data within a certain time interval respectively;

[0167] The change rate that meets the set change rate condition is determined as the target threshold.

[0168] In an optional implementation, the processor 101 is further configured to:

[0169] Determine target scene information of at least one set of captured data;

[0170] Determine the target data collection method of the target scene information based on the correspondence between the preset scene and the data collection method;

[0171] Apply the target collection method to obtain at least one set of snapshot information.

[0172] The embodiment of the present application also provides a computer storage medium, in which computer program instructions are stored. When the instructions are executed on a computer, the computer executes the steps of the above-mentioned monitoring method for captured data.

[0173] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0174] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0175] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0177] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for monitoring captured data, characterized in that: include: Acquire a first time series consisting of at least one group of snapshot data within a set time range; wherein each group of snapshot data includes the number of at least one target object; Based on part or all of the first time series, a target exponential smoothing model is applied to perform prediction to obtain a second time series consisting of the prediction data of each of the at least one target object within the set time range; wherein the target time type indicated by the training samples used to train the target exponential smoothing model is consistent with the set time range; the target time type is different time periods on weekdays, different time periods on holidays, or different time periods on weekends; the parameters of the target exponential smoothing model include a seasonal index and a smoothing coefficient; Calculating a target variance according to the first time series and the second time series; wherein the target variance is used to describe the degree of proximity between the first time series and the second time series; If the first time series and the second time series do not satisfy the variance condition of the target variance, or do not satisfy the target threshold, it is determined that the captured data in the first time series is abnormal; wherein the target threshold is determined according to the change rule of the captured data in the first time series.

2. The method according to claim 1, characterized in that The forecast data for each moment within the set time range is obtained in the following manner: For a first moment, residual data at the first moment is calculated according to the periodic data at the first moment and the captured data at the first moment; wherein the periodic data at the first moment is data used to describe the periodic variation law of the captured data at the first moment; Performing a sliding average process on the residual data to obtain residual data at a second moment; The predicted data at the second moment is calculated by applying the residual data at the second moment and the periodic data at the second moment.

3. The method according to claim 1, characterized in that The method further comprises: Acquire training samples; wherein the training samples are historical snapshot sequences composed of snapshot data within a historical time range, and the historical time range indicates the target time type; Inputting the training samples into an initial exponential smoothing model for training; Until the training end condition is met, the target exponential smoothing model is obtained.

4. The method according to claim 1, characterized in that: The method further includes determining that the first time series and the second time series do not satisfy the target variance condition by: Calculating an average value of each captured data included in the first time series; Calculate a first sum of first differences between each of a first number of predicted data in the second time series and the average value; Calculate a second sum of second differences between each of the second number of predicted data in the second time series and the average value; If the difference between the first sum and the second sum is greater than the product of the target variance and the set coefficient, it is determined that the first time series and the second time series do not meet the target variance condition.

5. The method according to claim 1, characterized in that The method further includes determining the target threshold by: Calculate the change rates of two groups of captured data within a certain time interval respectively; The change rate that satisfies the set change rate condition is determined to be the target threshold.

6. The method according to claim 1, characterized in that The method further comprises: Determining target scene information of the at least one set of captured data; Determining a target data collection method for the target scene information based on a correspondence between a preset scene and a data collection method; The target acquisition method is applied to obtain the at least one set of snapshot information.

7. The method according to any one of claims 1 to 6, characterized in that: After determining that the captured data in the first time series is abnormal, the method further includes: It is determined that a source capture device of the capture data in the first time series is abnormal.

8. An electronic device, characterized in that: including a processor and a memory; The memory is configured to execute: Store at least one set of snapshot data within a set time range The processor is configured to execute: Acquire a first time series consisting of at least one group of snapshot data within the set time range; wherein each group of snapshot data includes the number of at least one target object; Based on part or all of the first time series, a target exponential smoothing model is applied to perform prediction to obtain a second time series consisting of the prediction data of each of the at least one target object within the set time range; wherein the target time type indicated by the training samples used to train the target exponential smoothing model is consistent with the set time range; the target time type is different time periods on weekdays, different time periods on holidays, or different time periods on weekends; the parameters of the target exponential smoothing model include a seasonal index and a smoothing coefficient; Calculating a target variance according to the first time series and the second time series; wherein the target variance is used to describe the degree of proximity between the first time series and the second time series; If the first time series and the second time series do not satisfy the variance condition of the target variance, or do not satisfy the target threshold, it is determined that the captured data in the first time series is abnormal; wherein the target threshold is determined according to the change rule of the captured data in the first time series.

9. The electronic device according to claim 8, characterized in that: The processor is further configured to execute, obtaining the prediction data at each moment within the set time range in the following manner: For a first moment, residual data at the first moment is calculated according to the periodic data at the first moment and the captured data at the first moment; wherein the periodic data at the first moment is data used to describe the periodic variation law of the captured data at the first moment; Performing a sliding average process on the residual data to obtain residual data at a second moment; The predicted data at the second moment is calculated by applying the residual data at the second moment and the periodic data at the second moment.

10. The electronic device according to claim 8, characterized in that: The processor is further configured to perform: Acquire training samples; wherein the training samples are historical snapshot sequences composed of snapshot data within a historical time range, and the historical time range indicates the target time type; Inputting the training samples into an initial exponential smoothing model for training; Until the training end condition is met, the target exponential smoothing model is obtained.

Images