A data processing method, apparatus, electronic device, and storage medium
By combining anomaly detection models and data prediction models, the problem of insufficient adaptability of detection thresholds in network environments is solved, achieving highly accurate anomaly detection of network feature operation data, reducing false detections and missed detections, and improving detection efficiency.
Patent Information
- Application Number
- CN202510637968.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, the detection thresholds for network environments cannot adapt to complex changes, resulting in low accuracy in detecting abnormal network feature operation data and a high risk of missed detections or misjudgments.
This paper adopts a method that combines anomaly detection model and data prediction model. By acquiring measured operation data of network features, the anomaly detection model is used to identify candidate anomaly data, and the data prediction model is used for verification. The data prediction model captures periodic features, reduces the interference of periodic fluctuations, and improves the detection accuracy.
It improves the accuracy of anomaly detection in network characteristic operation data, reduces false positives and false negatives, and can more accurately identify abnormal data beyond periodic fluctuations.
Smart Images

Figure CN123123221A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of data processing technology, and in particular relates to a data processing method, apparatus, electronic device and storage medium. Background Technology
[0002] With the rapid development of network technology, the scale of network environments has become increasingly complex. Therefore, in order to ensure the stable operation of the network environment, it is possible to detect whether the network characteristic operation data in the network environment is abnormal by using preset detection conditions. The network environment includes communication networks and communication equipment, and the network characteristics include the network characteristics of the communication network and the network characteristics of the communication equipment.
[0003] In related technologies, the preset detection conditions are usually preset detection thresholds. However, the detection thresholds are usually fixed and cannot adapt to the complex changes in the network environment. They are prone to missing or misjudging some abnormal operating data, which reduces the accuracy of detecting whether the operating data of network features in the network environment is abnormal. Summary of the Invention
[0004] This application provides a data processing method, apparatus, electronic device, and storage medium that can solve the problem of low accuracy in detecting whether the operational data of network characteristics in a network environment is abnormal.
[0005] In a first aspect, embodiments of this application provide a data processing method, which includes:
[0006] Obtain the first measured operational data of the network characteristics within the first time period;
[0007] The first measured running data is input into the anomaly detection model and the data prediction model respectively. The anomaly detection model is used to determine the candidate abnormal running data in the first measured running data.
[0008] The data prediction model is used to determine the second predicted running data of the network characteristics in the second time period;
[0009] Based on the second predicted running data and the second measured running data of network features in the second time period, the candidate abnormal running data in the first measured running data are verified to obtain the detection results of the running data of network features.
[0010] Secondly, embodiments of this application provide a data processing apparatus, which includes:
[0011] The acquisition module is used to acquire the first measured running data of network features within the first time period;
[0012] The determination module is used to input the first measured running data into the anomaly detection model and the data prediction model respectively, and to determine the candidate abnormal running data in the first measured running data through the anomaly detection model;
[0013] The determination module is also used to determine the second predictive running data of network characteristics in the second time period through the data prediction model;
[0014] The verification module is used to verify the candidate abnormal running data in the first measured running data based on the second predicted running data and the second measured running data of network features in the second time period, so as to obtain the detection result of the running data of network features.
[0015] Thirdly, embodiments of this application provide a computer device, the computer device including: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the data processing method as described in any of the first aspects.
[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the data processing method as described in any of the first aspects.
[0017] Fifthly, embodiments of this application provide a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the data processing method as described in any of the first aspects.
[0018] The data processing method of this application embodiment can acquire the first measured running data of network features within a first time period, and input the first measured running data into an anomaly detection model and a data prediction model respectively. Through the anomaly detection model, candidate abnormal running data in the first measured running data is determined. Then, through the data prediction model, the second predicted running data of network features within a second time period is determined. Based on the second predicted running data and the second measured running data of network features within the second time period, the candidate abnormal running data in the first measured running data is verified to obtain the detection result of the running data of network features.
[0019] Therefore, the above data processing method first identifies candidate abnormal operating data in the first measured operating data using an anomaly detection model. Then, based on the predicted operating data determined by the data prediction model, the candidate abnormal operating data identified by the anomaly detection model is verified. This approach reduces false positives or false negatives in the anomaly detection model, improving the accuracy of detecting whether there is abnormal data in the operating data of network features. Furthermore, since the data prediction model can effectively capture the periodic characteristics of the first measured operating data, it can identify abnormal operating data beyond periodic fluctuations during the verification of candidate abnormal operating data. This reduces the interference of the periodic characteristics of the network operating status data on the detection of abnormal data, further improving the accuracy of detecting whether there is abnormal data in the operating data of network features. Attached Figure Description
[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application is shown;
[0022] Figure 2A This is a schematic diagram of a first measured operating data provided in the embodiments of this application;
[0023] Figure 2B This is a schematic diagram of the first measured running data after data aggregation and statistics provided in the embodiments of this application;
[0024] Figure 3 This application provides a schematic diagram of a data processing flow.
[0025] Figure 4 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;
[0026] Figure 5 This is a schematic diagram of the hardware structure of a data processing device provided in an embodiment of this application. Detailed Implementation
[0027] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0028] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.
[0029] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.
[0030] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.
[0031] In related technologies, static detection thresholds can be used to detect anomalies in the operational data of network features within a network environment. However, due to the real-time changes in the network environment, static detection thresholds cannot adapt well to these changes, easily missing or misjudging some abnormal operational data, resulting in low accuracy in detecting anomalies in the operational data of network features. Alternatively, dynamic detection thresholds can be used to detect anomalies in the operational data of network features. However, when the periodic fluctuations in operational data are large, dynamic thresholds may mistake these fluctuations for normal operational data, thus ignoring potential anomalies and leading to low accuracy in detecting anomalies in the operational data of network features.
[0032] To address the problems in the aforementioned related technologies, embodiments of this application provide a data processing method, apparatus, electronic device, storage medium, and program product. The data processing method provided in this application embodiment will first be described in detail below.
[0033] Figure 1 A flowchart illustrating a data processing method according to an embodiment of this application is shown. The execution entity of this data processing method can be a server; the following description uses a server as the execution entity to illustrate the data processing method. Figure 1 As shown, the method includes:
[0034] S101, Obtain the first measured running data of network features within the first time period.
[0035] Network characteristics can refer to metrics of communication networks or equipment within a network environment, such as network throughput, latency, packet loss rate, jitter, availability, and bandwidth. Other examples include communication equipment metrics such as port utilization, temperature, CPU utilization, signal strength, signal quality, and coverage. Communication equipment can include base stations, routers, switches, and wireless access points.
[0036] For example, the server could obtain the throughput of the communication network on January 1, 2025. Alternatively, the server could obtain the signal strength of base station A on January 2, 2025.
[0037] S102, the first measured running data is input into the anomaly detection model and the data prediction model respectively. The anomaly detection model is used to determine the candidate abnormal running data in the first measured running data.
[0038] For example, the server can input the throughput of the communication network on January 1, 2025 into the anomaly detection model and the data prediction model respectively. Through the anomaly detection model, the throughput of the communication network at 13:00 on January 1, 2025 is determined as candidate abnormal operation data.
[0039] S103, using a data prediction model, determines the second predicted running data of network characteristics within the second time period.
[0040] S104. Based on the second predicted running data and the second measured running data of network features in the second time period, the candidate abnormal running data in the first measured running data is verified to obtain the detection result of the running data of network features.
[0041] Continuing with the example above, the server can use a data prediction model to predict the throughput data of the communication network on January 2, 2025. Based on the predicted throughput data and the actual throughput data for January 2, 2025, the server can then verify the throughput of the communication network at 13:00 on January 1, 2025, to determine whether there is indeed an anomaly in the throughput at that time, thus obtaining a detection result indicating whether the throughput is abnormal.
[0042] For example, the server can use the predicted throughput data of the communication network on December 25, 2024, and the actual throughput data of the communication network on December 25, 2024, to check the throughput of the communication network at 13:00 on January 1, 2025, in order to determine whether there is really an anomaly in the throughput of the communication network at 13:00 on January 1, 2025, and obtain the detection result of whether the throughput is abnormal.
[0043] Therefore, the above data processing method first identifies candidate abnormal operating data in the first measured operating data using an anomaly detection model. Then, based on the predicted operating data determined by the data prediction model, the candidate abnormal operating data identified by the anomaly detection model is verified. This approach reduces false positives or false negatives in the anomaly detection model, improving the accuracy of detecting whether there is abnormal data in the operating data of network features. Furthermore, since the data prediction model can effectively capture the periodic characteristics of the first measured operating data, it can identify abnormal operating data beyond periodic fluctuations during the verification of candidate abnormal operating data. This reduces the interference of the periodic characteristics of the network operating status data on the detection of abnormal data, further improving the accuracy of detecting whether there is abnormal data in the operating data of network features.
[0044] The following is a detailed explanation of S101 to S104, as shown below.
[0045] Regarding S101, network characteristics can refer to indicators of the network environment. Specifically, network characteristics can be divided into two categories: one is the indicators of the communication network in the network environment, and the other is the indicators of the communication equipment in the network environment. Communication equipment can include base stations, routers, switches, wireless access points, etc. For example, network characteristics can include the throughput, latency, packet loss rate, jitter, availability, and bandwidth of the communication network. Another example is the port utilization, temperature, CPU utilization, signal strength, signal quality, and coverage of communication equipment.
[0046] Regarding S102, the server can input the first measured operational data into the anomaly detection model and the data prediction model respectively. The anomaly detection model then identifies candidate abnormal operational data within the first measured operational data. The data prediction model can be an ARIMA model, LSTM model, etc., while the anomaly detection model can be a PELT model, LOF model, etc. Candidate abnormal operational data are the abnormal operational data detected by the anomaly detection model from the first measured data within the first time period based on network characteristics. For example, if the network characteristic is the throughput of the communication network, and the first time period is January 1, 2025, the detected abnormal operational data could be the throughput at 12:00 on January 1, 2025, or the throughput at 12:30 on January 1, 2025.
[0047] Regarding S103, in some embodiments of this application, the server can determine the second predicted running data of network features within a second time period through a data prediction model, so as to verify the candidate abnormal running data based on the second predicted running data in the future. The second predicted running data can be the running data of network features before the first time period or the running data of network features after the first time period.
[0048] Based on this, S103 can include S1031 to S1033, as follows.
[0049] S1031, using a data prediction model, obtain the second prediction running data of network features in the second time period from the network feature database, wherein the first time period occurs later than the second time period.
[0050] In some embodiments of this application, if the second predicted running data is the running data of network features prior to the first time period, then when obtaining the second predicted running data, the second predicted running data of the network features within the second time period can be obtained from the network feature database through a data prediction model. The network feature database can be pre-constructed and may include the predicted running data of each network feature determined by the data prediction model. The server can query the network feature database to obtain the second predicted running data of the network features within the second time period corresponding to the first measured running data.
[0051] S1032, or, through a data prediction model, feature extraction is performed on the preprocessed first measured running data to obtain the temporal features of the preprocessed first measured running data within a first time period, wherein the preprocessing includes at least one of the following: data peak shaping, data aggregation and statistics, and the temporal features are used to characterize the changing trend of the preprocessed first measured running data within a first time period.
[0052] In some embodiments of this application, if the second predicted running data is the running data of network features after the first time period, then when obtaining the second predicted running data, a data prediction model can be directly used to determine the second predicted running data based on the first measured running data. Specifically, in order to reduce the impact of noise in the first measured running data and facilitate feature extraction by the data prediction model, the server can first preprocess the first measured running data, wherein the preprocessing may include at least one of data peak smoothing and data aggregation statistics.
[0053] One approach is to use peak smoothing to adjust extreme peak values in the distribution of the first measured operational data, thereby reducing the interference of extreme peak values on the prediction process of the data prediction model. For example, peak smoothing can be performed using the 3θ principle. Specifically, the mean and standard deviation of the first measured operational data within the first time period can be calculated first. The mean is the average level of each first measured operational data point, and the standard deviation measures the dispersion of each first measured operational data point. Then, according to the 3θ principle, a threshold of μ±3σ can be set. That is, the lower threshold is μ-3σ and the upper threshold is μ+3σ. For each data point in each first measured operational data point, data points exceeding the upper threshold are replaced with the upper threshold μ+3σ, and data points below the lower threshold are replaced with the lower threshold μ-3σ.
[0054] In addition, data aggregation and statistics is also a preprocessing method. In this embodiment, aggregation and statistics can be performed according to region, for example, aggregation can be performed according to the province or city to which each region belongs.
[0055] For example, to reduce the impact of random fluctuations in the first measured operational data on the data prediction model, the first measured operational data can be statistically analyzed at certain time intervals, as follows: Figure 2A and Figure 2B As shown, Figure 2A This is a schematic diagram of measured operating data provided in an embodiment of this application. Figure 2B This is a schematic diagram of measured operational data after data aggregation and statistics provided in an embodiment of this application.
[0056] Depend on Figure 2A It can be seen that multiple first-measurement operation data can be collected every minute, by... Figure 2B It can be seen that, in order to reduce the impact of random fluctuations in the first measured running data on the data prediction model, the first measured running data can be aggregated and statistically analyzed every five minutes. By taking the average value of the first measured running data within each five-minute period as the corresponding measured running data for that time period, it can be ensured that the statistical data is smoother and more representative.
[0057] For example, the average throughput of the communication network every five minutes on January 1, 2025 can be used as a statistical measure to ensure that the throughput data is smoother and more representative.
[0058] After the first measured running data is preprocessed, the preprocessed first measured running data can be input into the data prediction model. The data prediction model can then extract features from the preprocessed first measured running data to obtain the temporal features of the preprocessed first measured running data. The temporal features are used to characterize the changing trend of the preprocessed first measured running data within the first time period.
[0059] S1033, based on the temporal characteristics, the preprocessed first measured running data, and the error term, determine the second predicted running data of the network characteristics in the second time period, wherein the first time period occurs earlier than the second time period, and the error term includes the error of each two consecutive predicted running data.
[0060] Then, the server can use a pre-built data prediction model to predict the second predicted running data of the network features in the second time period. Taking the ARIMA model as an example, the data prediction model can recursively derive the second predicted running data for the second time period based on the preprocessed first measured running data, the error term of each adjacent prediction running data in the previous prediction process, and the temporal characteristics of the captured first measured running data. The error term refers to the difference between the running data predicted by the data prediction model and the actual running data in each prediction process. In the process of analyzing the preprocessed first measured running data, the random fluctuation part that cannot be explained by the data prediction model is also called the white noise term.
[0061] The specific processing procedure of the ARIMA model for the first measured running data is as follows: Formula (1):
[0062]
[0063] In formula (1), w t w is used to represent the value of the sequence of second-prediction running data after d-th order differencing at time t. t-1 The value of the sequence at time t-1 is used to represent the value of the sequence, where i is a count variable and μ represents the mean of the sequence. θ is used to represent the number of autoregressive terms. j Used to represent the number of terms in the moving average, ∈ t-j The white noise error term used to represent time tj, ∈ t The white noise error term at time t is used to represent time t, p is used to represent the autoregression order, and q is used to represent the moving average order.
[0064] Of course, in some embodiments of this application, the content of pre-constructing a data prediction model is also included. Taking the ARIMA model as an example, the sample running data can be first processed to reduce the impact of noise in the sample running data. The sample running data can include the running data of each network feature in the third time period. The data peak reduction process has been explained in the foregoing content and will not be repeated here. Then, the sample running data can be subjected to the ADF test. The ADF test can determine whether the sample running data contains a unit root, that is, whether the sample running data is stationary. If it is determined that the sample running data is not stationary, the sample running data can be further processed by difference to make it stationary. The first-order difference processing can be specifically performed as follows (2):
[0065] y t ′ =y t -y t-1 Formula (2)
[0066] In formula (2), y t ′ Used to represent data after first-order difference processing, y t y is used to represent the sample running data at time t. t-1 This is used to represent the sample running data at time t-1. The subsequent second-order difference processing is similar to or the same as the first-order difference processing, and will not be explained again here.
[0067] After each difference process, the ADF test needs to be performed on the sample data after the difference process until the ADF test shows that the current data is stationary. Then the order of the difference process can be used as the difference order of the ARIMA model.
[0068] The number of moving average terms in an ARIMA model can be determined using an Autocorrelation Factor (ACF) plot. An ACF plot shows the correlation between sample data and its lag order. The autocorrelation coefficients for different lag orders can be calculated using specialized statistical software or correlation functions in programming languages, and an ACF plot is created based on these autocorrelation coefficients for multiple lag orders. After generating the ACF plot, the periodicity and stationarity of the sample data can be analyzed. Furthermore, the cutoff point in the ACF plot can be identified. A cutoff point refers to the ACF plot rapidly approaching zero after a certain lag order; this lag order can then be determined as the number of moving average terms in the ARIMA model.
[0069] The number of autoregressive terms in an ARIMA model can be determined using a Partial Autocorrelation Calculation (PACF) plot. The PACF plot measures the correlation between data points separated by a specific lag order, after controlling for the influence of intermediate sample data. Simply put, it represents the direct correlation between two data points after eliminating the influence of intermediate terms. In a PACF plot, the horizontal axis represents the lag order, and the vertical axis represents the partial autocorrelation coefficient corresponding to each lag order. The partial autocorrelation coefficients for different lag orders can be calculated using specialized statistical software or partial autocorrelation functions in programming languages, and the PACF plot can be drawn based on these coefficients. After generating the PACF plot, a cutoff point can be found. Cutoff occurs when the PACF plot rapidly approaches zero after a certain lag order; this lag order can then be identified as the number of autoregressive terms in the ARIMA model.
[0070] Therefore, in this embodiment, in addition to verifying candidate abnormal data based on measured operational data of network characteristics in future time periods, candidate abnormal data can also be verified based on measured operational data of network characteristics in past time periods. Furthermore, when determining the corresponding predicted operational data, the periodicity and trend characteristics of the measured operational data can be captured by the data prediction model. This allows for full consideration of the periodicity and trend characteristics of the measured operational data, so that when verifying candidate abnormal data in the future, abnormal operational data beyond periodic fluctuations can be identified. This reduces the interference of the periodicity of network operational status data on the detection of abnormal data and improves the accuracy of detecting whether there is abnormal data in the operational data of network characteristics.
[0071] Regarding S104, in some embodiments of this application, after determining the second predicted running data, the candidate abnormal running data in the first measured running data can be verified based on the second predicted running data and the second measured running data of the network features in the second time period, thereby obtaining the detection result of the running data of the network features.
[0072] Based on this, S104 may include S1041 to S1045, as follows.
[0073] S1041, Based on the second predicted operating data and the second measured operating data, determine the first residual data sequence, which is used to characterize the degree of deviation between the second predicted operating data and the second measured operating data.
[0074] In some embodiments of this application, the server can use a residual judgment method to verify candidate abnormal running data in the first measured running data. Specifically, the server can perform residual calculations on the second predicted running data and the second measured running data to determine a first residual data sequence. This first residual data sequence includes the residual between each predicted running data in the second predicted running data and the corresponding measured running data in the second measured running data. This first residual data sequence can characterize the degree of offset between the second predicted running data and the second measured running data.
[0075] S1042, using an autocorrelation algorithm, determine the linear relationship between the first measured operating data and the second measured operating data based on the first measured operating data and the second measured operating data;
[0076] S1043, Based on the linear relationship, determine the mapping time corresponding to the abnormal time of the candidate abnormal running data;
[0077] Since the data prediction model can capture the periodic characteristics of the running data very well, in order to verify the candidate abnormal running data in the first time period, the candidate abnormal running data can be mapped to the second measured running data in the second time period based on the captured periodic characteristics of the running data. Then, based on the mapped measured running data and its corresponding predicted running data, the judgment residual method is used to verify the candidate abnormal running data.
[0078] Therefore, the server can use an autocorrelation algorithm to determine the linear relationship between the first and second measured operating data. This linear relationship characterizes the degree of correlation between the two data points, such as periodicity, trends in data change, and volatility. For example, the autocorrelation algorithm could be the Pearson correlation coefficient algorithm, the fast Fourier transform correlation algorithm, etc. Of course, the aforementioned autocorrelation algorithm could also be the one used in the ARIMA model.
[0079] After determining the linear relationship between the first and second measured operating data, the mapping time corresponding to the abnormal time of the candidate abnormal operating data can be determined based on this linear relationship. This allows the candidate abnormal operating data to be verified by calculating the residual based on the measured and predicted operating data using the mapping time.
[0080] S1044, Based on the predicted running data at the mapping time in the second predicted running data and the measured running data at the mapping time in the second measured running data, determine the second residual data. The second residual data is used to characterize the degree of deviation between the predicted running data at the mapping time and the measured running data at the mapping time.
[0081] S1045, Based on the first residual data sequence and the second residual data, the candidate abnormal running data is verified to obtain the detection results of the running data of network features.
[0082] The server can determine second residual data based on the predicted operating data and the measured operating data at the mapped time. This second residual data characterizes the degree of deviation between the predicted and measured operating data at the mapped time; the larger the second residual data value, the greater the deviation. Then, based on the aforementioned determined first and second residual data sequences, candidate abnormal operating data can be verified to determine whether the candidate abnormal operating data is truly abnormal. This yields the detection results of network characteristic operating data, indicating whether the anomaly detection of the candidate abnormal operating data is correct, whether there is truly abnormal operating data in the operating data of the first time period, and the time point or time period in which the truly abnormal operating data is located. This allows for source tracing analysis based on the detection results and specific business conditions, locating the specific communication device where the anomaly occurred, and initiating a fault alarm. This enables maintenance personnel to perform targeted maintenance on the abnormal communication device upon receiving the fault alarm.
[0083] Therefore, in this embodiment of the application, the difference between the current data point and the other data points can be quickly determined based on the second residual data and the first residual data sequence, thereby enabling rapid verification of candidate abnormal running data and improving the efficiency of detecting whether there is abnormal data in the running data of network features.
[0084] Regarding S1042, in some embodiments of this application, the server can determine the linear relationship between the first measured operating data and the second measured operating data based on the first measured operating data and the second measured operating data using an autocorrelation algorithm.
[0085] Based on this, S1042 may include S10421 to S10422, as follows.
[0086] The first measured operational data includes at least two measured operational data points, and the second measured operational data includes at least two measured operational data points;
[0087] S10421, using an autocorrelation algorithm, based on the first measured operating data and the second measured operating data, determine the first autocorrelation information between every two measured operating data in the first measured operating data, the second autocorrelation information between every two measured operating data in the second measured operating data, and the third autocorrelation information between the first measured operating data and the second measured operating data;
[0088] S10422, Based on the first autorelated information, the second autorelated information, and the third autorelated information, determine the linear relationship between the first measured operating data and the second measured operating data.
[0089] In some embodiments of this application, the server can determine the linear relationship between the first measured operating data and the second measured operating data based on the first measured operating data and the second measured operating data. Since both the first measured operating data and the second measured operating data often include multiple operating data, in determining the linear relationship between the first measured operating data and the second measured operating data, in addition to the autocorrelation information between the first measured operating data and the second measured operating data, the autocorrelation information between each measured operating data in the first measured operating data and the autocorrelation information between each measured operating data in the second measured operating data can also be referenced.
[0090] Specifically, the server can first use an autocorrelation algorithm to determine the first autocorrelation information between every two measured data points in the first measured data set, based on the first measured data set. Then, using the same autocorrelation algorithm, it can determine the second autocorrelation information between every two measured data points in the second measured data set. Finally, using the same autocorrelation algorithm, it can determine the third autocorrelation information between the first and second measured data sets.
[0091] Furthermore, the linear relationship between the first and second measured operating data can be determined based on the first, second, and third autorelated information. For example, principal component analysis can be used to analyze the first, second, and third autorelated information to determine the linear relationship between the first and second measured operating data. Alternatively, the linear relationship can be determined by calculating the correlation coefficient. For every two autorelated information pairs, the correlation coefficient can be calculated, which quantifies the strength of the linear relationship between them. Therefore, the linear relationship between the first and second measured operating data can be determined based on the correlation coefficient between every two autorelated information pairs.
[0092] Therefore, in this embodiment of the application, the linear relationship between the first measured running data and the second measured running data can be determined based on the autocorrelation information between each measured data within the first measured running data, the autocorrelation information between each measured data within the second measured running data, and the autocorrelation information between the first measured running data and the second measured running data. This allows for the full capture of the linear relationship between the first measured running data and the second measured running data, ensuring the accuracy of the mapping time subsequently determined based on the linear relationship.
[0093] Regarding S1043, in some embodiments of this application, the mapping time corresponding to the abnormal time of the candidate abnormal running data can be determined based on a linear relationship.
[0094] Based on this, S1043 can include S10431 to S10433, as follows.
[0095] S10431, Determine the mapping time window based on the linear relationship, wherein the duration of the mapping time window is greater than or equal to the abnormal time period including the abnormal time of the candidate abnormal running data;
[0096] In some embodiments of this application, the server can determine the mapping time window based on a linear relationship. Specifically, the mapping time window can be selected based on the changing trend information of the measured operating data characterized by the linear relationship. Specifically, the mapping time window can be based on the sudden drop trend near the candidate abnormal operating data, with the corresponding time period as the mapping time window. For example, a time period starting from the abnormal time in which the abnormal operating data occurs can be used as the mapping time window. Another example is a time period ending at the abnormal time in which the abnormal operating data occurs. Yet another example is a time period centered on the abnormal time in which the abnormal operating data occurs.
[0097] Furthermore, the duration of the mapping time window is related to the abnormal time of the candidate abnormal operation data. The duration of the mapping time window is greater than or equal to the abnormal time period that includes the abnormal time of the candidate abnormal operation data. For example, if the abnormal operation data in the first time period is the throughput at 15:00 on January 1, 2025 and the throughput at 15:05 on January 1, 2025, then the duration of the mapping time window can be ten minutes, and the trend information of the measured operation data corresponding to the mapping time window includes the trend information of the measured operation data from 15:00 on January 1, 2025 to 15:05 on January 1, 2025.
[0098] S10432, taking the start time of the second time period as the starting time of the translation, the mapping time window is moved to the end time of the second time period, and the change trend information of the second measured running data within the mapping time window during each movement is matched with the linear relationship;
[0099] S10433, if the trend information of the second measured running data within the mapping time window is successfully matched with the linear relationship, the center time point of the time period framed by the current mapping time window is determined as the mapping time corresponding to the abnormal time of the candidate abnormal running data.
[0100] After determining the mapping time window, the starting time point of the second time period can be used as the starting time point for translation, and the mapping time window can be moved along the direction of the ending time point of the second time period. For each movement, the trend information of the second measured running data at the moved position is matched with the linear relationship. Specifically, the linear relationship of the measured running data within the first time period corresponding to the mapping time window can be matched with the trend information of the second measured running data at the moved position.
[0101] If the trend information of the second measured operating data within the mapped time window successfully matches the aforementioned linear relationship, the center time point of the time period defined by the current mapped time window is determined as the mapped time corresponding to the abnormal time of the candidate abnormal operating data. For example, if the trend information of the second measured operating data within the mapped time window successfully matches the aforementioned linear relationship, and the time period defined by the current mapped time window is from 12:00 to 12:04 on January 1, 2025, then 12:02 on January 1, 2025 can be determined as the mapped time corresponding to the abnormal time of the candidate abnormal operating data.
[0102] In some embodiments of this application, when the trend information of the second measured running data within the mapping time window is successfully matched with the aforementioned linear relationship, in order to improve the accuracy of the mapping, in addition to directly determining the center time point of the time period framed by the current mapping time window as the mapping time corresponding to the abnormal time of the candidate abnormal running data, it is also necessary to refer to the specific abnormal time of the abnormal running data to determine the corresponding mapping time.
[0103] Specifically, if the abnormal time of the abnormal operation data is at the start of the time period corresponding to the mapped time window within the first time period, then the start of the time period defined by the current mapped time window can be determined as the mapped time corresponding to the abnormal time of the candidate abnormal operation data. For example, if the trend information of the second measured operation data within the mapped time window successfully matches the aforementioned linear relationship, and the time period defined by the current mapped time window is from 12:00 to 12:04 on January 1, 2025, then 12:00 on January 1, 2025 can be determined as the mapped time corresponding to the abnormal time of the candidate abnormal operation data.
[0104] Furthermore, if the anomalous time of the abnormal operation data falls at the end of the time period corresponding to the mapped time window within the first time period, then the end of the time period defined by the current mapped time window can be determined as the mapped time corresponding to the anomalous time of the candidate abnormal operation data. For example, if the trend information of the second measured operation data within the mapped time window successfully matches the aforementioned linear relationship, and the time period defined by the current mapped time window is from 12:00 to 12:04 on January 1, 2025, then 12:04 on January 1, 2025 can be determined as the mapped time corresponding to the anomalous time of the candidate abnormal operation data.
[0105] Therefore, in this embodiment of the application, the linear relationship of the measured running data can be accurately captured. Then, by using a moving mapping time window, the change trend information that matches the linear relationship is filtered in the second measured running data, and the time corresponding to the matched change trend information is determined as the mapping time, thereby improving the accuracy of determining the mapping time.
[0106] In some embodiments of this application, S1045 is involved, in which the server can verify the candidate abnormal running data according to the first residual data sequence and the second residual data, so as to verify whether the candidate abnormal running data is truly abnormal running data, and thus, based on the verification result, the detection result of the running data of network characteristics can be obtained.
[0107] Based on this, S1045 may include S10451 to S10455, as follows.
[0108] S10451, compare the standard deviation data of the first residual data sequence with the second residual data;
[0109] In some embodiments of this application, since the second residual data is determined by the predicted running data at the mapping time and the measured running data at the mapping time in the second measured running data, the degree of deviation of the measured running data at the mapping time from the overall measured running data can be determined by comparing the second residual data corresponding to the measured running data at the mapping time and the first residual data sequence corresponding to the second measured running data in the second time period. This reflects the degree of deviation of the candidate abnormal running data at the abnormal time corresponding to the mapping time from the overall measured running data in terms of trend, thereby realizing the verification of the candidate abnormal running data.
[0110] S10452, when the second residual data is greater than the standard deviation of the first residual data sequence, the verification result used to characterize the anomaly is determined as an abnormal result of the measured running data at the mapping time;
[0111] S10453, when the second residual data is less than or equal to the standard deviation of the first residual data sequence, the verification result used to characterize the absence of anomalies is determined as an abnormal result of the measured running data at the mapping time.
[0112] S10454, In cases where abnormal results are used to characterize the verification results as abnormal, the detection results of the running data used to characterize the correct detection results as network features are used.
[0113] S10455, when the abnormal result is used to characterize the verification result as not having any abnormality, the detection result used to characterize the error is determined as the detection result of the running data of the network feature.
[0114] When the second residual data is greater than the standard deviation of the first residual data sequence, the verification result used to characterize the anomaly is determined to be an anomaly result of the measured running data at the mapping time. That is, the measured running data at the mapping time deviates from the trend of the overall measured running data, and the measured running data at the mapping time is anomaly data. When the anomaly result is used to characterize the verification result as anomaly, the detection result used to characterize the correct detection result is determined to be the detection result of the running data of the network feature. That is, the anomaly detection result of the candidate anomaly running data is determined to be correct, and the candidate anomaly running data is truly anomaly data.
[0115] When the second residual data is less than or equal to the standard deviation of the first residual data sequence, the verification result used to characterize the absence of anomalies is determined as an anomaly result of the measured running data at the mapping time. That is, the measured running data at the mapping time does not deviate from the trend of the overall measured running data, and the measured running data at the mapping time is normal data. When the anomaly result is used to characterize the verification result as the absence of anomalies, the detection result used to characterize the error is determined as the detection result of the running data of the network features. That is, the anomaly detection result of the candidate anomaly running data is determined, and the candidate anomaly running data is not truly anomaly data.
[0116] Of course, in some embodiments of this application, in order to further improve the accuracy of detecting whether the running data of network features in the network environment is abnormal, multiple verification methods can be used to verify the candidate abnormal running data. For example, if the results of three consecutive verifications are consistent, the current verification result can be determined as the final detection result.
[0117] Therefore, in this embodiment of the application, the abnormal results of the second measured running data at the mapping time determined by calculating the residual can be used to verify the candidate abnormal running data, thereby improving the accuracy of detecting whether there is abnormal data in the running data of network features.
[0118] In some embodiments of this application, S1045 is involved, in which the server can verify the candidate abnormal running data according to the first residual data sequence and the second residual data, so as to verify whether the candidate abnormal running data is truly abnormal running data, and thus, based on the verification result, the detection result of the running data of network characteristics can be obtained.
[0119] Based on this, S1045 may include S10456 to S10458, as follows.
[0120] The mapping time includes at least two mapping times, and the second residual data includes at least two second residual data.
[0121] S10456, compare the standard deviation data of the first residual data sequence with each of the second residual data in at least two second residual data sequences;
[0122] S10457, when each second residual data is greater than the standard deviation data of the first residual data sequence, the verification result used to characterize the anomaly is determined as the anomaly result of the measured running data of the mapping time.
[0123] S10459, when abnormal results are used to characterize the verification results as abnormal, the detection results of the running data used to characterize the correct detection results are determined as network features.
[0124] In some embodiments of this application, there may be multiple candidate abnormal running data within the first time period. Correspondingly, there may also be multiple mapping times. Therefore, it is necessary to determine the detection result based on the verification result corresponding to each mapping time. Specifically, the standard deviation of the first residual data sequence is compared with each of at least two second residual data. If each second residual data is greater than the standard deviation of the first residual data sequence, the verification result used to characterize the abnormality is determined as the abnormal result of the measured running data at the mapping time. If the abnormal result is used to characterize the verification result as abnormal, the detection result used to characterize the correctness is determined as the detection result of the running data for network features. That is, the result of the abnormal detection of the candidate abnormal running data is determined to be correct, and the candidate abnormal running data is truly abnormal data.
[0125] If, among the second residual data corresponding to the measured running data at multiple mapping times, there exists a second residual data corresponding to the measured running data at any mapping time that is less than or equal to the standard deviation of the first residual data sequence, the verification result used to characterize the absence of anomalies will be determined as the abnormal result of the measured running data at the mapping time. If the abnormal result is used to characterize the verification result as the absence of anomalies, the detection result used to characterize the error will be determined as the detection result of the running data of the network features. That is, the result of the anomaly detection of the candidate abnormal running data is determined, and the candidate abnormal running data is not truly abnormal data.
[0126] Therefore, in this embodiment of the application, candidate abnormal running data can be further examined based on second measured running data of multiple mapping times, which improves the accuracy of detecting whether there is abnormal data in the running data of network features.
[0127] Regarding S102, in this embodiment of the application, candidate abnormal running data in the first measured running data can be determined by an anomaly detection model.
[0128] Based on this, S102 may include S1021 to S1025, as follows.
[0129] The first measured operational data includes measured operational data at N time points within the first time period, where N is a positive integer;
[0130] S1021, using the anomaly detection model, based on the first measured operating data, calculate the fitting loss function value of the measured operating data at each of the N time points. The fitting loss function value is used to characterize the deviation of the measured operating data at each time point from the overall trend of the first measured operating data in the first time period.
[0131] S1022, Based on the fitting loss function value and penalty factor of the measured running data at each time point, determine the cost function value corresponding to the measured running data at each time point;
[0132] In this embodiment, the anomaly detection model can be a PELT model, which detects candidate abnormal running data in the first measured running data by using a backtracking approach. Specifically, the server can accurately detect candidate abnormal running data in the first measured running data based on the cost function provided by the anomaly detection model. Specifically, it can calculate the cost function values corresponding to the measured running data at N time points within the first time period, and determine the candidate abnormal running data in the first measured running data by comparing multiple cost function values.
[0133] Specifically, the server can first calculate the fitting loss function value for the measured running data at each of the N time points based on the first measured running data. The fitting loss function value characterizes the deviation of the measured running data at each time point from the overall trend of the first measured running data within the first time period. The larger the fitting loss function value, the greater the deviation of the measured running data at the corresponding time point from the overall trend of the first measured running data within the first time period.
[0134] Then, the server can determine the cost function value corresponding to the measured running data at each time point based on the fitted loss function value and penalty factor of the measured running data at each time point. The penalty factor can be pre-set based on expert experience or the elbow rule; it is a parameter used to control model complexity or constrain specific behaviors.
[0135] The cost function can be determined using the following formula (3):
[0136] J=L+λP Formula (3)
[0137] In formula (3), J represents the cost function value, L represents the fitting loss function value, λ is the penalty factor, and P is the penalty term for model complexity or other constraints.
[0138] S1023, select the first candidate measured running data with the smallest cost function value from the measured running data at N time points, determine the first candidate measured running data as the first candidate abnormal running data, and determine the time point where the first candidate abnormal running data is located as the abnormal time point;
[0139] S1024, If the abnormal time point is not the start time point of the first time period, select the second candidate measured running data with the smallest cost function value from the measured running data before the abnormal time point to the start time point of the first time period, and determine the second candidate measured running data as the second candidate abnormal running data.
[0140] S1025, the time point where the second candidate abnormal running data is located is redefined as the abnormal time point, and the target step is executed until the abnormal time point is the start time point of the first time period. The first candidate abnormal running data and the second candidate abnormal running data are determined as candidate abnormal running data in the first measured running data. The target step includes: when the abnormal time point is not the start time point of the first time period, the second candidate measured running data with the smallest cost function value is selected from the measured running data from before the abnormal time point to the start time point of the first time period, and the second candidate measured running data is determined as the second candidate abnormal running data.
[0141] After determining the cost function values corresponding to the measured running data at N time points, the first candidate measured running data with the smallest cost function value can be selected from the measured running data at N time points. The first candidate measured running data can be identified as the first candidate abnormal running data, and the time point where the first candidate abnormal running data is located can be identified as the abnormal time point.
[0142] After identifying the first abnormal time point, backtracking can be performed, that is, further filtering can be done from the measured data that precedes the first candidate measured data in the first measured data. Specifically, if the abnormal time point is not the start time point of the first time period, the second candidate measured data with the smallest cost function value is selected from the time period before the abnormal time point to the start time point of the first time period. This second candidate abnormal data is then re-identified as the abnormal time point, and the backtracking and filtering of the measured data with the smallest cost function value is repeated until the abnormal time point is the start time point of the first time period. That is, the first measured data in the entire first time period is backtracked. The candidate abnormal data identified in the aforementioned backtracking process can then be identified as candidate abnormal data in the first measured data.
[0143] For example, the first measured running data includes four measured running data, namely [a, b, c, d, e]. If the measured running data with the smallest cost function value is d, then it is necessary to filter the measured running data with the smallest cost function value from [a, b, c]. If the measured running data with the smallest cost function value after filtering is a, then data a and data d are determined as candidate abnormal data.
[0144] Furthermore, this embodiment also includes a process for constructing an anomaly detection model. First, since the first measured running data usually exhibits trends and periodicity, a linear function can be used as the model function in the anomaly detection model to capture candidate anomaly running data from the trending and periodic first measured running data. Specifically, a PELT model can be created based on a linear L1 cost function. Then, the sample running data is input into the anomaly detection model. By adjusting parameters such as the penalty factor in the cost function, the model parameters of the anomaly detection model are fitted to optimize the performance of the data anomaly detection model. The sample running data can include the running data of each network feature in the fourth time period.
[0145] Therefore, in this embodiment of the application, candidate abnormal data can be determined first through an anomaly detection model, so that the candidate abnormal data can be verified in the subsequent process, thereby improving the accuracy of detecting whether there is abnormal data in the running data of the network features.
[0146] Regarding S1021, in this embodiment of the application, the anomaly detection model can be used to calculate the fitting loss function value of the measured running data at each of the N time points based on the first measured running data.
[0147] Based on this, S1021 may specifically include: using an anomaly detection model, and based on a fitting loss function, calculating the fitting loss function value of the measured running data at each of the N time points, wherein the fitting loss function includes a fitting loss function based on least squares, a fitting loss function based on mean and variance, or a fitting loss function based on information entropy.
[0148] In this embodiment, the server can use an anomaly detection model to calculate the fitting loss function value of the measured running data at each of the N time points based on the fitting loss function. For example, the fitting loss function value of the measured running data at each of the N time points can be calculated using a fitting loss function based on least squares, a fitting loss function based on mean and variance, or a fitting loss function based on information entropy.
[0149] For example, in a least squares-based fitting loss function, the fitting loss function value can be determined by calculating the squared error between each measured data point in the first set of measured data and the baseline measured data. As another example, in a mean-variance-based fitting loss function, the mean and variance of multiple measured data points in the first set of measured data can be calculated first, and then the fitting loss function value can be obtained based on each measured data point, its mean, and its variance.
[0150] Therefore, in this embodiment of the application, the fitting loss function value of the first measured running data can be determined based on the fitting loss function of the least squares method, the fitting loss function based on the mean and variance, or the fitting loss function based on the information entropy. Then, the candidate abnormal running data included in the first measured running data can be determined based on the fitting loss function value, so as to facilitate the subsequent verification of the candidate abnormal running data.
[0151] In S102, in this embodiment of the application, the server can input the first measured running data into the anomaly detection model and the data prediction model respectively, and determine the candidate abnormal running data in the first measured running data through the anomaly detection model.
[0152] Therefore, before S102, there are also S105 to S106, as detailed below.
[0153] The network characteristics are those within the network environment of the first region.
[0154] S105, Based on the correlation between the region and the anomaly detection model, obtain the anomaly detection model corresponding to the first region;
[0155] S106. Based on the correlation between the region and the data prediction model, obtain the data prediction model corresponding to the first region.
[0156] In this embodiment of the application, since the real-time network status of different network environments is not the same, different anomaly detection models and data prediction models can be constructed for different network environments, so that abnormal operation data can be detected in a targeted manner for different network environments.
[0157] Accordingly, after acquiring the first measured running data of the network feature within the first time period, if the network feature is a network feature within the network environment of the first region, then the anomaly detection model corresponding to the first region can be obtained based on the correlation between the region and the anomaly detection model. This allows candidate anomaly running data to be determined from the first measured running data based on the corresponding anomaly detection model. Furthermore, the data prediction model corresponding to the first region can be obtained based on the correlation between the region and the data prediction model, and the second predicted running data of the network feature within the second time period can be determined based on the data prediction model.
[0158] In practical applications, network environments in different regions can be divided based on factors such as provinces, with different data prediction and anomaly detection models corresponding to different provinces. Alternatively, network environments can be divided based on factors such as city districts, with different city districts corresponding to different data prediction and anomaly detection models. Of course, different division methods can be used to divide the network environments of different regions according to actual needs, so as to build data prediction and anomaly detection models adapted to the network environment of each region.
[0159] Therefore, in this embodiment of the application, different anomaly detection models and data prediction models can be used to process the measured operating data for different regions, thereby enhancing the applicability of the anomaly detection model and data prediction model to the measured operating data of different regions, and improving the accuracy of determining candidate abnormal operating data and the accuracy of determining the second predicted operating data.
[0160] In the aforementioned data processing method, candidate abnormal operating data in the first measured operating data can be identified first through an anomaly detection model. Then, based on the predicted operating data determined by the data prediction model, the candidate abnormal operating data identified by the anomaly detection model are verified. This approach reduces the likelihood of false positives or false negatives in the anomaly detection model, improving the accuracy of detecting whether there are abnormal data in the operating data of network features. Furthermore, since the data prediction model can effectively capture the periodic characteristics of the first measured operating data, it can identify abnormal operating data beyond periodic fluctuations during the verification of candidate abnormal operating data. This reduces the interference of the periodic characteristics of the network operating status data on anomaly detection, further improving the accuracy of detecting whether there are abnormal data in the operating data of network features.
[0161] To further illustrate the above data processing method, the overall processing flow will be described below. Figure 3 , Figure 3 This is a schematic diagram of a data processing flow provided in an embodiment of this application.
[0162] Depend on Figure 3 It can be seen that S301 obtains the first measured running data of the network characteristics in the first time period.
[0163] S302, the server can use the anomaly detection model to determine candidate abnormal running data in the first measured running data and determine the abnormal time of the candidate abnormal running data.
[0164] S303, the server can determine the mapping time corresponding to the abnormal time of the candidate abnormal running data, and determine the second residual data based on the predicted running data at the mapping time and the measured running data at the mapping time in the second predicted running data.
[0165] S304, the server can preprocess the first measured running data.
[0166] S305, the server can use the data prediction model to determine the second predicted running data of network characteristics in the second time period based on the preprocessed first measured running data.
[0167] S306, the server can determine the first residual data sequence based on the second predicted running data and the second measured running data of the second time period.
[0168] S307, the server can determine the standard deviation of the first residual data sequence.
[0169] S308, the server can determine whether the second residual data is greater than the standard deviation data.
[0170] S309, if the second residual data is greater than the standard deviation data, the server can determine the detection result of the network feature operation data as erroneous if the abnormal result of the measured operation data of the mapping time is normal.
[0171] S3010, if the second residual data is less than or equal to the standard deviation data, the server can determine the detection result of the network feature operation data as correct if the abnormal result of the measured operation data for determining the mapping time is abnormal.
[0172] In some embodiments of this application, after identifying the truly abnormal operational data among the candidate abnormal operational data, a fault alarm can be generated through a dynamic alarm algorithm execution module embedded in the FPGA logic unit. This allows maintenance personnel to perform targeted maintenance on the malfunctioning communication equipment after receiving the fault alarm. Furthermore, the algorithm corresponding to the model can be programmed onto the FPGA as an independent module, thereby improving the accuracy of identifying abnormal operational data.
[0173] like Figure 4 As shown, this application embodiment also provides a data processing apparatus 400, which includes:
[0174] The acquisition module 401 is used to acquire the first measured running data of network features within the first time period.
[0175] The determination module 402 is used to input the first measured running data into the anomaly detection model and the data prediction model respectively, and to determine the candidate abnormal running data in the first measured running data through the anomaly detection model;
[0176] The determination module 402 is also used to determine the second prediction running data of network characteristics in the second time period through the data prediction model;
[0177] The verification module 403 is used to verify the candidate abnormal running data in the first measured running data based on the second predicted running data and the second measured running data of the network features in the second time period, so as to obtain the detection result of the running data of the network features.
[0178] Therefore, the aforementioned data processing device can first identify candidate abnormal operating data in the first measured operating data using an anomaly detection model. Then, based on the predicted operating data determined by the data prediction model, the candidate abnormal operating data identified by the anomaly detection model is verified. This method reduces the likelihood of false positives or false negatives in the anomaly detection model, improving the accuracy of detecting whether abnormal data exists in the operating data of network features. Furthermore, since the data prediction model can effectively capture the periodic characteristics of the first measured operating data, it can identify abnormal operating data beyond periodic fluctuations during the verification of candidate abnormal operating data. This reduces the interference of the periodic characteristics of the network operating status data on anomaly detection, further improving the accuracy of detecting whether abnormal data exists in the operating data of network features.
[0179] In some embodiments of this application, the acquisition module 401 is further configured to acquire, through a data prediction model, second predicted running data of network features within a second time period from a network feature database, wherein the first time period occurs later than the second time period.
[0180] The determination module 402 is also used to, or, through a data prediction model, extract features from the preprocessed first measured running data to obtain the temporal features of the preprocessed first measured running data in the first time period, wherein the preprocessing includes at least one of the following: data peak shaping, data aggregation and statistics, and the temporal features are used to characterize the changing trend of the preprocessed first measured running data in the first time period.
[0181] The determination module 402 is further configured to determine the second predicted running data of the network features in the second time period based on the time series features, the preprocessed first measured running data and the error term, wherein the first time period occurs earlier than the second time period, and the error term includes the error of each two consecutive predicted running data.
[0182] In some embodiments of this application, the determining module 402 is further configured to determine a first residual data sequence based on the second predicted running data and the second measured running data, wherein the first residual data sequence is used to characterize the degree of deviation between the second predicted running data and the second measured running data;
[0183] The determination module 402 is further configured to determine the linear relationship between the first measured operating data and the second measured operating data based on the first measured operating data and the second measured operating data using an autocorrelation algorithm;
[0184] The determination module 402 is also used to determine, based on the linear relationship, the mapping time corresponding to the abnormal time where the candidate abnormal running data is located;
[0185] The determining module 402 is further configured to determine second residual data based on the predicted running data at the mapping time in the second predicted running data and the measured running data at the mapping time in the second measured running data. The second residual data is used to characterize the degree of deviation between the predicted running data at the mapping time and the measured running data at the mapping time.
[0186] The verification module 403 is also used to verify the candidate abnormal running data based on the first residual data sequence and the second residual data, and obtain the detection result of the running data of the network features.
[0187] In some embodiments of this application, the first measured operating data includes at least two measured operating data, and the second measured operating data includes at least two measured operating data;
[0188] The determination module 402 is further configured to determine, through an autocorrelation algorithm, a first autocorrelation relationship between every two measured operating data in the first measured operating data, a second autocorrelation relationship between every two measured operating data in the second measured operating data, and a third autocorrelation relationship between the first measured operating data and the second measured operating data, based on the first measured operating data and the second measured operating data;
[0189] The determination module 402 is also used to determine the linear relationship between the first measured operating data and the second measured operating data based on the first autorelated information, the second autorelated information, and the third autorelated information.
[0190] In some embodiments of this application, the determining module 402 is further configured to determine a mapping time window based on a linear relationship, wherein the duration of the mapping time window is greater than or equal to the abnormal time period including the abnormal time of the candidate abnormal running data.
[0191] The data processing device 400 also includes a translation module, which is used to move the mapped time window from the start time point of the second time period to the end time point of the second time period, and match the change trend information of the second measured running data within the mapped time window with the linear relationship during each movement.
[0192] The determination module 402 is also used to determine the center time point of the time period framed by the current mapping time window as the mapping time corresponding to the abnormal time of the candidate abnormal running data, when the trend information of the change of the second measured running data within the mapping time window is successfully matched with the linear relationship.
[0193] In some embodiments of this application, the data processing device 400 further includes a comparison module for comparing the standard deviation data of the first residual data sequence with the second residual data;
[0194] The determination module 402 is further configured to determine the verification result used to characterize the anomaly as an abnormal result of the measured running data at the mapping time when the second residual data is greater than the standard deviation data of the first residual data sequence.
[0195] The determination module 402 is further configured to determine the verification result used to characterize the absence of anomalies as an abnormal result of the measured running data at the mapping time when the second residual data is less than or equal to the standard deviation data of the first residual data sequence.
[0196] The determination module 402 is also used to determine the detection results of the running data used to characterize the correct detection results as network features when the abnormal results are used to characterize the verification results as abnormal.
[0197] The determination module 402 is also used to determine the detection result used to characterize the error as the detection result of the running data of the network feature when the abnormal result is used to characterize the verification result as not having any abnormality.
[0198] In some embodiments of this application, the mapping time includes at least two mapping times, and the second residual data includes at least two second residual data.
[0199] The comparison module is also used to compare the standard deviation data of the first residual data sequence with each of the second residual data in at least two second residual data sequences;
[0200] The determination module 402 is further configured to determine the verification result used to characterize the anomaly as the anomaly result of the measured running data of the mapping time when each second residual data is greater than the standard deviation data of the first residual data sequence.
[0201] The determination module 402 is also used to determine the detection results of the running data used to characterize the correct detection results as network features when the abnormal results are used to characterize the verification results as abnormal.
[0202] In some embodiments of this application, the first measured operating data includes measured operating data at N time points within a first time period, where N is a positive integer;
[0203] The determination module 402 is also used to calculate the fitting loss function value of the measured running data at each of the N time points based on the first measured running data through the anomaly detection model. The fitting loss function value is used to characterize the deviation of the measured running data at each time point from the overall trend of the first measured running data in the first time period.
[0204] The determination module 402 is also used to determine the cost function value corresponding to the measured running data at each time point based on the fitting loss function value and penalty factor of the measured running data at each time point;
[0205] The determination module 402 is also used to filter out the first candidate measured running data with the smallest cost function value from the measured running data at N time points, determine the first candidate measured running data as the first candidate abnormal running data, and determine the time point where the first candidate abnormal running data is located as the abnormal time point;
[0206] The determination module 402 is also used to filter out the second candidate measured running data with the smallest cost function value from the measured running data from before the abnormal time point to the start time point of the first time period when the abnormal time point is not the start time point of the first time period, and determine the second candidate measured running data as the second candidate abnormal running data.
[0207] The determination module 402 is further configured to re-determine the time point of the second candidate abnormal running data as the abnormal time point, and execute the target steps until the abnormal time point is the start time point of the first time period, and determine the first candidate abnormal running data and the second candidate abnormal running data as candidate abnormal running data in the first measured running data; wherein, the target steps include: when the abnormal time point is not the start time point of the first time period, selecting the second candidate measured running data with the smallest cost function value from the measured running data before the abnormal time point to the start time point of the first time period, and determining the second candidate measured running data as the second candidate abnormal running data.
[0208] In some embodiments of this application, the determining module 402 is further configured to calculate the fitting loss function value of the measured running data at each of the N time points based on the fitting loss function using an anomaly detection model. The fitting loss function includes a fitting loss function based on least squares, a fitting loss function based on mean and variance, or a fitting loss function based on information entropy.
[0209] In some embodiments of this application, the network features are network features in the network environment of the first region;
[0210] The acquisition module 401 is also used to acquire the anomaly detection model corresponding to the first region based on the correlation between the region and the anomaly detection model;
[0211] The acquisition module 401 is also used to acquire the data prediction model corresponding to the first region based on the correlation between the region and the data prediction model.
[0212] Figure 5 The illustration shows a schematic diagram of the hardware structure of a computer device provided in some embodiments of this application.
[0213] The computer device 500 may include a processor 501 and a memory 502 storing computer program instructions.
[0214] Specifically, the processor 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0215] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory.
[0216] In a particular embodiment, memory 502 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory 502 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the UAV detection method according to the first aspect of this application.
[0217] The processor 501 implements any of the data processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 502.
[0218] In one example, the computer device may also include a communication interface 503 and a bus 510. Wherein, as... Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.
[0219] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0220] Bus 510 includes hardware, software, or both, that couples components of a computer device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.
[0221] The computer device can execute the data processing method described in the embodiments of this application, thereby achieving the combination Figures 1 to 4 The data processing methods and apparatus described.
[0222] Furthermore, in conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data processing methods in the above embodiments. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.
[0223] Furthermore, in conjunction with the data processing methods in the above embodiments, this application embodiment can provide a computer program product for implementation. This program product is stored in a storage medium and may specifically include a computer program or instructions. When executed by a processor, the computer program or instructions implement any of the data processing methods in the above embodiments. This program product is executed by at least one processor to implement the various processes of the data processing method embodiments described above, and can achieve the same technical effects. To avoid repetition, further details are omitted here.
[0224] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0225] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0226] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0227] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.
[0228] The above description is merely a specific embodiment of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A data processing method, characterized in that, include: Obtain the first measured operational data of the network characteristics within the first time period; The first measured running data is input into the anomaly detection model and the data prediction model respectively. The anomaly detection model is used to determine the candidate abnormal running data in the first measured running data. The data prediction model is used to determine the second predicted running data of the network features within the second time period; Based on the second predicted running data and the second measured running data of the network feature within the second time period, the candidate abnormal running data in the first measured running data are verified to obtain the detection result of the running data of the network feature.
2. The method according to claim 1, characterized in that, The step of determining the second predicted running data of the network features within the second time period using the data prediction model includes: The data prediction model is used to obtain the second prediction data of the network features in the second time period from the network feature database, wherein the first time period occurs later than the second time period. Alternatively, the data prediction model can be used to extract features from the preprocessed first measured running data to obtain the temporal features of the preprocessed first measured running data within the first time period. The preprocessing includes at least one of the following: data peak shaping and data aggregation statistics. The temporal features are used to characterize the changing trend of the preprocessed first measured running data within the first time period. Based on the time-series features, the preprocessed first measured running data, and the error term, the second predicted running data of the network features within the second time period is determined, wherein the first time period occurs earlier than the second time period, and the error term includes the error of each two consecutive predicted running data.
3. The method according to claim 1, characterized in that, The step of verifying candidate abnormal operation data in the first measured operation data based on the second predicted operation data and the second measured operation data of the network feature within the second time period to obtain the detection result of the operation data of the network feature includes: Based on the second predicted operating data and the second measured operating data, a first residual data sequence is determined, which is used to characterize the degree of deviation between the second predicted operating data and the second measured operating data. The linear relationship between the first measured operating data and the second measured operating data is determined by using an autocorrelation algorithm. Based on the linear relationship, determine the mapping time corresponding to the abnormal time of the candidate abnormal running data; Based on the predicted running data at the mapping time in the second predicted running data and the measured running data at the mapping time in the second measured running data, a second residual data is determined. The second residual data is used to characterize the degree of deviation between the predicted running data at the mapping time and the measured running data at the mapping time. Based on the first residual data sequence and the second residual data, the candidate abnormal running data is verified to obtain the detection result of the running data of the network features.
4. The method according to claim 3, characterized in that, The first measured operational data includes at least two measured operational data points, and the second measured operational data includes at least two measured operational data points; the step of determining the linear relationship between the first measured operational data and the second measured operational data using an autocorrelation algorithm includes: Using an autocorrelation algorithm, based on the first measured running data and the second measured running data, the first autocorrelation information between every two measured running data in the first measured running data, the second autocorrelation information between every two measured running data in the second measured running data, and the third autocorrelation information between the first measured running data and the second measured running data are determined. Based on the first autorelated information, the second autorelated information, and the third autorelated information, a linear relationship between the first measured operating data and the second measured operating data is determined.
5. The method according to claim 3, characterized in that, The step of determining the mapping time corresponding to the abnormal time of the candidate abnormal running data based on the linear relationship includes: Based on the linear relationship, a mapping time window is determined, wherein the duration of the mapping time window is greater than or equal to the abnormal time period that includes the abnormal time of the candidate abnormal running data. Using the start time of the second time period as the starting time of the translation, the mapped time window is moved towards the end time of the second time period, and the trend information of the change of the second measured running data within the mapped time window at each movement is matched with the linear relationship. If the trend information of the second measured running data within the mapping time window is successfully matched with the linear relationship, the center time point of the time period framed by the current mapping time window is determined as the mapping time corresponding to the abnormal time of the candidate abnormal running data.
6. The method according to claim 3, characterized in that, The step of verifying the candidate abnormal running data based on the first residual data sequence and the second residual data to obtain the detection result of the running data of the network features includes: The standard deviation data of the first residual data sequence is compared with the second residual data; If the second residual data is greater than the standard deviation of the first residual data sequence, the verification result used to characterize the anomaly is determined as an abnormal result of the measured running data at the mapping time. If the second residual data is less than or equal to the standard deviation of the first residual data sequence, the verification result used to characterize the absence of anomalies is determined as an abnormal result of the measured running data at the mapping time. In cases where the abnormal result is used to characterize the verification result as abnormal, the detection result used to characterize the correct detection result is determined as the detection result of the running data of the network feature; When the abnormal result is used to characterize that the verification result does not have any abnormality, the detection result used to characterize the error is determined as the detection result of the running data of the network feature.
7. The method according to claim 3, characterized in that, The mapping time includes at least two mapping times, and the second residual data includes at least two second residual data. The step of verifying the candidate abnormal running data based on the first residual data sequence and the second residual data to determine the detection result of the running data of the network features includes: The standard deviation data of the first residual data sequence is compared with each of the at least two second residual data; If each of the second residual data is greater than the standard deviation of the first residual data sequence, the verification result used to characterize the anomaly is determined as the anomaly result of the measured running data of the mapping time. In cases where the abnormal result is used to characterize the verification result as abnormal, the detection result used to characterize the correct detection result is determined as the detection result of the running data of the network feature.
8. The method according to claim 1, characterized in that, The first measured operational data includes measured operational data at N time points within the first time period, where N is a positive integer; the step of determining candidate abnormal operational data in the first measured operational data using the anomaly detection model includes: Using the anomaly detection model, based on the first measured operational data, the fitting loss function value of the measured operational data at each of the N time points is calculated. The fitting loss function value is used to characterize the deviation of the measured operational data at each time point from the overall trend of the first measured operational data in the first time period. Based on the fitting loss function value and penalty factor of the measured running data at each time point, determine the cost function value corresponding to the measured running data at each time point; From the measured running data at the N time points, the first candidate measured running data with the smallest cost function value is selected, the first candidate measured running data is determined as the first candidate abnormal running data, and the time point where the first candidate abnormal running data is located is determined as the abnormal time point. If the abnormal time point is not the start time point of the first time period, the second candidate measured running data with the smallest cost function value is selected from the measured running data from before the abnormal time point to the start time point of the first time period, and the second candidate measured running data is determined as the second candidate abnormal running data. The time point where the second candidate abnormal running data is located is redefined as the abnormal time point, and the target steps are executed until the abnormal time point is the start time point of the first time period. The first candidate abnormal running data and the second candidate abnormal running data are determined as candidate abnormal running data in the first measured running data. The target step includes: when the abnormal time point is not the start time point of the first time period, selecting the second candidate measured running data with the smallest cost function value from the measured running data before the abnormal time point to the start time point of the first time period, and determining the second candidate measured running data as the second candidate abnormal running data.
9. The method according to claim 8, characterized in that, The step of calculating the fitting loss function value of the measured operational data at each of the N time points using the anomaly detection model, based on the first measured operational data, includes: Using the anomaly detection model, the fitting loss function value of the measured running data at each of the N time points is calculated based on the fitting loss function. The fitting loss function includes a fitting loss function based on least squares, a fitting loss function based on mean and variance, or a fitting loss function based on information entropy.
10. The method according to claim 1, characterized in that, The network characteristics are network characteristics in the network environment of the first region; Before inputting the first measured operational data into the anomaly detection model and the data prediction model respectively, the method further includes: Based on the correlation between the region and the anomaly detection model, obtain the anomaly detection model corresponding to the first region; Based on the correlation between the region and the data prediction model, obtain the data prediction model corresponding to the first region.
11. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the first measured running data of network features within the first time period; The determination module is used to input the first measured running data into the anomaly detection model and the data prediction model respectively, and determine the candidate abnormal running data in the first measured running data through the anomaly detection model; The determining module is also used to determine the second predicted running data of the network features in the second time period through the data prediction model; The verification module is used to verify the candidate abnormal running data in the first measured running data based on the second predicted running data and the second measured running data of the network feature in the second time period, so as to obtain the detection result of the running data of the network feature.
12. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the data processing method as described in any one of claims 1-10.
13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1-10.
14. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the data processing method as described in any one of claims 1-10.
