A method for efficient data processing in communication networks

Abstract

This invention relates to the field of data processing technology, specifically to an efficient data processing method for communication networks. The method includes: acquiring a network data sequence to be processed; obtaining a feature value sequence based on the differences between adjacent network data in the network data sequence; obtaining suspected drift anomaly data based on the similarity between the normal distribution curve and probability density curve corresponding to the feature value sequence; obtaining an optimal second data window corresponding to each suspected drift anomaly data based on the difference between the feature value corresponding to each network data in each second data window and the mean corresponding to the feature value sequence; obtaining each drift anomaly data based on the first data window and the optimal second data window; and obtaining the target network data sequence based on the optimal second data window. This invention can eliminate the impact of drift anomaly data on detection accuracy, resulting in higher detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a method for efficient data processing in communication networks. Background Technology

[0002] In today's society, with the rapid development of the economy and the increasing progress of society, communication networks have been applied to various industries. In these communication networks, a large amount of data flows through network devices. This data may contain various noises and redundant information. Therefore, in order to ensure the accuracy and reliability of data analysis, network data is usually cleaned first, and then the cleaned data is analyzed, such as anomaly monitoring after data cleaning.

[0003] Existing technologies typically employ static cleaning strategies to clean data. However, the network data sequences to be cleaned or processed exhibit progressive data drift, meaning that local upward or downward data shifts may occur within the data sequence. This data shift refers to a general or partial offset within the network data sequence. Static cleaning strategies cannot eliminate this progressive data drift. Therefore, when performing anomaly detection on the cleaned data sequence, the shifted data can easily be confused with genuine anomalies, potentially being detected as anomalies. However, the detected anomaly shifted data often exhibits normal shape and trend, meaning it is not actually anomaly. This leads to low accuracy in anomaly detection. Therefore, relying solely on static cleaning strategies for data processing results in low accuracy in anomaly detection. Summary of the Invention

[0004] To address the above problems, this invention provides a method for efficient data processing in communication networks, the specific technical solution of which is as follows:

[0005] One embodiment of the present invention provides a method for efficient data processing in a communication network, comprising the following steps:

[0006] Obtain the network data sequence to be processed; the network data sequence to be processed consists of B network data, where B is greater than 0;

[0007] Based on the differences between adjacent network data in the network data sequence to be processed, the feature value sequence corresponding to the network data sequence to be processed is obtained; based on the mean and standard deviation of the feature value sequence, the normal distribution curve and probability density curve corresponding to the feature value sequence are obtained; based on the similarity between the normal distribution curve and the probability density curve, each suspected drift anomaly data in the network data sequence to be processed is obtained.

[0008] Obtain the first data window and the second data window corresponding to each suspected drift anomaly data. Based on the difference between the feature value corresponding to each network data in each second data window and the mean value corresponding to the feature value sequence, obtain the optimal second data window corresponding to each suspected drift anomaly data.

[0009] Based on the first data window and the optimal second data window corresponding to each suspected drift anomaly data, each drift anomaly data in the suspected drift anomaly data is obtained;

[0010] Based on the optimal second data window corresponding to each drift anomaly data, the replacement value corresponding to each drift anomaly data is obtained;

[0011] Each drift anomaly data is replaced with the corresponding replacement value of the drift anomaly data, and the replaced network data sequence to be processed is recorded as the target network data sequence.

[0012] Preferably, the method for obtaining the feature value sequence corresponding to the network data sequence to be processed includes:

[0013] For any network data in the network data sequence to be processed:

[0014] The difference between the network data that is adjacent to and follows the network data in the network data sequence to be processed is denoted as the first difference corresponding to the network data.

[0015] The first difference corresponding to the network data is smoothed, and the first difference after smoothing is recorded as the second difference corresponding to the network data.

[0016] Subtract the first difference corresponding to the network data from its corresponding second difference, and record the value after the subtraction as the feature value corresponding to the network data;

[0017] The sequence of feature values ​​corresponding to each network data in the network data sequence to be processed is denoted as the feature value sequence corresponding to the network data sequence to be processed.

[0018] Preferably, the method for obtaining each suspected drift anomaly data in the network data sequence to be processed includes:

[0019] The similarity between the normal distribution curve and the probability density curve is calculated using the DTW algorithm.

[0020] If the similarity between the normal distribution curve and the probability density curve is greater than or equal to a preset similarity threshold, then all network data corresponding to feature values ​​outside the first target judgment interval are recorded as suspected drift anomaly data; otherwise, all network data corresponding to feature values ​​outside the second target judgment interval are recorded as suspected drift anomaly data.

[0021] Preferably, the minimum value of the first target judgment interval is the value obtained by subtracting the mean of the feature value sequence from three times the standard deviation of the feature value sequence, and the maximum value of the first target judgment interval is the value obtained by adding the mean of the feature value sequence to three times the standard deviation of the feature value sequence.

[0022] Preferably, the minimum value of the second target judgment interval is the sum of the mean of the feature value sequence minus the standard deviation variable and the standard deviation of the feature value sequence, and the maximum value of the second target judgment interval is the sum of the mean of the feature value sequence, the standard deviation variable and the standard deviation of the feature value sequence.

[0023] Preferably, the method for obtaining the standard deviation variable includes:

[0024] The standard deviation of the feature value sequence is multiplied by the similarity between the normal distribution curve and the probability density curve. The ratio of the multiplied value to a preset similarity threshold is denoted as the standard deviation variable.

[0025] Preferably, the method for obtaining the first data window corresponding to each suspected drift anomaly data and the second data window corresponding to each suspected drift anomaly data includes:

[0026] Using each network data point in the network data sequence to be processed as the center point of the window, a window corresponding to each network data point in the network data sequence to be processed is constructed.

[0027] For any suspected drift anomaly data in the network data sequence to be processed: the window corresponding to the suspected drift anomaly data is recorded as the first data window corresponding to the suspected drift anomaly data, and the windows corresponding to all network data in the network data sequence to be processed except for the suspected drift anomaly data are recorded as the second data window corresponding to the suspected drift anomaly data.

[0028] Preferably, the method for obtaining the optimal second data window corresponding to each suspected drift anomaly data includes:

[0029] For any second data window corresponding to any suspected drift anomaly in the network data sequence to be processed:

[0030] The absolute value of the difference between the feature value corresponding to each network data in the second data window and the mean value corresponding to the feature value sequence is recorded as the initial representation value corresponding to the network data in the second data window.

[0031] Negative correlation mapping is performed on the initial characterization values ​​corresponding to each network data in the second data window. The initial characterization values ​​after negative correlation mapping are recorded as the target characterization values. The mean of the target characterization values ​​corresponding to all network data in the second data window is recorded as the first degree of preference corresponding to the second data window.

[0032] The absolute value of the difference between the label value corresponding to the central network data in the second data window and the label value corresponding to the central network data in the first data window corresponding to the suspected drift anomaly data is negatively correlated and mapped. The absolute value of the difference after negative correlation mapping is recorded as the second preference degree corresponding to the second data window. The label value of the a-th network data in the network data sequence to be processed is a.

[0033] The product of the first preference level corresponding to the second data window and the corresponding second preference level is denoted as the preference level corresponding to the second data window;

[0034] Obtain the maximum optimization degree among the optimization degrees of each second data window corresponding to the suspected drift anomaly data, and record the second data window corresponding to the maximum optimization degree as the optimal second data window corresponding to the suspected drift anomaly data.

[0035] Preferably, the method for obtaining each drift anomaly data in the suspected drift anomaly data includes:

[0036] The similarity between the first data window corresponding to each suspected drift anomaly data and its corresponding optimal second data window, calculated using the DTW algorithm, is denoted as the first similarity corresponding to the suspected drift anomaly data.

[0037] Remove the central network data from the first data window corresponding to each suspected drift anomaly data and the corresponding optimal second data window, and record the removed first data window and optimal second data window as the first feature data window and optimal second feature data window corresponding to the suspected drift anomaly data.

[0038] The similarity between the first feature data window corresponding to each suspected drift anomaly data and its corresponding optimal second feature data window, calculated using the DTW algorithm, is denoted as the second similarity corresponding to the suspected drift anomaly data.

[0039] The absolute value of the difference between the first similarity and the second similarity corresponding to each of the calculated suspected drift anomaly data is negatively correlated and mapped, and the value after negative correlation mapping is recorded as the target judgment index corresponding to the suspected drift anomaly data.

[0040] Determine whether the target judgment index corresponding to each suspected drift anomaly data is greater than or equal to the judgment threshold. If so, record the corresponding suspected drift anomaly data as drift anomaly data.

[0041] Preferably, the method for obtaining the replacement values ​​corresponding to each drift anomaly data includes:

[0042] The sequence of all network data in the optimal second data window corresponding to each drift anomaly data is denoted as the sub-network data sequence corresponding to the optimal second data window.

[0043] The sub-network data sequence corresponding to the optimal second data window for each drift anomaly data is input into the fitted prediction model to obtain the predicted value corresponding to the optimal second data window for each drift anomaly data.

[0044] The predicted value corresponding to the optimal second data window for each drift anomaly data is recorded as the replacement value corresponding to the drift anomaly data.

[0045] Beneficial Effects: This invention first obtains the network data sequence to be processed; then, based on the differences between adjacent network data in the network data sequence, it obtains the feature value sequence corresponding to the network data sequence; and based on the similarity between the normal distribution curve and probability density curve corresponding to the feature value sequence, it obtains each suspected drift anomaly data in the network data sequence; then, based on the difference between the feature value corresponding to each network data in each second data window corresponding to each suspected drift anomaly data and the mean corresponding to the feature value sequence, it obtains the optimal second data window corresponding to each suspected drift anomaly data; and based on the first data window and the optimal second data window corresponding to each suspected drift anomaly data, it obtains each drift anomaly data in each suspected drift anomaly data; finally, based on the optimal second data window corresponding to each drift anomaly data, it obtains the replacement value corresponding to each drift anomaly data, and replaces each drift anomaly data with the corresponding replacement value to obtain the replaced network data sequence to be processed. This invention can more accurately obtain drift anomaly data in the network data sequence to be processed and eliminate the influence of drift anomaly data on the detection accuracy during anomaly detection, thereby achieving higher detection accuracy. Attached Figure Description

[0046] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a method for efficient data processing in a communication network according to the present invention. Detailed Implementation

[0048] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the protection scope of the embodiments of the present invention.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

[0050] This embodiment provides a method for efficient data processing in a communication network, detailed below:

[0051] like Figure 1 As shown, the efficient data processing method for this communication network includes the following steps:

[0052] Step S001: Obtain the network data sequence to be processed.

[0053] The purpose of this embodiment is to eliminate the problem of low anomaly detection accuracy caused by progressive data drift, obtain a more reliable data sequence, and thus improve anomaly detection accuracy. That is, the efficient data processing method for communication networks provided in this embodiment can efficiently and reliably process the communication network data sequence to be processed, obtain the processed sequence, and the anomaly detection accuracy based on the processed sequence is high.

[0054] In this embodiment, the communication network data sequence to be processed is first obtained, and the obtained network data sequence to be processed is time-series, that is, all network data in the network data sequence to be processed are arranged in chronological order.

[0055] In this embodiment, the number of network data in the communication network data sequence to be processed is B. The number of network data in the communication network data sequence to be processed is related to the length of the time period corresponding to the network data sequence to be processed. The time period corresponding to the network data sequence to be processed and the length of the time period can be set by the implementer according to the actual situation. The method of obtaining the network data sequence to be processed is related to the type of network data obtained. That is, the method of obtaining the network data sequence to be processed can be set by the implementer according to the actual situation.

[0056] The network data types in this embodiment include traffic data, log data, routing data, alarm data, and QoS data, etc., and all network data in the network data sequence to be processed obtained in this embodiment are of the same type. For example, all network data in the network data sequence to be processed in this embodiment are of the type of traffic data.

[0057] Traffic data refers to network traffic information, including source address, destination address, transmission protocol, port number, etc. It can be captured using network traffic monitoring tools to extract relevant information, such as Wireshark and tcpdump. Log data refers to system and network activity logs, including login, error, and warning events, which can be collected using log collection tools. Routing data refers to the path information of routers in the network, including routing tables and topology, which can be obtained through network management systems or router CLIs. Alarm data refers to network security-related alarm information, such as intrusion detection and firewall logs, which can be collected through security information and event management systems. QoS data refers to network service quality information, such as latency, packet loss rate, and bandwidth utilization, which can be obtained through SNMP protocol or device-specific APIs.

[0058] Thus, the network data sequence to be processed has been obtained through the above process.

[0059] Step S002: Based on the differences between adjacent network data in the network data sequence to be processed, obtain the feature value sequence corresponding to the network data sequence to be processed; based on the mean and standard deviation of the feature value sequence, obtain the normal distribution curve and probability density curve corresponding to the feature value sequence; based on the similarity between the normal distribution curve and the probability density curve, obtain each suspected drift anomaly data in the network data sequence to be processed.

[0060] This embodiment will next acquire each suspected drift anomaly data in the network data sequence to be processed, and then further analyze each suspected drift anomaly data to confirm the drift anomaly data; the specific process is as follows:

[0061] Obtain the feature value sequence of the network data sequence to be processed. That is, for any network data in the network data sequence to be processed: the difference between the network data adjacent to and following the network data in the network data sequence to be processed is recorded as the first difference corresponding to the network data; and in this embodiment, the first difference corresponding to the last network data in the network data sequence to be processed is the difference between the natural number 0 and the last network data in the network data sequence to be processed.

[0062] Therefore, the first difference corresponding to each network data in the network data sequence to be processed is obtained. Then, the first difference corresponding to each network data is smoothed, and the first difference after smoothing is recorded as the second difference corresponding to the network data. Thus, the first difference and the second difference corresponding to each network data in the network data sequence to be processed are obtained.

[0063] In practical applications, it is necessary to set a method for smoothing the first difference according to the actual situation. For example, in this embodiment, the SuperSmoother algorithm can be used to smooth all the first differences. The process of smoothing the first difference based on the SuperSmoother algorithm is a well-known technology, so it will not be described in detail in this embodiment.

[0064] Since when a network data in the network data sequence to be processed is suspected drift anomaly data, the first difference and second difference of this network data are significantly different from the first difference and second difference of other normal network data in the difference sequence, that is, the first difference and second difference of the suspected drift anomaly data are significantly different from the first difference and second difference of other network data that are not suspected drift anomaly data; therefore, this embodiment obtains each suspected drift anomaly data in the network data sequence to be processed based on the difference features between the first difference and second difference of the suspected drift anomaly data and the first difference and second difference of other network data that are not suspected drift anomaly data, specifically:

[0065] Subtract the first difference and the second difference of each network data in the network data sequence to be processed, and record the value after the subtraction as the feature value of the corresponding network data. The sequence of feature values ​​corresponding to each network data in the network data sequence to be processed is recorded as the feature value sequence of the network data sequence to be processed; that is, the j-th feature value in the feature value sequence of the network data sequence to be processed is the feature value of the j-th network data in the network data sequence to be processed.

[0066] Next, the mean and standard deviation of all eigenvalues ​​in the eigenvalue sequence are calculated, thus obtaining the mean and standard deviation of the eigenvalue sequence; based on the mean of the eigenvalue sequence... and standard deviation This generates the normal distribution curve and probability density curve corresponding to the feature value sequence.

[0067] Since a high similarity between the obtained normal distribution curve and probability density curve indicates that the eigenvalue sequence conforms to a normal distribution, and the probability of suspected drift anomalies appearing in the network data sequence to be processed corresponding to the eigenvalue sequence is low (i.e., drift anomalies are low-probability events), the eigenvalue sequence is relatively consistent with a normal distribution. The criterion is also to indicate that the eigenvalue sequence exceeds... The probability that the network data corresponding to the eigenvalues ​​in a given interval is high is considered to be suspected drift anomalies. However, when the similarity between the obtained normal distribution curve and probability density curve is low, it indicates that the eigenvalue sequence does not perfectly conform to a normal distribution. This discrepancy may be due to a large number of suspected drift anomalies appearing in the network data sequence to be processed, meaning the probability of such anomalies is high. Therefore, it is necessary to narrow down the range, i.e., reduce the number of regions to be processed. The interval; and it is known that it lies within the interval. Ninety-nine percent of the eigenvalues ​​are within the range. The data within this range accounts for 68%, so the closer it conforms to a normal distribution, the lower the probability of suspected drift anomalies appearing in the network data sequence to be processed. Otherwise, the interval needs to be narrowed. Therefore, this embodiment will analyze the similarity between the normal distribution curve and the probability density curve to obtain the suspected drift anomaly data in the network data sequence to be processed.

[0068] First, the DTW algorithm is used to calculate the similarity between the normal distribution curve and the probability density curve, and then... The interval is denoted as the judgment interval; then, based on the similarity between the normal distribution curve and the probability density curve, the judgment interval, and a preset similarity threshold, the standard deviation variable is obtained; and the standard deviation variable represents the degree to which the judgment interval is narrowed when the similarity between the normal distribution curve and the probability density curve is less than the similarity threshold. Therefore, the formula for obtaining the standard deviation variable is:

[0069]

[0070] in, For the standard deviation variable, The standard deviation of the eigenvalue sequence, Preset similarity threshold, This represents the similarity between the normal distribution curve and the probability density curve.

[0071] Furthermore, the smaller the similarity between the normal distribution curve and the probability density curve, The smaller the value, the greater the reduction in the judgment interval; the greater the similarity between the normal distribution curve and the probability density curve, The larger the value, the smaller the subsequent reduction in the judgment interval.

[0072] Since the process of obtaining the similarity between the normal distribution curve and the probability density curve using the DTW algorithm is a well-known technique, it will not be described in detail in this embodiment. In specific applications, it is necessary to set a preset similarity threshold according to the actual situation. For example, in this embodiment, the preset similarity threshold is set to 0.85.

[0073] Next, it is determined whether the similarity between the normal distribution curve and the probability density curve is greater than or equal to a preset similarity threshold. Since a similarity threshold indicates that the feature value sequence closely matches a normal distribution, it also indicates that the feature value sequence closely matches... Accuracy means that the probability of suspected drift anomalies appearing in the network data sequence to be processed is small. Therefore, when the similarity between the normal distribution curve and the probability density curve is greater than or equal to the preset similarity threshold, the judgment interval is determined. Let this be the first target judgment interval, and in the normal distribution curve corresponding to the feature value sequence, select the features in the feature value sequence that fall within the first target judgment interval. All network data corresponding to features other than those specified are recorded as suspected drift anomalies.

[0074] Since a similarity between the normal distribution curve and the probability density curve is less than a preset similarity threshold, it indicates that the feature value sequence does not perfectly conform to a normal distribution. In this case, it is necessary to adjust the judgment interval. To accurately and reliably obtain suspected drift anomaly data, the judgment interval is narrowed down. Therefore, when the similarity between the normal distribution curve and the probability density curve is less than the preset similarity threshold, the second target judgment interval is obtained based on the standard deviation variable. In the normal distribution curve corresponding to the feature value sequence, all network data corresponding to feature values ​​outside the second target judgment interval in the feature value sequence are recorded as suspected drift anomaly data.

[0075] The second target judgment interval in this embodiment is: That is, the minimum value of the second target judgment interval is the value obtained by subtracting the mean of the feature value sequence from the sum of the standard deviation of the feature value sequence and the standard deviation variable, and the maximum value of the second target judgment interval is the value obtained by adding the mean of the feature value sequence from the sum of the standard deviation of the feature value sequence and the standard deviation variable.

[0076] At this point, all suspected drift anomalies in the network data sequence to be processed have been obtained.

[0077] Step S003: Obtain the first data window and the second data window corresponding to each suspected drift anomaly data. Based on the difference between the feature value corresponding to each network data in the second data window and the mean value corresponding to the feature value sequence, obtain the optimal second data window corresponding to each suspected drift anomaly data. Based on the first data window and the optimal second data window corresponding to each suspected drift anomaly data, obtain each drift anomaly data in the suspected drift anomaly data.

[0078] Because progressive data drift in the network data sequence to be processed can affect anomaly detection results—that is, local upward or downward data shifts in the network data sequence can affect anomaly detection results—directly performing anomaly detection on the network data sequence to be processed will lead to low accuracy. This is because, when performing anomaly detection on the network data sequence to be processed, the shifted data portion can easily be confused with real anomaly data, noise data, or abrupt changes. Specifically, the suspected drift anomaly data obtained in step S002 may also contain drift anomaly data, real anomaly data, noise data, and abrupt changes. Furthermore, in this embodiment, the data to be processed... The reason for the presence of noisy data in the network data sequence is that this embodiment did not perform data cleaning to eliminate noise in the network data sequence to be processed; and because the detected abnormal offset data portions do not have abnormal data patterns and trends, that is, the detected abnormal offset data will not change the trend, while the real abnormal data, noisy data, or abrupt data will change the trend, this embodiment will analyze all suspected drift anomalies in the obtained network data sequence to be processed based on the characteristic that abnormal offset data will not change the trend, while the real abnormal data, noisy data, or abrupt data will change the trend, to obtain each drift anomaly data in each suspected drift anomaly data; the specific process is as follows:

[0079] Using each network data point in the network data sequence to be processed as the center point of the window, a window corresponding to each network data point in the network data sequence to be processed is constructed. The size of the constructed window is [missing information]. That is, the number of network data in the window corresponding to each network data is m, and in specific applications, the value of m needs to be set according to the actual situation. In this embodiment, the value of m is set to 19.

[0080] For any suspected drift anomaly in the network data sequence to be processed:

[0081] The window corresponding to the suspected drift anomaly data is recorded as the first data window corresponding to the suspected drift anomaly data, and the windows corresponding to all network data in the network data sequence to be processed, excluding the suspected drift anomaly data, are recorded as the second data window corresponding to the suspected drift anomaly data.

[0082] Therefore, this embodiment obtains the second data windows corresponding to the suspected drift anomaly data. Next, this embodiment will analyze the first data window and the second data windows corresponding to the suspected drift anomaly data to obtain the optimal second data window. The purpose of obtaining the optimal second data window is to determine the probability that the suspected drift anomaly data is indeed drift anomaly data. Furthermore, the replacement value corresponding to the drift anomaly data will be obtained based on the optimal second data window. Since the closer the second data window corresponding to the suspected drift anomaly data is to the first data window, the greater the correlation between the second data window and the first data window representing the changing trend of the network data subsequence, i.e., the greater the probability that the corresponding second data window is the optimal second data window, and the more normal network data in the second data window, the greater the credibility of the corresponding second data window. In other words, in the normal distribution curve, the closer the characteristic values ​​of each network data in each second data window corresponding to the suspected drift anomaly data are to the mean. This indicates that the greater the confidence level of the corresponding second data window, the greater the probability that the corresponding second data window is the optimal second data window. Therefore, in the following embodiment, based on the features analyzed above, the optimization degree of each second data window corresponding to the suspected drift anomaly data is obtained:

[0083] Each network data point is labeled according to its position in the network data sequence to be processed, thus obtaining the label value of each network data point in the network data sequence to be processed; that is, the label value of the a-th network data point in the network data sequence to be processed is a.

[0084] For any second data window corresponding to this suspected drift anomaly data:

[0085] In the normal distribution curve, obtain the characteristic values ​​and mean values ​​corresponding to each network data in the second data window. The absolute value of the difference between the two is recorded as the initial representation value corresponding to the network data in the second data window. A negative correlation mapping is performed on the initial representation values ​​corresponding to each network data in the second data window, and the initial representation value after negative correlation mapping is recorded as the target representation value corresponding to the network data in the second data window. The mean of the target representation values ​​corresponding to all network data in the second data window is recorded as the first optimization degree corresponding to the second data window. The absolute value of the difference between the label value corresponding to the central network data in the second data window and the label value corresponding to the central network data in the first data window corresponding to the suspected drift anomaly data is calculated, and a negative correlation mapping is performed on the absolute value of the difference. The absolute value of the difference after negative correlation mapping is recorded as the second optimization degree corresponding to the second data window. The product of the first optimization degree corresponding to the second data window and its corresponding second optimization degree is recorded as the optimization degree corresponding to the second data window. The optimization degree corresponding to the second data window is calculated according to the following formula:

[0086] in, The degree of optimization corresponding to this second data window. This refers to the feature value corresponding to the i-th network data in the second data window. The mean of the eigenvalue sequence. The amount of network data in this second data window, exp() is the absolute value of the difference between the marker value corresponding to the central network data in the second data window and the marker value corresponding to the central network data in the first data window corresponding to the suspected drift anomaly data, and is an exponential function with the natural constant e as the base.

[0087] In addition, when The larger the value, the greater the probability that the second data window is the optimal second data window corresponding to the suspected drift anomaly data; conversely, when... The smaller the value, the lower the probability that the second data window is the optimal second data window corresponding to the suspected drift anomaly data; and when The larger, When it is larger, The smaller the value, the better; conversely, when... The smaller The smaller, The larger the value, the better.

[0088] Therefore, the optimization degree of each second data window corresponding to the suspected drift anomaly data is obtained through the above process, and the second data window corresponding to the maximum optimization degree of each second data window corresponding to the suspected drift anomaly data is recorded as the optimal second data window corresponding to the suspected drift anomaly data.

[0089] Therefore, in this embodiment, the first data window and its corresponding optimal second data window can be obtained for each suspected drift anomaly data in the network data sequence to be processed through the above-described process of obtaining the optimal second data window corresponding to the suspected drift anomaly data.

[0090] Then, using the DTW algorithm, the similarity between the first data window corresponding to each suspected drift anomaly and its corresponding optimal second data window is calculated and recorded as the first similarity corresponding to the suspected drift anomaly. Next, the central network data of the first data window and its corresponding optimal second data window corresponding to each suspected drift anomaly are removed. The removed first data window and optimal second data window are recorded as the first feature data window and optimal second feature data window corresponding to the suspected drift anomaly. Then, using the DTW algorithm, the similarity between the first feature data window and its corresponding optimal second feature data window corresponding to each suspected drift anomaly is calculated and recorded as the second similarity corresponding to the suspected drift anomaly.

[0091] Next, the absolute value of the difference between the first similarity and the second similarity corresponding to each suspected drift anomaly data is calculated, and the absolute value of the difference between the first similarity and the second similarity corresponding to each suspected drift anomaly data is recorded as the judgment index corresponding to the suspected drift anomaly data.

[0092] Furthermore, the smaller the absolute value of the difference between the first similarity and the corresponding second similarity for each suspected drift anomaly, the more similar the trends of the data windows without removing the central network data and those without, meaning the suspected drift anomaly has little impact on similarity. Conversely, the larger the absolute value of the difference between the first similarity and the corresponding second similarity for each suspected drift anomaly, the more dissimilar the trends of the data windows without removing the central network data and those without, meaning the suspected drift anomaly has a significant impact on similarity. Also, due to progressive anomaly shift... Some data may not exhibit abnormal patterns or trends, meaning that anomalies caused by gradual drift do not alter the trend, while anomalies inherent in the data itself can. Therefore, the smaller the value of the judgment indicator corresponding to suspected drift anomaly data, the greater the probability that the suspected drift anomaly data is caused by gradual drift, i.e., the greater the probability that the suspected drift anomaly data is indeed drift anomaly data. Conversely, the larger the value of the judgment indicator corresponding to suspected drift anomaly data, the smaller the probability that the suspected drift anomaly data is caused by gradual drift, i.e., the greater the probability that the corresponding anomaly data is itself anomaly, noisy data, or abrupt change in data.

[0093] Therefore, the next step is to perform negative correlation mapping on the judgment indicators corresponding to each suspected drift anomaly data, and then use the judgment indicators after negative correlation mapping as the target judgment indicators corresponding to the suspected drift anomaly data; that is, for any suspected drift anomaly data, Let be the target judgment index corresponding to the suspected drift anomaly data, where The first similarity score corresponds to the suspected drift anomaly data. The second similarity is calculated for the suspected drift anomaly data. Then, it is determined whether the target judgment index corresponding to each suspected drift anomaly data is greater than or equal to the judgment threshold. If so, it indicates that the corresponding anomaly data belongs to the anomaly caused by gradual drift, and the corresponding suspected drift anomaly data is recorded as drift anomaly data. Otherwise, it indicates that the corresponding suspected drift anomaly data does not belong to the anomaly caused by gradual drift, and the probability that the corresponding suspected drift anomaly data is an anomaly of the data itself, noise data, or mutation data is greater.

[0094] In practical applications, the judgment threshold needs to be set according to the actual situation. In this embodiment, the judgment threshold is set to 0.82.

[0095] Therefore, through the above process, each drift anomaly data in each suspected drift anomaly data was obtained, that is, each drift anomaly data in the network data sequence to be processed was obtained.

[0096] Step S004: Based on the optimal second data window corresponding to each drift anomaly data, obtain the replacement value corresponding to each drift anomaly data; replace each drift anomaly data with the replacement value of the corresponding drift anomaly data, and record the replaced network data sequence to be processed as the target network data sequence.

[0097] Since step S003 yielded the drift anomaly data in the network data sequence to be processed, this embodiment will next process these drift anomaly data to obtain the target network data sequence, specifically as follows:

[0098] First, the sequence of all network data in the optimal second data window corresponding to each drift anomaly data is constructed and denoted as the sub-network data sequence corresponding to the optimal second data window. Then, the sub-network data sequence corresponding to the optimal second data window corresponding to each drift anomaly data is input into the fitted prediction model to obtain the predicted value corresponding to the optimal second data window corresponding to each drift anomaly data, and denoted as the replacement value corresponding to the drift anomaly data. Then, each drift anomaly data is replaced with the replacement value corresponding to the drift anomaly data, and the replaced network data sequence to be processed is denoted as the target network data sequence.

[0099] In practical applications, the prediction model needs to be selected according to the actual situation. The prediction model used in this embodiment is the ARIMA model. In this embodiment, the sub-network data sequence corresponding to the optimal second data window is input into the prediction model, and the prediction model outputs the predicted value. Since there are still some data between each drift anomaly data and the sub-network data sequence corresponding to its optimal second data window, these data are used as validation data. That is, the predicted value output by the model is compared with it, so as to perform feedback adjustment and adjust the parameters of the prediction model until a suitable parameter is selected so that the predicted value is closer to the true value. After the model adjustment is completed, the predicted value at the corresponding position of the anomaly data point is predicted and used as the replacement value.

[0100] Since the fitting and prediction processes of the ARIMA model are well-known techniques, they will not be described in detail in this embodiment.

[0101] Then, the target network data sequence is cleaned to obtain a cleaned target network data sequence. The data cleaning can remove noisy and redundant data in the target network data sequence. After that, an anomaly detection algorithm is used to detect anomalies in the cleaned target network data sequence to obtain abnormal data in the cleaned target network data sequence. In this embodiment, anomaly detection is performed on the network data sequence after the drift anomaly data has been processed, which can make the detection results more accurate.

[0102] This embodiment uses a static cleaning strategy to clean the communication network data sequence to be cleaned. Since the specific data cleaning process is a well-known technique, it will not be described in detail here. In practical applications, anomaly detection algorithms can be selected according to the actual situation. This embodiment uses the Isolation Forest algorithm to detect anomalies in the target network data sequence, obtain abnormal data in the target network data sequence, and complete the processing of the communication network data by marking the abnormal data. Since the Isolation Forest algorithm is a well-known technique, it will not be described in detail here.

[0103] This embodiment first acquires the network data sequence to be processed; then, based on the differences between adjacent network data in the network data sequence, it obtains the feature value sequence corresponding to the network data sequence. Based on the similarity between the normal distribution curve and probability density curve corresponding to the feature value sequence, it obtains each suspected drift anomaly data in the network data sequence. Next, based on the difference between the feature value corresponding to each network data in each second data window corresponding to each suspected drift anomaly data and the mean corresponding to the feature value sequence, it obtains the optimal second data window corresponding to each suspected drift anomaly data. Based on the first data window and the optimal second data window corresponding to each suspected drift anomaly data, it obtains each drift anomaly data within each suspected drift anomaly data. Finally, based on the optimal second data window corresponding to each drift anomaly data, it obtains the replacement value corresponding to each drift anomaly data, and replaces each drift anomaly data with the corresponding replacement value to obtain the replaced network data sequence to be processed. This embodiment can more accurately acquire drift anomaly data in the network data sequence to be processed and eliminate the influence of drift anomaly data on detection accuracy during anomaly detection, thereby achieving higher detection accuracy.

[0104] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for efficient data processing in a communication network, characterized in that, The method includes the following steps: Obtain the network data sequence to be processed; the network data sequence to be processed consists of B network data, where B is greater than 0; Based on the differences between adjacent network data in the network data sequence to be processed, the feature value sequence corresponding to the network data sequence to be processed is obtained; based on the mean and standard deviation of the feature value sequence, the normal distribution curve and probability density curve corresponding to the feature value sequence are obtained; based on the similarity between the normal distribution curve and the probability density curve, each suspected drift anomaly data in the network data sequence to be processed is obtained. Obtain the first data window corresponding to each suspected drift anomaly data and the second data window corresponding to each suspected drift anomaly data. Based on the difference between the feature value corresponding to each network data in each second data window corresponding to each suspected drift anomaly data and the mean value corresponding to the feature value sequence, obtain the optimal second data window corresponding to each suspected drift anomaly data. Based on the first data window and the optimal second data window corresponding to each suspected drift anomaly data, each drift anomaly data in the suspected drift anomaly data is obtained; Based on the optimal second data window corresponding to each drift anomaly data, the replacement value corresponding to each drift anomaly data is obtained; Each drift anomaly data is replaced with the corresponding replacement value of the drift anomaly data, and the replaced network data sequence to be processed is recorded as the target network data sequence.

2. The efficient data processing method for a communication network as described in claim 1, characterized in that, Methods for obtaining the feature value sequence corresponding to the network data sequence to be processed include: For any network data in the network data sequence to be processed: The difference between the network data that is adjacent to and follows the network data in the network data sequence to be processed is denoted as the first difference corresponding to the network data. The first difference corresponding to the network data is smoothed, and the first difference after smoothing is recorded as the second difference corresponding to the network data. Subtract the first difference corresponding to the network data from its corresponding second difference, and record the value after the subtraction as the feature value corresponding to the network data; The sequence of feature values ​​corresponding to each network data in the network data sequence to be processed is denoted as the feature value sequence corresponding to the network data sequence to be processed.

3. The efficient data processing method for a communication network as described in claim 2, characterized in that, The method for obtaining each suspected drift anomaly data in the network data sequence to be processed includes: The similarity between the normal distribution curve and the probability density curve is calculated using the DTW algorithm. If the similarity between the normal distribution curve and the probability density curve is greater than or equal to a preset similarity threshold, then all network data corresponding to feature values ​​outside the first target judgment interval are recorded as suspected drift anomaly data; otherwise, all network data corresponding to feature values ​​outside the second target judgment interval are recorded as suspected drift anomaly data.

4. The efficient data processing method for a communication network as described in claim 3, characterized in that, The minimum value of the first target judgment interval is the value obtained by subtracting the mean of the feature value sequence from three times the standard deviation of the feature value sequence, and the maximum value of the first target judgment interval is the value obtained by adding the mean of the feature value sequence to three times the standard deviation of the feature value sequence.

5. The efficient data processing method for a communication network as described in claim 3, characterized in that, The minimum value of the second target judgment interval is the sum of the mean of the feature value sequence minus the standard deviation variable and the standard deviation of the feature value sequence, and the maximum value of the second target judgment interval is the sum of the mean of the feature value sequence, the standard deviation variable and the standard deviation of the feature value sequence.

6. The efficient data processing method for a communication network as described in claim 5, characterized in that, Methods for obtaining the standard deviation variable include: The standard deviation of the feature value sequence is multiplied by the similarity between the normal distribution curve and the probability density curve. The ratio of the multiplied value to a preset similarity threshold is denoted as the standard deviation variable.

7. The efficient data processing method for a communication network as described in claim 1, characterized in that, A method for obtaining a first data window corresponding to each suspected drift anomaly data and a second data window corresponding to each suspected drift anomaly data includes: Using each network data point in the network data sequence to be processed as the center point of the window, a window corresponding to each network data point in the network data sequence to be processed is constructed. For any suspected drift anomaly data in the network data sequence to be processed: the window corresponding to the suspected drift anomaly data is recorded as the first data window corresponding to the suspected drift anomaly data, and the windows corresponding to all network data in the network data sequence to be processed except for the suspected drift anomaly data are recorded as the second data window corresponding to the suspected drift anomaly data.

8. The efficient data processing method for a communication network as described in claim 1, characterized in that, The method for obtaining the optimal second data window corresponding to each suspected drift anomaly data includes: For any second data window corresponding to any suspected drift anomaly data in the network data sequence to be processed: The absolute value of the difference between the feature value corresponding to each network data in the second data window and the mean value corresponding to the feature value sequence is recorded as the initial representation value corresponding to the network data in the second data window. Negative correlation mapping is performed on the initial characterization values ​​corresponding to each network data in the second data window. The initial characterization values ​​after negative correlation mapping are recorded as the target characterization values. The mean of the target characterization values ​​corresponding to all network data in the second data window is recorded as the first degree of preference corresponding to the second data window. The absolute value of the difference between the label value corresponding to the central network data in the second data window and the label value corresponding to the central network data in the first data window corresponding to the suspected drift anomaly data is negatively correlated and mapped. The absolute value of the difference after negative correlation mapping is recorded as the second preference degree corresponding to the second data window. The label value of the a-th network data in the network data sequence to be processed is a. The product of the first preference level corresponding to the second data window and the corresponding second preference level is denoted as the preference level corresponding to the second data window. Obtain the maximum optimization degree among the optimization degrees of each second data window corresponding to the suspected drift anomaly data, and record the second data window corresponding to the maximum optimization degree as the optimal second data window corresponding to the suspected drift anomaly data.

9. The efficient data processing method for a communication network as described in claim 1, characterized in that, The method for obtaining each drift anomaly data in each of the suspected drift anomaly data includes: The similarity between the first data window corresponding to each suspected drift anomaly data and its corresponding optimal second data window, calculated using the DTW algorithm, is denoted as the first similarity corresponding to the suspected drift anomaly data. Remove the central network data from the first data window corresponding to each suspected drift anomaly data and the corresponding optimal second data window, and record the removed first data window and optimal second data window as the first feature data window and optimal second feature data window corresponding to the suspected drift anomaly data. The similarity between the first feature data window corresponding to each suspected drift anomaly data and its corresponding optimal second feature data window, calculated using the DTW algorithm, is denoted as the second similarity corresponding to the suspected drift anomaly data. The absolute value of the difference between the first similarity and the second similarity corresponding to each of the calculated suspected drift anomaly data is negatively correlated and mapped, and the value after negative correlation mapping is recorded as the target judgment index corresponding to the suspected drift anomaly data. Determine whether the target judgment index corresponding to each suspected drift anomaly data is greater than or equal to the judgment threshold. If so, record the corresponding suspected drift anomaly data as drift anomaly data.

10. The efficient data processing method for a communication network as described in claim 1, characterized in that, The method for obtaining the replacement values ​​corresponding to each drift anomaly data includes: The sequence of all network data in the optimal second data window corresponding to each drift anomaly data is denoted as the sub-network data sequence corresponding to the optimal second data window. The sub-network data sequence corresponding to the optimal second data window for each drift anomaly data is input into the fitted prediction model to obtain the predicted value corresponding to the optimal second data window for each drift anomaly data. The predicted value corresponding to the optimal second data window for each drift anomaly data is recorded as the replacement value corresponding to the drift anomaly data.

Images