Efficient data processing method for communication network

Abstract

The invention relates to the technical field of data processing, in particular to an efficient data processing method for a communication network. The method comprises the steps of obtaining a to-be-processed network data sequence; obtaining a characteristic value sequence based on the difference between adjacent network data in the to-be-processed network data sequence, and obtaining each piece of suspected drift anomaly data according to the similarity between a normal distribution curve and a probability density curve corresponding to the characteristic value sequence; obtaining an optimal second data window corresponding to each piece of suspected drift abnormal data according to a difference between a feature value corresponding to each piece of network data in each second data window corresponding to each piece of suspected drift abnormal data and a mean value corresponding to the feature value sequence; according to the first data window and the optimal second data window corresponding to each piece of suspected drift abnormal data, obtaining each piece of drift abnormal data; and obtaining a target network data sequence according to the optimal second data window. The method can eliminate the influence of the drift abnormal data on the detection precision, and enables the detection precision to be higher.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for efficiently processing data in a communication network. Background Art

[0002] In today's society, with the rapid development of the economy and the advancement of society, communication networks have been applied to various industries. In communication networks, a large amount of data will flow through network devices. These data flowing through network devices may contain various noises and redundant information. Therefore, in order to ensure the accuracy and reliability of data analysis, network data is usually cleaned first, and then the cleaned data is analyzed, such as anomaly monitoring of the cleaned data.

[0003] In the prior art, static cleaning strategies are usually adopted to achieve data cleaning. However, there is a gradual data drift phenomenon in the network data sequence to be cleaned or processed, that is, a local upward or local downward data offset may appear in the network data sequence to be cleaned. The data offset refers to the overall or partial offset generated in the network data sequence; and the static cleaning strategy cannot eliminate the gradual data drift phenomenon. Therefore, when abnormal data detection is performed on the data sequence after data cleaning, the data of the offset part in the data sequence is easily confused with the real abnormal data, that is, the data of the offset part may also be detected as abnormal, but under normal circumstances, the shape and trend of the data of the abnormal offset part detected are not abnormal, that is, the data of the abnormal offset part detected is not abnormal, which leads to low accuracy in data anomaly detection; therefore, only using static cleaning strategies to process data will result in low accuracy in anomaly detection. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a method for efficiently processing data in a communication network. The technical solutions adopted are as follows: An embodiment of the present invention provides a method for efficiently processing data in a communication network, comprising the following steps: Obtain a sequence of network data to be processed; the sequence of network data to be processed is composed of B network data, where B is greater than 0; Obtaining a characteristic value sequence corresponding to the network data sequence to be processed based on the differences between adjacent network data in the network data sequence to be processed; obtaining a normal distribution curve and a probability density curve corresponding to the characteristic value sequence based on the mean and standard deviation corresponding to the characteristic value sequence; and obtaining each suspected drift abnormal data in the network data sequence to be processed based on the similarity between the normal distribution curve and the probability density curve; Obtaining first data windows corresponding to each suspected drift abnormal data and second data windows corresponding to each suspected drift abnormal data, and obtaining optimal second data windows corresponding to each suspected drift abnormal data based on differences between eigenvalues ​​corresponding to each network data in each second data window corresponding to each suspected drift abnormal data and means corresponding to the eigenvalue sequences; Obtaining each drift abnormality data in each suspected drift abnormality data according to the first data window and the optimal second data window corresponding to each suspected drift abnormality data; Obtaining a replacement value corresponding to each drift abnormal data according to the optimal second data window corresponding to each drift abnormal data; Each drift abnormal data is replaced with a replacement value of the corresponding drift abnormal data, and the network data sequence to be processed after the replacement is recorded as a target network data sequence.

[0005] Preferably, the method for obtaining the eigenvalue sequence corresponding to the network data sequence to be processed includes: For any network data in the sequence of network data to be processed: Recording the difference between the network data adjacent to and following the network data in the sequence of network data to be processed and the network data as the first difference corresponding to the network data; Performing smoothing on the first difference value corresponding to the network data, and recording the first difference value after the smoothing as the second difference value corresponding to the network data; Subtracting the first difference value corresponding to the network data from the second difference value corresponding to the network data, and recording the value after the subtraction as the characteristic value corresponding to the network data; The sequence constructed by the characteristic values ​​corresponding to each network data in the network data sequence to be processed is recorded as the characteristic value sequence corresponding to the network data sequence to be processed.

[0006] Preferably, the method for obtaining each suspected drift abnormal data in the network data sequence to be processed includes: Using the DTW algorithm, the similarity between the normal distribution curve and the probability density curve is calculated; Determine whether the similarity between the normal distribution curve and the probability density curve is greater than or equal to a preset similarity threshold. If so, the network data corresponding to all characteristic values ​​outside the first target judgment interval are recorded as suspected drift abnormal data; otherwise, the network data corresponding to all characteristic values ​​outside the second target judgment interval are recorded as suspected drift abnormal data.

[0007] Preferably, the minimum value of the first target judgment interval is the value obtained by subtracting the mean of the eigenvalue sequence from three times the standard deviation of the eigenvalue sequence, and the maximum value of the first target judgment interval is the value obtained by adding the mean of the eigenvalue sequence to three times the standard deviation of the eigenvalue sequence.

[0008] Preferably, the minimum value of the second target judgment interval is the mean of the eigenvalue sequence minus the sum of the standard deviation variable and the standard deviation of the eigenvalue sequence, and the maximum value of the second target judgment interval is the sum of the mean of the eigenvalue sequence, the standard deviation variable and the standard deviation of the eigenvalue sequence.

[0009] Preferably, the method for obtaining the standard deviation variable includes: The similarity between the normal distribution curve and the probability density curve is multiplied by twice the standard deviation of the feature value sequence, and the ratio of the multiplied value to a preset similarity threshold is recorded as the standard deviation variable.

[0010] Preferably, the method for obtaining the first data window corresponding to each suspected drift abnormal data and the second data window corresponding to each suspected drift abnormal data includes: Taking each network data in the sequence of network data to be processed as a window center point, constructing a window corresponding to each network data in the sequence of network data to be processed; For any suspected drift abnormal data in the network data sequence to be processed: the window corresponding to the suspected drift abnormal data is recorded as the first data window corresponding to the suspected drift abnormal data, and the windows corresponding to all network data in the network data sequence to be processed except the suspected drift abnormal data are recorded as the second data window corresponding to the suspected drift abnormal data.

[0011] Preferably, the method for obtaining the optimal second data window corresponding to each suspected drift abnormal data includes: For any second data window corresponding to any suspected drift abnormal data in the network data sequence to be processed: Recording the absolute value of the difference between the characteristic value corresponding to each network data in the second data window and the mean value corresponding to the characteristic value sequence as the initial characterization value corresponding to the corresponding network data in the second data window; Performing negative correlation mapping on the initial characterization values ​​corresponding to the network data in the second data window, recording the initial characterization values ​​after the negative correlation mapping as target characterization values, and recording the average of the target characterization values ​​corresponding to all the network data in the second data window as the first preference level corresponding to the second data window; Performing negative correlation mapping on the absolute value of the difference between the label value corresponding to the central network data in the second data window and the label value corresponding to the central network data in the first data window corresponding to the suspected drift abnormal data, recording the absolute value of the difference after the negative correlation mapping as the second preference level corresponding to the second data window, and the label value of the a-th network data in the network data sequence to be processed is a; The product of the first preference level corresponding to the second data window and the second preference level corresponding to the second data window is recorded as the preference level corresponding to the second data window; The maximum preferred degree among the preferred degrees corresponding to the second data windows corresponding to the suspected drift abnormal data is obtained, and the second data window corresponding to the maximum preferred degree is recorded as the optimal second data window corresponding to the suspected drift abnormal data.

[0012] Preferably, the method for obtaining each drift abnormality data in the each suspected drift abnormality data includes: The similarity between the first data window corresponding to each suspected drift abnormal data and the corresponding optimal second data window calculated by the DTW algorithm is recorded as the first similarity corresponding to the suspected drift abnormal data; Removing the central network data from the first data window corresponding to each suspected drift abnormal data and the corresponding optimal second data window, and recording the first data window and the optimal second data window after the removal as the first characteristic data window and the optimal second characteristic data window corresponding to the suspected drift abnormal data; The similarity between the first feature data window corresponding to each suspected drift abnormal data and its corresponding optimal second feature data window calculated using the DTW algorithm is recorded as the second similarity corresponding to the suspected drift abnormal data; Performing negative correlation mapping on the absolute value of the difference between the calculated first similarity corresponding to each suspected drift abnormal data and the calculated second similarity, and recording the value after the negative correlation mapping as the target judgment indicator corresponding to the suspected drift abnormal data; It is determined whether the target judgment index corresponding to each suspected drift abnormal data is greater than or equal to a judgment threshold. If so, the corresponding suspected drift abnormal data is recorded as drift abnormal data.

[0013] Preferably, the method for obtaining the replacement value corresponding to each drift abnormal data includes: Recording a sequence constructed from all network data in the optimal second data window corresponding to each drift abnormal data as a sub-network data sequence corresponding to the optimal second data window; Inputting the subnetwork data sequence corresponding to the optimal second data window corresponding to each drift abnormal data into the fitted prediction model to obtain the predicted value corresponding to the optimal second data window corresponding to each drift abnormal data; The predicted value corresponding to the optimal second data window corresponding to each drift abnormal data is recorded as the replacement value corresponding to the corresponding drift abnormal data.

[0014] Beneficial effect: The present invention first obtains a network data sequence to be processed; then, based on the difference between adjacent network data in the network data sequence to be processed, obtains a eigenvalue sequence corresponding to the network data sequence to be processed, and obtains each suspected drift abnormal data in the network data sequence to be processed according to the similarity between the normal distribution curve and the probability density curve corresponding to the eigenvalue sequence; then, based on the difference between the eigenvalue corresponding to each network data in each second data window corresponding to each suspected drift abnormal data and the mean value corresponding to the eigenvalue sequence, obtains the optimal second data window corresponding to each suspected drift abnormal data, and according to the first data window and the optimal second data window corresponding to each suspected drift abnormal data, obtains each drift abnormal data in each suspected drift abnormal data; finally, based on the optimal second data window corresponding to each drift abnormal data, obtains a replacement value corresponding to each drift abnormal data, and replaces each drift abnormal data with the replacement value of the corresponding drift abnormal data to obtain the replaced network data sequence to be processed. The present invention can more accurately obtain the drift abnormal data in the network data sequence to be processed, and eliminate the influence of the drift abnormal data on the detection accuracy during anomaly detection, thereby making the detection accuracy higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 The present invention is a flow chart of a method for efficiently processing data in a communication network. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field fall within the scope of protection of the embodiments of the present invention.

[0018] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] This embodiment provides a method for efficiently processing data in a communication network, which is described in detail as follows: like Figure 1 As shown, the data efficient processing method of the communication network includes the following steps: Step S001: Obtain a sequence of network data to be processed.

[0020] The purpose of this embodiment is to eliminate the problem of low anomaly detection accuracy caused by progressive data drift, obtain a more reliable data sequence, and thus improve the anomaly detection accuracy. That is, the efficient data processing method of the communication network provided by this embodiment can efficiently and reliably process the communication network data sequence to be processed, obtain the processed sequence, and the accuracy of anomaly detection based on the processed sequence is high.

[0021] This embodiment first obtains a sequence of communication network data to be processed, and the obtained sequence of network data to be processed is time-sequential, that is, all network data in the sequence of network data to be processed are arranged in chronological order.

[0022] In this embodiment, the number of network data in the communication network data sequence to be processed is B, and the number of network data in the communication network data sequence to be processed is related to the length of the time period corresponding to obtaining the network data sequence to be processed, and the time period corresponding to the network data sequence to be processed and the length of the corresponding time period can be set by the implementer according to actual conditions; the method of obtaining the network data sequence to be processed is related to the type of network data obtained, that is, the method of obtaining the network data sequence to be processed can be set by the implementer according to actual conditions.

[0023] The network data types in this embodiment include traffic data, log data, routing data, alarm data, and QoS data, etc., and the types of all network data in the sequence of network data to be processed obtained in this embodiment are the same. For example, the types of all network data in the sequence of network data to be processed in this embodiment are traffic data.

[0024] Traffic data refers to the data flow information in the network, including source address, destination address, transmission protocol, port number, etc., which can be captured based on network traffic monitoring tools to extract relevant information, such as using Wireshark and tcpdump to collect traffic data; log data refers to the log information of system and network activities, including login, error, warning and other events, which can be collected through log collection tools; routing data refers to the path information of routers in the network, including routing tables, topology structures, etc., which can be obtained through the network management system or router CLI; alarm data refers to alarm information related to network security, such as intrusion detection, firewall logs, etc., which can be collected through the security information and event management system; QoS data refers to service quality information in the network, such as delay, packet loss rate, bandwidth utilization, etc., which can be obtained through the SNMP protocol or device-specific API.

[0025] So far, the network data sequence to be processed has been obtained through the above process.

[0026] Step S002: obtaining a characteristic value sequence corresponding to the network data sequence to be processed based on the differences between adjacent network data in the network data sequence to be processed; obtaining a normal distribution curve and a probability density curve corresponding to the characteristic value sequence based on the mean and standard deviation corresponding to the characteristic value sequence; and obtaining each suspected drift abnormal data in the network data sequence to be processed based on the similarity between the normal distribution curve and the probability density curve.

[0027] This embodiment will then obtain each suspected drift abnormal data in the network data sequence to be processed, and will subsequently further analyze each suspected drift abnormal data to determine the drift abnormal data; the specific process is: Obtain a characteristic value sequence of the network data sequence to be processed, that is, for any network data in the network data sequence to be processed: the difference between the network data adjacent to and located after the network data in the network data sequence to be processed and the network data is recorded as the first difference corresponding to the network data; and the first difference corresponding to the last network data in the network data sequence to be processed in this embodiment is the difference between the natural number 0 and the last network data in the network data sequence to be processed.

[0028] Therefore, the first difference value corresponding to each network data in the network data sequence to be processed is obtained, and then the first difference value corresponding to each network data is smoothed, and the first difference value after smoothing is recorded as the second difference value corresponding to the corresponding network data; thus, the first difference value and the second difference value corresponding to each network data in the network data sequence to be processed are obtained.

[0029] In specific applications, it is necessary to set a method for smoothing the first difference according to actual conditions. For example, in this embodiment, the SuperSmoother algorithm can be used to smooth all first differences. The process of smoothing the first difference based on the SuperSmoother algorithm is a well-known technology, so this embodiment will not be described in detail.

[0030] When a certain network data in the sequence of network data to be processed is suspected drift abnormal data, the first difference and the second difference of the network data are significantly different from the first difference and the second difference corresponding to other normal network data in the difference sequence, that is, the first difference and the second difference of the suspected drift abnormal data are significantly different from the first difference and the second difference of other network data that are not suspected drift abnormal data; therefore, this embodiment obtains each suspected drift abnormal data in the sequence of network data to be processed based on the distinguishing features between the first difference and the second difference of the suspected drift abnormal data and the first difference and the second difference of other network data that are not suspected drift abnormal data, specifically: Subtract the first difference value corresponding to each network data in the network data sequence to be processed from its corresponding second difference value, record the value after subtraction as the eigenvalue corresponding to the corresponding network data, and record the sequence constructed by the eigenvalue corresponding to each network data in the network data sequence to be processed as the eigenvalue sequence corresponding to the network data sequence to be processed; that is, the j-th eigenvalue in the eigenvalue sequence corresponding to the network data sequence to be processed is the eigenvalue corresponding to the j-th network data in the network data sequence to be processed.

[0031] Then calculate the mean and standard deviation of all eigenvalues ​​in the eigenvalue sequence, that is, calculate the mean and standard deviation of the eigenvalue sequence; according to the mean of the eigenvalue sequence and standard deviation , generate the normal distribution curve and probability density curve corresponding to the eigenvalue sequence.

[0032] When the similarity between the obtained normal distribution curve and the probability density curve is relatively high, it indicates that the eigenvalue sequence is more consistent with the normal distribution, and the probability of the suspected drift abnormal data appearing in the to-be-processed network data sequence corresponding to the eigenvalue sequence is small, that is, the drift abnormal data is a low-probability event, so at this time the eigenvalue sequence is more consistent with the normal distribution. The criterion is also to indicate that the eigenvalue sequence exceeds The probability that the network data corresponding to the eigenvalue of the interval is suspected drift abnormal data is relatively high; when the similarity between the obtained normal distribution curve and the probability density curve is relatively low, it indicates that the eigenvalue sequence does not completely conform to the normal distribution. The reason for not completely conforming to the normal distribution at this time may be that there are more suspected drift abnormal data in the network data sequence to be processed corresponding to the eigenvalue sequence, that is, the probability of suspected drift abnormal data appearing in the network data sequence to be processed is relatively high, so it is necessary to narrow the range, that is, to narrow the range. interval; and it is known that in the interval Ninety-nine percent of the eigenvalues ​​are within the interval The data within this range accounts for 68%, so the more it conforms to the normal distribution, the smaller the probability of suspected drift abnormal data appearing in the network data sequence to be processed. Otherwise, the interval needs to be narrowed. Therefore, this embodiment will analyze the similarity between the normal distribution curve and the probability density curve, thereby achieving the purpose of obtaining each suspected drift abnormal data in the network data sequence to be processed.

[0033] First, the DTW algorithm is used to calculate the similarity between the normal distribution curve and the probability density curve, and the The interval is recorded as the judgment interval; then, the standard deviation variable is obtained according to the similarity between the normal distribution curve and the probability density curve, the judgment interval, and the preset similarity threshold; and the standard deviation variable is the degree to which the judgment interval is narrowed when the similarity between the normal distribution curve and the probability density curve is judged to be less than the similarity threshold. Therefore, the formula for obtaining the standard deviation variable is:

[0034] in, is the standard deviation variable, is the standard deviation of the eigenvalue sequence, The preset similarity threshold, is the similarity between the normal distribution curve and the probability density curve.

[0035] In addition, when the similarity between the normal distribution curve and the probability density curve is smaller, The smaller the value of , the greater the degree of reduction of the subsequent judgment interval; when the similarity between the normal distribution curve and the probability density curve is greater, The larger the value of , the smaller the subsequent reduction of the judgment interval.

[0036] Since the process of obtaining the similarity between the normal distribution curve and the probability density curve using the DTW algorithm is a well-known technology, it will not be described in detail in this embodiment; and in specific applications, it is necessary to set a preset similarity threshold according to actual conditions. For example, in this embodiment, the preset similarity threshold is set to 0.85.

[0037] Then, it is determined whether the similarity between the normal distribution curve and the probability density curve is greater than or equal to the preset similarity threshold; when the similarity between the normal distribution curve and the probability density curve is greater than or equal to the similarity threshold, it indicates that the eigenvalue sequence is more consistent with the normal distribution, which also indicates that the eigenvalue sequence is more consistent with the normal distribution. Accurate, that is, the probability of suspected drift abnormal data appearing in the network data sequence to be processed is small, so when the similarity between the normal distribution curve and the probability density curve is greater than or equal to the preset similarity threshold, the interval is judged to be Recorded as the first target judgment interval, and in the normal distribution curve corresponding to the eigenvalue sequence, the eigenvalue sequence in the first target judgment interval The network data corresponding to all eigenvalues ​​other than are recorded as suspected drift anomaly data.

[0038] When the similarity between the normal distribution curve and the probability density curve is less than the preset similarity threshold, it indicates that the eigenvalue sequence does not completely conform to the normal distribution. The narrowed judgment interval is used to accurately and reliably obtain suspected drift abnormal data. Therefore, when the similarity between the normal distribution curve and the probability density curve is determined to be less than the preset similarity threshold, the second target judgment interval is obtained according to the standard deviation variable, and in the normal distribution curve corresponding to the eigenvalue sequence, the network data corresponding to all eigenvalues ​​in the eigenvalue sequence that are outside the second target judgment interval are recorded as suspected drift abnormal data.

[0039] The second target judgment interval in this embodiment is , that is, the minimum value of the second target judgment interval is the value obtained by subtracting the mean of the eigenvalue sequence from the standard deviation of the eigenvalue sequence and the sum of the standard deviation variable, and the maximum value of the second target judgment interval is the value obtained by adding the mean of the eigenvalue sequence from the standard deviation of the eigenvalue sequence and the standard deviation variable.

[0040] At this point, all suspected drift anomaly data in the network data sequence to be processed are obtained.

[0041] Step S003: Obtain the first data window corresponding to each suspected drift abnormal data and the second data window corresponding to each suspected drift abnormal data; obtain the optimal second data window corresponding to each suspected drift abnormal data based on the difference between the eigenvalue corresponding to each network data in each second data window corresponding to each suspected drift abnormal data and the mean value corresponding to the eigenvalue sequence; obtain each drift abnormal data in the suspected drift abnormal data based on the first data window and the optimal second data window corresponding to each suspected drift abnormal data.

[0042] Since the gradual data drift phenomenon in the network data sequence to be processed will affect the anomaly detection result, that is, the presence of local upward or local downward data offset in the network data sequence to be processed will affect the anomaly detection result, if the anomaly detection is directly performed on the network data sequence to be processed, the accuracy of the anomaly detection will be relatively low. Because when the network data sequence to be processed is subjected to anomaly data detection, the offset data part in the network data sequence to be processed is easily confused with the real anomaly data, noise data or mutation data, that is, the suspected drift anomaly data obtained in step S002 may also contain drift anomaly data, real anomaly data, noise data and mutation data, and the network data to be processed in this embodiment is not necessarily the same as the real anomaly data. The reason why noise data exists in the network data sequence to be processed is that this embodiment does not perform data cleaning on the network data sequence to be processed to eliminate noise. In addition, because the data form and trend of the detected abnormal offset data are not abnormal, that is, the detected abnormal offset data will not change the trend, while real abnormal data, noise data or mutation data will change the trend, this embodiment will analyze all suspected drift abnormal data in the obtained network data sequence to be processed based on the characteristics that abnormal offset data will not change the trend, while real abnormal data, noise data or mutation data will change the trend, and obtain each drift abnormal data in each suspected drift abnormal data; the specific process is as follows: Taking each network data in the network data sequence to be processed as the window center point, a window corresponding to each network data in the network data sequence to be processed is constructed, and the size of the constructed window is , that is, the number of network data in the window corresponding to each network data is m, and in specific applications, the value of m needs to be set according to actual conditions. In this embodiment, the value of m is set to 19.

[0043] For any suspected drift abnormal data in the network data sequence to be processed: The window corresponding to the suspected drift abnormal data is recorded as the first data window corresponding to the suspected drift abnormal data, and the windows corresponding to all network data in the to-be-processed network data sequence except the suspected drift abnormal data are recorded as the second data window corresponding to the suspected drift abnormal data.

[0044] Therefore, this embodiment obtains each second data window corresponding to the suspected drift abnormal data. Next, this embodiment will analyze the first data window corresponding to the suspected drift abnormal data and each second data window corresponding to the suspected drift abnormal data to obtain the optimal second data window corresponding to the suspected drift abnormal data. The purpose of obtaining the optimal second data window corresponding to the suspected drift abnormal data in this embodiment is to determine the probability that the suspected drift abnormal data is drift abnormal data, and subsequently, the replacement value corresponding to the corresponding drift abnormal data will be obtained based on the optimal second data window corresponding to the drift abnormal data. Because when the second data window corresponding to the suspected drift abnormal data is closer to the first data window corresponding to the suspected drift abnormal data, it indicates that the second data window and the first data window correspond to the change trend of the network data subsequence. That is, the probability that the corresponding second data window is the optimal second data window is greater, and the more normal network data in the second data window, the greater the credibility of the corresponding second data window. That is, in the normal distribution curve, the characteristic value corresponding to each network data in each second data window corresponding to the suspected drift abnormal data is closer to the mean. , the greater the credibility of the corresponding second data window, that is, the greater the probability that the corresponding second data window is the optimal second data window. Therefore, this embodiment obtains the preference degree of each second data window corresponding to the suspected drift abnormal data based on the characteristics of the above analysis: Each network data is marked according to its position in the sequence of network data to be processed to obtain a mark value of each network data in the sequence of network data to be processed; that is, the mark value of the a-th network data in the sequence of network data to be processed is a.

[0045] For any second data window corresponding to the suspected drift abnormal data: In the normal distribution curve, obtain the characteristic value and mean corresponding to each network data in the second data window The absolute value of the difference between the two values ​​is recorded as the initial characterization value corresponding to the corresponding network data in the second data window, and the initial characterization value corresponding to each network data in the second data window is negatively correlated and mapped, and the initial characterization value after the negative correlation mapping is recorded as the target characterization value corresponding to the corresponding network data in the second data window, and the average of the target characterization values ​​corresponding to all network data in the second data window is recorded as the first preference level corresponding to the second data window; the absolute value of the difference between the label value corresponding to the central network data in the second data window and the label value corresponding to the central network data in the first data window corresponding to the suspected drift abnormal data is calculated, and the absolute value of the obtained difference is negatively correlated and mapped, and the absolute value of the difference after the negative correlation mapping is recorded as the second preference level corresponding to the second data window; the product of the first preference level corresponding to the second data window and the second preference level corresponding to the second data window is recorded as the preference level corresponding to the second data window; the preference level corresponding to the second data window is calculated according to the following formula:

[0046] in, is the preference level corresponding to the second data window, is the eigenvalue corresponding to the i-th network data in the second data window, is the mean corresponding to the eigenvalue sequence, The amount of network data in the second data window, It is the absolute value of the difference between the label value corresponding to the central network data in the second data window and the label value corresponding to the central network data in the first data window corresponding to the suspected drift abnormal data, and exp() is an exponential function with the natural constant e as the base.

[0047] In addition, when The larger the value of , the greater the probability that the second data window is the optimal second data window corresponding to the suspected drift abnormal data; on the contrary, when The smaller the value of , the smaller the probability that the second data window is the optimal second data window corresponding to the suspected drift abnormal data; and when The bigger, The bigger it is, The smaller the value of The smaller, The smaller the time, The larger the value of .

[0048] Therefore, through the above process, the preference degree of each second data window corresponding to the suspected drift abnormal data is obtained, and the second data window corresponding to the maximum preference degree among the second data windows corresponding to the suspected drift abnormal data is recorded as the optimal second data window corresponding to the suspected drift abnormal data.

[0049] Therefore, this embodiment can obtain the first data window and the corresponding optimal second data window corresponding to each suspected drift abnormal data in the network data sequence to be processed through the above process of obtaining the optimal second data window corresponding to the suspected drift abnormal data.

[0050] Then, the DTW algorithm is used to calculate the similarity between the first data window corresponding to each suspected drift abnormal data and its corresponding optimal second data window, and record it as the first similarity corresponding to the suspected drift abnormal data; then, the central network data in the first data window corresponding to each suspected drift abnormal data and its corresponding optimal second data window are removed, and the first data window and the optimal second data window after removal are recorded as the first feature data window and the optimal second feature data window corresponding to the suspected drift abnormal data, and then the DTW algorithm is used to calculate the similarity between the first feature data window corresponding to each suspected drift abnormal data and its corresponding optimal second feature data window, and record it as the second similarity corresponding to the suspected drift abnormal data.

[0051] Then, the absolute value of the difference between the first similarity corresponding to each suspected drift abnormal data and its corresponding second similarity is calculated, and the absolute value of the difference between the first similarity corresponding to each suspected drift abnormal data and its corresponding second similarity is recorded as the judgment indicator corresponding to the corresponding suspected drift abnormal data.

[0052] And when the absolute value of the difference between the first similarity corresponding to each suspected drift abnormal data and its corresponding second similarity is smaller, it indicates that the trend of the data window without removing the central network data is similar to that of the data window with the central network data removed, that is, the influence of the suspected drift abnormal data on the similarity is not great; and when the absolute value of the difference between the first similarity corresponding to each suspected drift abnormal data and its corresponding second similarity is larger, it indicates that the trend of the data window without removing the central network data and the data window with the central network data removed is dissimilar, that is, the influence of the suspected drift abnormal data on the similarity is greater; and because of the progressive abnormal offset There is nothing abnormal in the shape and trend of some data, that is, the anomaly caused by gradual drift will not change the trend, but the anomaly of the data itself will change the trend. Therefore, when the value of the judgment index corresponding to the suspected drift abnormal data is smaller, the probability that the corresponding suspected drift abnormal data is caused by gradual drift is greater, that is, the probability that the corresponding suspected drift abnormal data is drift abnormal data is greater; when the value of the judgment index corresponding to the suspected drift abnormal data is larger, the probability that the corresponding suspected drift abnormal data is caused by gradual drift is smaller, that is, the probability that the corresponding abnormal data is an anomaly of the data itself, noise data or mutation data is greater.

[0053] Therefore, the judgment index corresponding to each suspected drift abnormal data is negatively correlated and mapped, and the judgment index after negative correlation mapping is marked as the target judgment index corresponding to the suspected drift abnormal data; that is, for any suspected drift abnormal data, Denote as the target judgment index corresponding to the suspected drift abnormal data, where is the first similarity corresponding to the suspected drift abnormal data, is the second similarity corresponding to the suspected drift abnormal data; then, it is determined whether the target judgment index corresponding to each suspected drift abnormal data is greater than or equal to the judgment threshold. If so, it indicates that the corresponding abnormal data belongs to the abnormality caused by gradual drift, and the corresponding suspected drift abnormal data is recorded as drift abnormal data. Otherwise, it indicates that the corresponding suspected drift abnormal data does not belong to the abnormality caused by gradual drift, and the greater the probability that the corresponding suspected drift abnormal data is the abnormality of the data itself, noise data or mutation data.

[0054] In specific applications, the judgment threshold needs to be set according to actual conditions. In this embodiment, the judgment threshold is set to 0.82.

[0055] Therefore, through the above process, each drift abnormal data in each suspected drift abnormal data is obtained, that is, each drift abnormal data in the network data sequence to be processed is obtained.

[0056] Step S004: obtaining a replacement value corresponding to each drift abnormal data according to the optimal second data window corresponding to each drift abnormal data; replacing each drift abnormal data with the replacement value of the corresponding drift abnormal data, and recording the replaced network data sequence to be processed as the target network data sequence.

[0057] Since each drift abnormal data in the to-be-processed network data sequence is obtained in step S003, this embodiment will then process each drift abnormal data in the to-be-processed network data sequence to obtain a target network data sequence, specifically: First, a sequence constructed from all network data in the optimal second data window corresponding to each drift abnormal data is recorded as the sub-network data sequence corresponding to the optimal second data window; the sub-network data sequence corresponding to the optimal second data window corresponding to each drift abnormal data is input into the fitted prediction model to obtain the predicted value corresponding to the optimal second data window corresponding to each drift abnormal data, and recorded as the replacement value corresponding to the corresponding drift abnormal data; then, each drift abnormal data is replaced with the replacement value of the corresponding drift abnormal data, and the replaced network data sequence to be processed is recorded as the target network data sequence.

[0058] In specific applications, it is necessary to select a prediction model based on actual conditions. The prediction model used in this embodiment is the ARIMA model. In this embodiment, the sub-network data sequence corresponding to the optimal second data window is input into the prediction model, and the prediction model outputs a prediction value. Since there are some data between each drift abnormal data and the sub-network data sequence corresponding to the optimal second data window corresponding to the corresponding drift abnormal data, these data are used as verification data, that is, the prediction value output by the model is compared with it, so as to perform feedback adjustment and adjust the parameters of the prediction model until suitable parameters are selected to make the prediction value closer to the true value. When the model adjustment is completed, the prediction value of the corresponding position of the abnormal data point is output by prediction, and the prediction value is used as the replacement value.

[0059] Since the fitting process and prediction process of the ARIMA model are well-known technologies, they will not be described in detail in this embodiment.

[0060] Then, the target network data sequence is cleaned to obtain a cleaned target network data sequence; the data cleaning can clean the noise data and redundant data in the target network data sequence; then, anomaly detection is performed on the cleaned target network data sequence using an anomaly detection algorithm to obtain abnormal data in the cleaned target network data sequence. In addition, this embodiment performs anomaly detection on the network data sequence after processing the drift abnormal data, which can make the detection result more accurate.

[0061] This embodiment uses a static cleaning strategy to cleanse the communication network data sequence to be cleaned. Since the specific data cleaning process is well-known, this embodiment will not be described in detail. In specific applications, an anomaly detection algorithm can be selected based on actual circumstances. This embodiment uses the isolation forest algorithm to detect anomalies in the target network data sequence, obtain anomaly data within the target network data sequence, and complete the processing of the communication network data by marking the anomaly data. Since the isolation forest algorithm is well-known, this embodiment will not be described in detail.

[0062] This embodiment first obtains a network data sequence to be processed; then, based on the differences between adjacent network data in the network data sequence to be processed, obtains a eigenvalue sequence corresponding to the network data sequence to be processed, and obtains each suspected drift anomaly data in the network data sequence to be processed based on the similarity between the normal distribution curve and the probability density curve corresponding to the eigenvalue sequence; then, based on the differences between the eigenvalues ​​corresponding to each network data in each second data window corresponding to each suspected drift anomaly data and the mean value corresponding to the eigenvalue sequence, obtains the optimal second data window corresponding to each suspected drift anomaly data, and based on the first data window and the optimal second data window corresponding to each suspected drift anomaly data, obtains each drift anomaly data in each suspected drift anomaly data; finally, based on the optimal second data window corresponding to each drift anomaly data, obtains a replacement value corresponding to each drift anomaly data, and replaces each drift anomaly data with the replacement value of the corresponding drift anomaly data to obtain the replaced network data sequence to be processed. This embodiment can more accurately obtain the drift anomaly data in the network data sequence to be processed, and eliminate the influence of the drift anomaly data on the detection accuracy during anomaly detection, thereby improving the detection accuracy.

[0063] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for efficiently processing data in a communication network, characterized in that: The method comprises the following steps: Obtain a sequence of network data to be processed; the sequence of network data to be processed is composed of B network data, where B is greater than 0; Obtaining a characteristic value sequence corresponding to the network data sequence to be processed based on the differences between adjacent network data in the network data sequence to be processed; obtaining a normal distribution curve and a probability density curve corresponding to the characteristic value sequence based on the mean and standard deviation corresponding to the characteristic value sequence; and obtaining each suspected drift abnormal data in the network data sequence to be processed based on the similarity between the normal distribution curve and the probability density curve; Obtaining first data windows corresponding to each suspected drift abnormal data and second data windows corresponding to each suspected drift abnormal data, and obtaining optimal second data windows corresponding to each suspected drift abnormal data based on differences between eigenvalues ​​corresponding to each network data in each second data window corresponding to each suspected drift abnormal data and means corresponding to the eigenvalue sequences; Obtaining each drift abnormality data in each suspected drift abnormality data according to the first data window and the optimal second data window corresponding to each suspected drift abnormality data; Obtaining a replacement value corresponding to each drift abnormal data according to the optimal second data window corresponding to each drift abnormal data; Each drift abnormal data is replaced with a replacement value of the corresponding drift abnormal data, and the network data sequence to be processed after the replacement is recorded as a target network data sequence.

2. A method for efficiently processing data in a communication network according to claim 1, characterized in that: The method for obtaining a eigenvalue sequence corresponding to a network data sequence to be processed includes: For any network data in the sequence of network data to be processed: Recording the difference between the network data adjacent to and following the network data in the sequence of network data to be processed and the network data as the first difference corresponding to the network data; Performing smoothing on the first difference value corresponding to the network data, and recording the first difference value after the smoothing as the second difference value corresponding to the network data; Subtracting the first difference value corresponding to the network data from the second difference value corresponding to the network data, and recording the value after the subtraction as the characteristic value corresponding to the network data; The sequence constructed by the characteristic values ​​corresponding to each network data in the network data sequence to be processed is recorded as the characteristic value sequence corresponding to the network data sequence to be processed.

3. A method for efficiently processing data in a communication network according to claim 2, characterized in that: The method for obtaining each suspected drift abnormal data in the network data sequence to be processed includes: Using the DTW algorithm, the similarity between the normal distribution curve and the probability density curve is calculated; Determine whether the similarity between the normal distribution curve and the probability density curve is greater than or equal to a preset similarity threshold. If so, the network data corresponding to all characteristic values ​​outside the first target judgment interval are recorded as suspected drift abnormal data; otherwise, the network data corresponding to all characteristic values ​​outside the second target judgment interval are recorded as suspected drift abnormal data.

4. A method for efficiently processing data in a communication network according to claim 3, characterized in that: The minimum value of the first target judgment interval is the value obtained by subtracting the mean of the eigenvalue sequence from three times the standard deviation of the eigenvalue sequence, and the maximum value of the first target judgment interval is the value obtained by adding the mean of the eigenvalue sequence to three times the standard deviation of the eigenvalue sequence.

5. The method for efficiently processing data in a communication network according to claim 3, wherein: The minimum value of the second target judgment interval is the mean of the eigenvalue sequence minus the sum of the standard deviation variable and the standard deviation of the eigenvalue sequence, and the maximum value of the second target judgment interval is the sum of the mean of the eigenvalue sequence, the standard deviation variable and the standard deviation of the eigenvalue sequence.

6. The method for efficiently processing data in a communication network according to claim 5, wherein: Methods for obtaining standard deviation variables include: The similarity between the normal distribution curve and the probability density curve is multiplied by twice the standard deviation of the feature value sequence, and the ratio of the multiplied value to a preset similarity threshold is recorded as the standard deviation variable.

7. The method for efficiently processing data in a communication network according to claim 1, wherein: The method for obtaining the first data window corresponding to each suspected drift abnormal data and the second data window corresponding to each suspected drift abnormal data includes: Taking each network data in the sequence of network data to be processed as a window center point, constructing a window corresponding to each network data in the sequence of network data to be processed; For any suspected drift abnormal data in the network data sequence to be processed: the window corresponding to the suspected drift abnormal data is recorded as the first data window corresponding to the suspected drift abnormal data, and the windows corresponding to all network data in the network data sequence to be processed except the suspected drift abnormal data are recorded as the second data window corresponding to the suspected drift abnormal data.

8. The method for efficiently processing data in a communication network according to claim 1, wherein: The method for obtaining the optimal second data window corresponding to each suspected drift abnormal data includes: For any second data window corresponding to any suspected drift abnormal data in the network data sequence to be processed: Recording the absolute value of the difference between the characteristic value corresponding to each network data in the second data window and the mean value corresponding to the characteristic value sequence as the initial characterization value corresponding to the corresponding network data in the second data window; Performing negative correlation mapping on the initial characterization values ​​corresponding to the network data in the second data window, recording the initial characterization values ​​after the negative correlation mapping as target characterization values, and recording the average of the target characterization values ​​corresponding to all the network data in the second data window as the first preference level corresponding to the second data window; Performing negative correlation mapping on the absolute value of the difference between the label value corresponding to the central network data in the second data window and the label value corresponding to the central network data in the first data window corresponding to the suspected drift abnormal data, recording the absolute value of the difference after the negative correlation mapping as the second preference level corresponding to the second data window, and the label value of the a-th network data in the network data sequence to be processed is a; The product of the first preference level corresponding to the second data window and the second preference level corresponding to the second data window is recorded as the preference level corresponding to the second data window; The maximum preferred degree among the preferred degrees corresponding to the second data windows corresponding to the suspected drift abnormal data is obtained, and the second data window corresponding to the maximum preferred degree is recorded as the optimal second data window corresponding to the suspected drift abnormal data.

9. The method for efficiently processing data in a communication network according to claim 1, wherein: The method for obtaining each drift abnormality data in the each suspected drift abnormality data includes: The similarity between the first data window corresponding to each suspected drift abnormal data and the corresponding optimal second data window calculated by the DTW algorithm is recorded as the first similarity corresponding to the suspected drift abnormal data; Removing the central network data from the first data window corresponding to each suspected drift abnormal data and the corresponding optimal second data window, and recording the first data window and the optimal second data window after the removal as the first characteristic data window and the optimal second characteristic data window corresponding to the suspected drift abnormal data; The similarity between the first feature data window corresponding to each suspected drift abnormal data and its corresponding optimal second feature data window calculated using the DTW algorithm is recorded as the second similarity corresponding to the suspected drift abnormal data; Performing negative correlation mapping on the absolute value of the difference between the calculated first similarity corresponding to each suspected drift abnormal data and the calculated second similarity, and recording the value after the negative correlation mapping as the target judgment indicator corresponding to the suspected drift abnormal data; It is determined whether the target judgment index corresponding to each suspected drift abnormal data is greater than or equal to a judgment threshold. If so, the corresponding suspected drift abnormal data is recorded as drift abnormal data.

10. The method for efficiently processing data in a communication network according to claim 1, wherein: The method for obtaining the replacement value corresponding to each drift abnormal data includes: Recording a sequence constructed from all network data in the optimal second data window corresponding to each drift abnormal data as a sub-network data sequence corresponding to the optimal second data window; Inputting the subnetwork data sequence corresponding to the optimal second data window corresponding to each drift abnormal data into the fitted prediction model to obtain the predicted value corresponding to the optimal second data window corresponding to each drift abnormal data; The predicted value corresponding to the optimal second data window corresponding to each drift abnormal data is recorded as the replacement value corresponding to the corresponding drift abnormal data.

Images