A method and device for processing network data flow

Abstract

A method and device for processing network data traffic. The method includes: obtaining a time domain sequence of network data traffic; dividing the time domain sequence into at least two traffic slices; determining suspected abnormal traffic values ​​in at least two traffic slices based on the Gaussian distribution of the traffic value of the network data; determining a first fitting value corresponding to the suspected abnormal traffic value; determining an abnormal traffic value in the suspected abnormal traffic value based on a first model; determining a second fitting value of the abnormal traffic value based on a second model; and performing abnormal recovery on the abnormal traffic value according to the first fitting value and the second fitting value of the abnormal traffic value. In this way, the universality of network data traffic processing is improved, the accuracy of network data traffic processing is increased, and the processing efficiency of network data traffic is improved.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a method and device for processing network data traffic. Background Art

[0002] With the continuous advancement of the network, more and more data is generated in the network. In order to better manage the traffic generated by network data, it is necessary to collect the traffic value of the target object's network data, and then implement traffic billing and traffic profiling operations on the target object's network data. Among them, the target object can be an IP (Internet Protocol Address), a geographical area, an APP (application), etc.

[0003] However, network data traffic values ​​often suffer from traffic value loss, traffic value anomalies, etc. In order to ensure the accuracy of traffic billing, traffic profiling, etc., it is necessary to perform anomaly detection and anomaly recovery on the network data traffic values.

[0004] Therefore, how to improve the accuracy of network data flow processing and improve processing efficiency is a problem that needs to be solved at present. Summary of the invention

[0005] The embodiments of the present application provide a method and device for processing network data traffic, which are used to implement network data traffic processing in various scenarios, ensure the universality of network data traffic processing, improve the accuracy of network data traffic processing, and enhance the processing efficiency of network data traffic.

[0006] In the first aspect, the present application proposes a method for processing network data traffic, the method comprising: obtaining a time domain sequence of network data traffic; the time domain sequence comprises traffic values ​​of network data collected at set time intervals; dividing the time domain sequence into at least two traffic slices; determining an abnormal traffic slice among the at least two traffic slices based on the Gaussian distribution of the traffic values ​​of the network data, the abnormal traffic slice comprising at least one suspected abnormal traffic value; determining a first fitting value corresponding to the at least one suspected abnormal traffic value; determining an abnormal traffic value among the at least one suspected abnormal traffic value based on a first model; determining a second fitting value of the abnormal traffic value based on a second model; and performing abnormal recovery on the abnormal traffic value according to the first fitting value and the second fitting value of the abnormal traffic value.

[0007] Based on the above scheme, the present application analyzes the flow value of network data in any flow slice based on Gaussian distribution, and then determines the abnormal flow slice from at least two flow slices, and determines the suspected abnormal flow value in the abnormal flow slice. Because Gaussian distribution can be applied to various scenarios, the universality of network data flow processing is guaranteed.

[0008] In the present application, a suspected abnormal flow value indicates that the flow value may be an abnormal flow value, rather than being finally determined to be an abnormal flow value. After determining the suspected abnormal flow value, it is determined whether the suspected abnormal flow value is an abnormal flow value based on the first model. This is equivalent to the present application including two steps when determining abnormal flow; the first step is to determine the suspected abnormal flow based on Gaussian distribution; the second step is to determine the abnormal flow from the suspected abnormal flow based on the first model. In this way, the error rate of determining abnormal flow values ​​based on Gaussian distribution is reduced, thereby improving the accuracy of abnormal flow detection.

[0009] After determining the abnormal traffic value, the first fitting value of the abnormal traffic is determined based on the Gaussian distribution, and the second fitting value of the abnormal traffic is determined based on the second model. The interpolation algorithm is no longer used, the amount of calculation is reduced, the calculation efficiency of the fitting value is improved, the recovery efficiency of the abnormal traffic is improved, and the processing efficiency of the network data traffic is improved. When abnormal recovery of the abnormal traffic is performed, it is implemented according to the first fitting value and the second fitting value of the abnormal traffic, which increases the credibility of the abnormal recovery, improves the accuracy of the abnormal recovery, and thus improves the accuracy of the network data traffic processing.

[0010] In some embodiments, the time domain sequence is divided into at least two traffic slices, including: converting the time domain sequence into a frequency domain sequence, the frequency domain sequence being used to represent the characteristic that the amplitude value of the network data traffic in the frequency domain changes with the frequency; determining the statistical interval time length corresponding to the traffic slice according to the frequency domain sequence; dividing the time domain sequence into at least two traffic slices according to the statistical interval time length, each of the at least two traffic slices including the traffic value of the network data collected within the statistical interval time length.

[0011] Based on the above scheme, when splitting the time domain sequence, the present application first converts the time domain sequence into a frequency domain sequence, and then determines the length of the statistical interval time according to the frequency domain sequence. Among them, because the frequency domain sequence can reflect the network data traffic in the time domain sequence through the characteristic that the amplitude value changes with the frequency, such as reflecting that the traffic value of the network data used from 11:00 to 12:00 in a day is greater than the traffic value of the network data used from 8:00 to 9:00 in a day, so according to the frequency domain sequence, the length of the statistical interval time used to split the time domain sequence is determined, which can increase the credibility and rationality of the time domain sequence segmentation, and thus increase the accuracy of any traffic fragment abnormality detection after the time domain sequence segmentation.

[0012] In some embodiments, determining the statistical interval time length corresponding to the traffic slice according to the frequency domain sequence includes: determining the ratio of the maximum amplitude value to the first value in the frequency domain sequence to obtain an amplitude threshold; determining a frequency threshold according to the amplitude threshold, so that the amplitude value corresponding to the frequency value greater than the frequency threshold is less than the amplitude threshold; and using the inverse of the frequency threshold as the statistical interval time length corresponding to the traffic slice.

[0013] Based on the above scheme, when calculating the length of the statistical interval time, the present application determines the frequency range whose amplitude value is less than the amplitude threshold based on the amplitude threshold, and then uses the minimum absolute value in the frequency range as the frequency threshold. The amplitude value corresponding to the frequency threshold is close to the average amplitude value of the frequency domain sequence. Therefore, by determining the length of the statistical interval time according to the frequency threshold, the credibility and rationality of the time domain sequence segmentation can be increased, thereby increasing the accuracy of anomaly detection of any traffic segment after the time domain sequence segmentation.

[0014] In some embodiments, determining an abnormal traffic slice among the at least two traffic slices based on the Gaussian distribution of the traffic values ​​of the network data includes: determining a normal traffic value range corresponding to each traffic slice among the at least two traffic slices based on the Gaussian distribution of the traffic values ​​of the network data; if any traffic value in a first traffic slice among the at least two traffic slices is outside the normal traffic value range corresponding to the first traffic slice, determining the first traffic slice as an abnormal traffic slice; and / or, if a second traffic slice among the at least two traffic slices lacks a traffic value, determining the second traffic slice as an abnormal traffic slice.

[0015] Based on the above scheme, abnormal traffic generally includes two situations; one is abnormal traffic value, and the other is missing traffic value. Among them, abnormal traffic value means that the traffic value corresponding to a certain time point in the time domain sequence is too large or too small; missing traffic value means that there is no traffic value at a certain time point in the time domain sequence, or the traffic value corresponding to the time point is 0. Therefore, after obtaining at least two traffic slices, the abnormal traffic slice can be determined from the at least two traffic slices according to the judgment basis of abnormal traffic. Then, subsequent data processing for abnormal traffic slices is realized, reducing the amount of data processing and improving data processing efficiency.

[0016] In some embodiments, before determining the first fitting value corresponding to the at least one suspected abnormal traffic value, it also includes: if the second traffic slice of the at least two traffic slices lacks a traffic value, adding the suspected abnormal traffic value to the second traffic slice.

[0017] Based on the above scheme, in order to ensure the accuracy of abnormal traffic segment detection, the present application adds suspected abnormal traffic values ​​in the abnormal traffic segment so that the number of traffic values ​​in the second traffic segment is consistent with the number of traffic values ​​in other traffic segments.

[0018] In some embodiments, determining the first fitting value corresponding to the at least one suspected abnormal traffic value includes: for the suspected abnormal traffic value in any abnormal traffic slice, taking the traffic mean of the abnormal traffic slice as the first fitting value corresponding to the suspected abnormal traffic value.

[0019] Based on the above scheme, this application uses the traffic mean of abnormal traffic slices as an abnormal recovery factor for abnormal traffic values, so as to avoid the single abnormal recovery condition for abnormal traffic values, increase the credibility of abnormal recovery, improve the accuracy of abnormal recovery, and thus improve the accuracy of network data traffic processing.

[0020] In some embodiments, performing abnormal recovery on the abnormal flow value based on the first fitting value and the second fitting value of the abnormal flow value includes: determining a flow recovery value of the abnormal flow value based on the first fitting value of the abnormal flow value and the weight corresponding to the first fitting value, and the second fitting value of the abnormal flow value and the weight corresponding to the second fitting value; and performing abnormal recovery on the abnormal flow value based on the flow recovery value of the abnormal flow value.

[0021] Based on the above scheme, the abnormal flow value is determined from the suspected abnormal flow value, so the first fitting value of the abnormal flow value is the first fitting value of the suspected abnormal flow value corresponding to the abnormal flow value. The abnormal recovery of the abnormal flow value is achieved through the first fitting value and the second fitting value of the abnormal flow value, which increases the credibility of the abnormal recovery, improves the accuracy of the abnormal recovery, and thus improves the accuracy of network data flow processing.

[0022] In the second aspect, an embodiment of the present application proposes a device for processing network data traffic, the device comprising: an acquisition module, used to acquire a time domain sequence of network data traffic; the time domain sequence comprises traffic values ​​of network data collected at set time intervals; a processing module, used to divide the time domain sequence into at least two traffic slices; based on the Gaussian distribution of the traffic values ​​of the network data, determining an abnormal traffic slice in the at least two traffic slices, the abnormal traffic slice comprising at least one suspected abnormal traffic value; determining a first fitting value corresponding to the at least one suspected abnormal traffic value; determining an abnormal traffic value in the at least one suspected abnormal traffic value based on a first model; determining a second fitting value of the abnormal traffic value based on a second model; and performing abnormal recovery on the abnormal traffic value according to the first fitting value and the second fitting value of the abnormal traffic value.

[0023] In some embodiments, the processing module is specifically used to: convert the time domain sequence into a frequency domain sequence, the frequency domain sequence being used to represent the characteristic of the amplitude value of the network data traffic in the frequency domain changing with the frequency; determine the statistical interval time length corresponding to the traffic slice according to the frequency domain sequence; divide the time domain sequence into at least two traffic slices according to the statistical interval time length, each of the at least two traffic slices including the traffic value of the network data collected within the statistical interval time length.

[0024] In some embodiments, the processing module is specifically used to: determine the ratio of the maximum amplitude value to the first value in the frequency domain sequence to obtain an amplitude threshold; determine a frequency threshold based on the amplitude threshold, so that the amplitude value corresponding to the frequency value greater than the frequency threshold is less than the amplitude threshold; and use the inverse of the frequency threshold as the statistical interval time length corresponding to the traffic segment.

[0025] In some embodiments, the processing module is specifically used to: determine the normal traffic value range corresponding to each of the at least two traffic slices based on the Gaussian distribution of the traffic value of the network data; if any traffic value in a first traffic slice of the at least two traffic slices is outside the normal traffic value range corresponding to the first traffic slice, then determine the first traffic slice as an abnormal traffic slice; and / or, if a second traffic slice of the at least two traffic slices lacks a traffic value, then determine the second traffic slice as an abnormal traffic slice.

[0026] In some embodiments, the processing module is specifically used to: before determining the first fitting value corresponding to the at least one suspected abnormal traffic value, if the second traffic slice of the at least two traffic slices is missing a traffic value, then add the suspected abnormal traffic value to the second traffic slice.

[0027] In some embodiments, the processing module is specifically used to: for any suspected abnormal traffic value in any abnormal traffic slice, use the traffic mean of the abnormal traffic slice as the first fitting value corresponding to the suspected abnormal traffic value.

[0028] In some embodiments, the processing module is specifically used to: determine the flow recovery value of the abnormal flow value based on the first fitting value of the abnormal flow value and the weight corresponding to the first fitting value, as well as the second fitting value of the abnormal flow value and the weight corresponding to the second fitting value; and perform abnormal recovery on the abnormal flow value based on the flow recovery value of the abnormal flow value.

[0029] In a third aspect, an embodiment of the present application provides a device, comprising: a memory for storing program instructions; the program instructions are used to implement the method described in the first aspect above.

[0030] In a fourth aspect, an embodiment of the present application provides a device, comprising: one or more processors; wherein, when instructions of one or more computer programs are executed by the one or more processors, the device executes the method described in the first aspect above.

[0031] In a fifth aspect, an embodiment of the present application provides a readable storage medium, wherein the readable storage medium stores a program, and when the program code runs on a device, the device executes the method described in the first aspect above.

[0032] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the first aspect above.

[0033] In the seventh aspect, an embodiment of the present application provides a chip system, comprising at least one chip and a memory, wherein the at least one chip is coupled to the memory and is used to read and execute program instructions stored in the memory to implement the method described in the first aspect above.

[0034] Based on the implementations provided in the above aspects, the embodiments of the present application can be further combined to provide more implementations.

[0035] The technical effects that can be achieved in any of the second to seventh aspects mentioned above can be referred to the description of the technical effects that can be achieved in the first aspect mentioned above, and the repeated parts will not be discussed here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a normal distribution provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a box plot algorithm provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0039] Figure 4 A flowchart of a method for processing network data traffic provided by an embodiment of the present application;

[0040] Figure 5 A schematic diagram of a time domain sequence provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of a frequency domain sequence provided in an embodiment of the present application;

[0042] Figure 7 A schematic diagram of monitoring network data traffic based on Gaussian distribution provided in an embodiment of the present application;

[0043] Figure 8 A schematic diagram of a missing flow value detection provided in an embodiment of the present application;

[0044] Fig. 9 A schematic diagram of determining an abnormal flow value based on a first model provided in an embodiment of the present application;

[0045] Fig.10 A schematic diagram of determining a second fitting value based on a second model provided in an embodiment of the present application;

[0046] Fig.11 A schematic diagram of a method for processing network data traffic provided by an embodiment of the present application;

[0047] Fig.12 A schematic diagram of determining a second fitting value based on a second model provided in an embodiment of the present application;

[0048] Fig.13 A schematic diagram of a method for processing network data traffic provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0050] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0051] The multiple involved in the embodiments of the present application means greater than or equal to two. It is understood that in the description of the embodiments of the present application, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0052] In order to better explain the method for processing network data traffic provided by the embodiments of the present application, the concepts and terms involved in the embodiments of the present application are first briefly explained.

[0053] Gaussian distribution: also known as normal distribution. Used to calculate the probability value of a data set.

[0054] Discrete Fourier Transform (DFT): used to transform data from the time domain to the frequency domain, and then study the frequency domain structure and frequency domain change law of the data.

[0055] Fast Fourier transform (FFT): A method to improve the computational efficiency of discrete Fourier transform by reducing the number of multiplications required by the computer to calculate the discrete Fourier transform.

[0056] Traffic billing: Billing is based on the traffic value of network data generated after accessing the network.

[0057] Traffic profile: a profile used to represent the traffic value of network data; such as histogram profile, bar chart profile, line chart profile, etc.

[0058] Autoencoder (AE): A self-supervised neural network model based on Encoder-Decoder.

[0059] Transformer model: a time series model based on self-attention.

[0060] MB: The unit of network data traffic.

[0061] In order to perform traffic billing and traffic profiling for each service, IP address, APP and other target objects in the network business flow, it is necessary to collect the network data traffic generated by the target objects.

[0062] Taking the IP address as the target object, the traffic value of the network data generated by the mobile terminal with the IP address "10.5.***.***" in 72 hours is collected, and then the fee that the mobile terminal needs to pay is calculated based on the traffic value of the network data generated in 72 hours. The traffic profile can also be made based on the traffic value used by each application in the mobile terminal, such as the traffic value used by APP1 is 100MB, and the traffic value used by APP2 is 723MB.

[0063] Based on the above description, in some embodiments, the flow value of network data can be collected through SNMP (Simple Gateway Monitoring Protocol-SGMP). However, the SNMP flow collection system includes multiple components, and abnormal flow often occurs due to flow collection loss, flow collection error and / or flow collection interruption, which leads to problems such as poor accuracy of flow billing and flow profiling.

[0064] Therefore, in order to improve the accuracy of traffic billing and traffic profiling, it is necessary to first perform anomaly detection on the collected network data traffic to determine the abnormal traffic. Then, the abnormal traffic is repaired and fitted to achieve abnormal recovery of the abnormal traffic. By eliminating the anomaly of network data traffic, the accuracy of traffic billing and traffic profiling is guaranteed.

[0065] In the related art, the method for processing network data traffic is generally to divide the traffic value of the collected network data into multiple traffic slices according to the set interval; for any traffic slice, the traffic value in the traffic slice is detected for abnormality according to the set detection rules or according to the mathematical probability, and then the abnormal traffic is determined. Then, the detected abnormal traffic value is processed for abnormal recovery according to the preset interpolation algorithm.

[0066] In some embodiments, when collecting network data traffic, the network data traffic can be collected according to the information of the target object, according to the set collection time interval and the set period; wherein the set collection time interval and the set period can be set values, which are not specifically limited here. For example, the collection time length is set to 1min (minutes), and the set period is set to 24h (hours), etc. For example, the target object is the IP address "10.5.***.***", and the network data traffic generated by the IP address "10.5.***.***" within 24 hours is collected at intervals of 1min.

[0067] After collecting network data traffic, the network data traffic is divided according to preset intervals to obtain traffic slices; wherein any traffic slice includes a collection time point and a traffic value corresponding to the collection time point. For example, the 24-hour network data traffic is divided according to 1 hour (preset interval) to obtain 24 traffic slices. Any traffic slice includes 60 collection time points and traffic values ​​corresponding to the collection time points; for example, the collection time points include 7:00, 7:01, 7:02, ..., 7:59, and the traffic value corresponding to 7:00 is 52MB, the traffic value corresponding to 7:01 is 48MB, ..., and the traffic value corresponding to 7:59 is 129MB.

[0068] After the traffic slices are obtained, anomaly detection and anomaly recovery are performed on any traffic slice.

[0069] In some embodiments, for any traffic slice, anomaly detection is performed on the network data traffic in the collected traffic slice based on the set anomaly detection rules to determine abnormal traffic. The set anomaly detection rules may be rules preset by network operation and maintenance personnel based on experience, including but not limited to the following rules: the rate of traffic is less than the specified bandwidth value, the rate of traffic is greater than 0, the traffic value is greater than 0, the collection quantity is consistent with the set quantity, etc.

[0070] Furthermore, the traffic that does not meet any of the above rules is determined as abnormal traffic, and then the fitting value corresponding to the abnormal traffic is determined based on the historical traffic before the abnormal traffic, and then the abnormal traffic is repaired and fitted based on the fitting value. Based on the above description, if the traffic value at each collection time point in the traffic slice is detected to not meet any of the above rules, the collection time point that does not meet the rule is determined as an abnormal collection point, and the traffic value corresponding to the abnormal collection point is determined as abnormal traffic.

[0071] After the abnormal flow is determined, the fitting value is calculated according to the nearest neighbor algorithm. Specifically, the fitting value of the abnormal flow is determined according to the historical flow before the abnormal flow, and the fitting value is used as the flow value of the abnormal collection point, thereby realizing the fitting repair of the abnormal flow. For example, the abnormal collection point is 7:00, and the flow value corresponding to 7:00 is the abnormal flow; the average value of the historical flow (flow from 6:30 to 6:59) 0.5 hours before 7:00 is calculated, and the average value is used as the fitting value corresponding to 7:00.

[0072] However, in the above technical solution, the design requirements of the preset detection rules are high, and the network operation and maintenance personnel who design the rules need to have strong professionalism. The process of designing rules is cumbersome and cannot adapt to detection in various scenarios, that is, the universality is poor. In addition, it is impossible to detect abnormal traffic outside the rules, and the fitting value is too dependent on historical traffic. If the standard deviation of historical traffic is large, it means that the traffic value of historical traffic fluctuates greatly, which leads to poor accuracy of the fitting value, making the accuracy of network data traffic processing of the overall solution poor.

[0073] In some embodiments, for any traffic slice, anomaly detection is performed on the network data traffic in the traffic slice based on a mathematical probability method to determine abnormal traffic, wherein the data probability method includes a normal distribution law, a box plot algorithm, and the like.

[0074] Furthermore, the normal flow range of the traffic slice is determined based on the normal distribution law and the box plot algorithm, and the flow outside the normal flow range is determined as abnormal flow.

[0075] Based on the above description, an example is given to calculate the traffic mean (μ), standard deviation (σ), lower quartile (Q1), median (Q2, i.e., traffic mean), upper quartile (Q3), difference between the lower quartile and the upper quartile (IQR) and other information according to the traffic values ​​at each collection time point in the traffic slice; wherein, IQR = Q3-Q1.

[0076] Figure 1 A schematic diagram of a normal distribution is shown as an example in this application. Figure 1 As shown, according to the 3σ principle of normal distribution, the difference between the flow mean and 3 times the standard deviation (μ-3σ) and the sum of the flow mean and 3 times the standard deviation (μ+3σ) are taken as the first normal flow range, that is, the first normal flow range is [μ-3σ,μ+3σ].

[0077] Figure 2 A schematic diagram of a box plot algorithm exemplified in this application is shown in FIG. Figure 2 As shown, the upper edge and the lower edge are used as the second normal flow range; wherein the upper edge is the sum of Q3 and k times IQR, that is, the upper edge s1 = Q3 + k * IQR; the lower edge is the difference between Q1 and k times IQR, that is, the lower edge s2 = Q1-k * IQR; k can be a value preset based on experience, such as k = 1.5, k = 3, etc. In the embodiment of the present application, k = 3 is taken as an example. That is, the second normal flow range is [Q1-3IQR, Q3+3IQR].

[0078] The flow value that is not within the first normal flow range and / or the second normal flow range is determined as abnormal flow. After the abnormal flow is determined, a fitting value corresponding to the abnormal flow is determined according to a set interpolation algorithm, and then the abnormal flow is repaired and fitted according to the fitting value.

[0079] Based on the above description, an example is given, assuming that the flow value 129MB corresponding to the collection time point 7:59 is neither in the first normal flow range nor in the second normal flow range, then the abnormal collection point is determined to be 7:59, and the flow value corresponding to 7:59 is abnormal flow. In the embodiment of the present invention, the interpolation algorithm set may be a linear interpolation algorithm, a Lagrange interpolation algorithm, a sample interpolation algorithm, etc., and the specific algorithm content is not limited in the embodiment of the present application.

[0080] However, in the above technical solution, based on the traffic mean, standard deviation, lower quartile, upper quartile and other information of the traffic slice as the standard for detecting abnormal traffic, it is impossible to determine whether the entire traffic slice is an abnormal traffic slice, and then the abnormal traffic cannot be detected, affecting the accuracy of network data traffic processing. In addition, misjudgment is prone to occur in scenarios such as non-periodic traffic surges or drops. For example, the traffic value at 7:00 is 53MB, and the traffic values ​​at each time point from 7:01 to 7:59 are less than 1MB. There is a misjudgment that the traffic value at 7:00 is abnormal traffic, and the fitting value corresponding to the traffic value at 7:00 cannot be calculated using the set difference algorithm.

[0081] In some embodiments, after the abnormal flow is determined, the fitting value corresponding to the abnormal flow is determined according to ARIMA (Auto Regression Integreate Moving Average, differential integrated moving average autoregressive model), and then the abnormal flow is repaired and fitted according to the fitting value. Among them, the ARIMA model is a prediction model based on time series, which implements repair fitting for time series data. Specifically, the ARIMA model models the next step in the time series as a linear function of the differential observation value and residual of the previous time step, and the predicted value y of the ARIMA model = a constant and / or a weighted value of y at one or more recent times and / or a prediction error at one or more recent times. The specific function formula is shown in the following formula (1).

[0082]

[0083] Where c is a constant; is the coefficient of the AR term, AR represents the number of lags of the time series data itself used in the ARIMA model; θ is the coefficient of the MR term, MR represents the number of lags of the forecast error used in the ARIMA model.

[0084] Based on the above description, an example is given, where the historical traffic before the abnormal traffic is input into the ARIMA model, and the fitting value corresponding to the abnormal traffic output by the ARIMA model is obtained, and then the abnormal traffic is repaired and fitted according to the fitting value.

[0085] However, in the above technical solution, the ARIMA model is affected by historical data, and there are problems such as inaccurate prediction values ​​caused by translation fitting, which affects the accuracy of network data traffic processing. In addition, the ARIMA model needs to calculate The θ coefficient has a large amount of calculation and high complexity, which affects the efficiency of network data traffic processing, making the network data traffic processing efficiency low.

[0086] Therefore, there is an urgent need for a method for processing network data traffic, which not only ensures the universality of network data traffic processing, but also improves the accuracy of network data traffic processing and the processing efficiency of network data traffic.

[0087] See also Figure 3 , which is a schematic diagram of a system architecture provided by an embodiment of the present application, includes a server 300 , which may include a processing unit 310 , a communication unit 320 , and a storage unit 330 .

[0088] The communication unit 320 is used to obtain the data to be processed written by the user.

[0089] The processing unit 310 is the control center of the server 300, and uses various interfaces and routes to connect various parts of the entire server 300, and executes various functions of the server 300 and processes data by running or executing software programs and / or modules stored in the storage unit 330, and calling data stored in the storage unit 330. Optionally, the processing unit 310 may include one or more processing units.

[0090] The storage unit 330 can be used to store software programs and modules. The processing unit 310 executes various functional applications and data processing by running the software programs and modules stored in the storage unit 330. The storage unit 330 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc.; the data storage area can store data created according to business processing, etc. In addition, the storage unit 330 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0091] It should be noted that the above Figure 3 The structure shown is only an example and is not limited to this embodiment of the present invention.

[0092] Based on the above description, Figure 4 A flowchart of a method for processing network data traffic provided in an embodiment of the present application is exemplarily shown, and the flowchart can be executed by a network data traffic processing device.

[0093] like Figure 4 As shown in the figure, the process specifically includes:

[0094] Step 410, obtaining a time domain sequence of network data traffic.

[0095] In an embodiment of the present application, the time domain sequence includes the flow value of network data collected at set time intervals; wherein the set time interval represents the collection interval when collecting network data flow; the time domain sequence can represent each collection time point and the flow value corresponding to the collection time point.

[0096] See also Figure 5 , Figure 5 A schematic diagram of a time domain sequence shown in an example of the present application, Figure 5 It represents the traffic value of network data of a certain IP address within 24 hours collected at a set time interval of 1 minute; wherein, the horizontal axis represents the collection time point of the network data traffic, and the vertical axis represents the traffic value corresponding to the collection time point.

[0097] Step 420: divide the time domain sequence into at least two traffic slices.

[0098] In the embodiment of the present application, the time domain sequence is converted into a frequency domain sequence according to the discrete Fourier transform, and the importance of each flow value at different frequencies is reflected by the frequency domain sequence. Then, the statistical interval time length used to segment the time domain sequence is calculated according to the frequency domain sequence, thereby increasing the credibility and rationality of the time domain sequence segmentation and improving the accuracy of abnormal detection of any flow segment after the time domain sequence segmentation.

[0099] Step 430: Determine an abnormal traffic slice among the at least two traffic slices based on the Gaussian distribution of the traffic value of the network data.

[0100] In the embodiment of the present application, for any traffic slice, it is determined whether the traffic slice has suspected abnormal traffic according to Gaussian distribution. The traffic slice with abnormal traffic is determined as an abnormal traffic slice. That is, the abnormal traffic slice includes at least one suspected abnormal traffic value.

[0101] Step 440: Determine a first fitting value corresponding to the at least one suspected abnormal flow value.

[0102] In the embodiment of the present application, for any suspected abnormal traffic value, the traffic mean of the abnormal traffic slice corresponding to the suspected abnormal traffic value is used as the first fitting value of the suspected abnormal traffic value.

[0103] Step 450: determining an abnormal flow value in the at least one suspected abnormal flow value based on the first model.

[0104] In an embodiment of the present application, the first model is a self-supervised model obtained after training with historical traffic samples based on an autoencoder; the first model is a one-dimensional convolutional network structure. The first model is used to model the input data (i.e., the abnormal volume traffic slice) to obtain the reconstructed first traffic slice; wherein the first traffic slice is time series data. Then, the abnormal traffic in the abnormal volume traffic slice is determined by the overall error and / or local error between the abnormal volume traffic slice and the first traffic slice.

[0105] Step 460: Determine a second fitting value of the abnormal flow value based on the second model.

[0106] In an embodiment of the present application, the second model may be a Transformer model; wherein the training convergence target of the second model is the mean and standard deviation of the predicted flow and the input flow.

[0107] Step 470: Perform abnormal recovery on the abnormal flow value according to the first fitting value and the second fitting value of the abnormal flow value.

[0108] In the embodiment of the present application, after obtaining the first fitting value and the second fitting value of the abnormal flow value, the fitting value of the abnormal flow value can be calculated according to the weighted summation method, and then the abnormal flow value is replaced with the fitting value of the abnormal flow value to achieve abnormal recovery of the abnormal flow value.

[0109] In step 420, before the time domain sequence is divided into at least two traffic slices, in order to reduce the amount of calculation and improve the calculation efficiency, the time domain sequence can be preprocessed so that the mean of the processed time domain sequence is 0, and then the processed time domain sequence is fast Fourier transformed to obtain a frequency domain sequence. Among them, the DC component of the frequency domain sequence is 0. The specific function formula of the fast Fourier transform is shown in the following formula (2).

[0110]

[0111] Among them, x(k) is a complex number sequence, that is, an N-dimensional sequence, representing the amplitude value; N represents the number of acquisition time points in the time domain sequence; 0≤n≤N-1; k represents the sequence number, 0≤k≤N-1. Based on the above Figure 5 For example, the time domain sequence is the traffic value of network data of a certain IP address within 24 hours collected at a set time interval of 1 minute, then N=1440, x(k) is a 1440-dimensional sequence, and then the frequency domain sequence is generated according to x(k).

[0112] See also Figure 6 , Figure 6This is a schematic diagram of a frequency domain sequence exemplarily shown in an embodiment of the present application. For ease of representation, Figure 6 The horizontal axis represents the frequency f (f = 1 / T, T represents the length of the statistical interval), and the vertical axis represents the logarithm of the amplitude value with base 10. It can be seen that the frequency domain sequence can represent the characteristics of the amplitude value of the time domain sequence in the frequency domain changing with the frequency. It should be noted that Figure 6 The negative sign on the horizontal axis has no actual meaning, but is only a formal meaning. The influence of the negative sign is not considered during calculation.

[0113] After determining the frequency domain sequence, the ratio of the maximum amplitude value in the frequency domain sequence to the first value is calculated to obtain the amplitude threshold; wherein the first value may be a value preset based on experience, which is not limited here. In the embodiment of the present application, the first value is 1000. See above Figure 6 , Figure 6 The solid horizontal line in the middle represents the amplitude threshold.

[0114] The frequency threshold is determined according to the amplitude threshold, so that the amplitude value corresponding to each frequency value greater than the frequency threshold is less than the amplitude threshold. Figure 6 , Figure 6 The vertical dashed line indicates the frequency threshold. Figure 6 It can be seen that the amplitude values ​​corresponding to the frequency values ​​greater than the frequency threshold are all below the horizontal solid line, that is, they are all less than the amplitude threshold.

[0115] Further, based on the above formula (2), find the i-th sequence value k i , so that it is greater than k i The amplitude values ​​corresponding to the sequence values ​​of are all less than the amplitude threshold. i The ratio of the frequency threshold to the number of acquisition time points is used as the frequency threshold. In the embodiment of the present application, the multiple is set to 2, but the multiple is not specifically limited. That is, the frequency threshold f = 2*k i / N.

[0116] In summary, the inverse of the frequency threshold is used as the statistical interval time length corresponding to the traffic slice. Then, the time domain sequence is divided into at least two traffic slices according to the statistical interval time length. Figure 6 , assuming the frequency threshold f = 0.05, then T = 10min. See above Figure 5 , the time domain sequence is divided into 144 traffic slices according to the interval of 10 minutes (statistical interval time length), and any traffic slice includes the traffic value of the network data collected within the statistical interval time length. For example, the collection time point of the traffic slice is 5:00-5:09; among them, the traffic value collected at 5:00 is 18MB, the traffic value at 5:01 is 35MB, ..., and the traffic value collected at 5:09 is 29MB.

[0117] It can be understood that for the same target object, if the traffic characteristics of the target object do not change significantly, the length of the statistical interval time does not need to be recalculated.

[0118] In step 430, the determination of abnormal traffic is divided into the following two scenarios.

[0119] The first scenario:

[0120] Based on the Gaussian distribution of traffic values ​​of network data, determine the normal traffic value range corresponding to each traffic shard in at least two traffic shards. If any traffic value in the first traffic shard of at least two traffic shards is outside the normal traffic value range corresponding to the first traffic shard, the first traffic shard is determined as an abnormal traffic shard.

[0121] See also Figure 1 , for any traffic slice, calculate the traffic mean and traffic standard deviation of the traffic slice. Then, according to the sum of the traffic mean of the traffic slice and the traffic standard deviation of the set multiple, the first abnormal threshold is obtained, and then the difference between the traffic mean of the traffic slice and the traffic standard deviation of the set multiple is calculated to obtain the second abnormal threshold. The range less than the first abnormal threshold and greater than the second abnormal threshold is taken as the normal traffic value range of the traffic slice.

[0122] Based on the above description, for example, the normal traffic value range of the traffic slice from 5:00 to 5:09 is [45.5, 46.5]. Assuming that the traffic value collected at 5:07 is 51MB, that is, the traffic value collected at 5:07 is outside the normal traffic value range, the traffic slice from 5:00 to 5:09 is determined as an abnormal traffic slice, and the traffic value at the collection time point of 5:07 is determined as suspected abnormal traffic.

[0123] For further information, see Figure 7 , Figure 7 The following is a schematic diagram of monitoring network data flow based on Gaussian distribution, which is exemplified in the embodiment of the present application. Figure 7 As shown in the figure, the horizontal axis represents the mth traffic slice, 0≤m≤143; the vertical axis represents the logarithm of the traffic value with base 10. The h1 curve represents the upper limit of the normal traffic value range of each traffic slice; the h2 curve represents the traffic mean of each traffic slice; the h3 curve represents the lower limit of the normal traffic value range of each traffic slice; and j represents the traffic value of each traffic slice. Figure 7 It can be seen that the flow values ​​j1, j2, and j3 are not within the normal flow value range of the corresponding flow slices. Therefore, the flow values ​​j1, j2, and j3 are determined to be abnormal flow.

[0124] In the embodiment of the present application, the normal flow value range of each flow segment can also be calculated according to other methods, such as the above Figure 2The box plot method shown, etc., is not specifically limited here.

[0125] Second scenario:

[0126] If the second traffic slice of at least two traffic slices lacks a traffic value, the second traffic slice is determined as an abnormal traffic slice.

[0127] In one implementable method, for any traffic slice, if the time interval between any two adjacent traffic values ​​in the traffic slice is greater than a time threshold, then the missing traffic value between the two adjacent traffic values ​​is determined; wherein the time threshold is calculated based on the average value of the time interval between the fth traffic value and the f+1th traffic value in one or more traffic slices.

[0128] Exemplarily, the collection time points of the traffic slice include 10, which are calculated according to the following formula (3).

[0129]

[0130] Among them, t m is the time threshold; t f represents the collection time point of the fth flow value; t f+1 Indicates that f The next adjacent acquisition time point.

[0131] Assume that t 11 The flow value and t 10 If there is a missing flow value between the flow values ​​of 11 With t 10 If the time interval is greater than 1.2t m , then determine t 11 The flow value and t 10 The flow value is missing between the flow values ​​of , otherwise determine t 11 The flow value and t 10 There are no missing flow values ​​between the flow values ​​of .

[0132] See also Figure 8 , Figure 8 FIG. 1 is a schematic diagram of a missing flow value detection exemplarily shown in an embodiment of the present application. Figure 8 As shown in the figure, the horizontal axis represents the tth acquisition time point, 0≤t≤1443; the vertical axis represents the logarithm of the flow value with base 10. Figure 8 It can be seen that the flow values ​​j4 and j5 corresponding to the dotted lines are missing flow values.

[0133] After determining that a flow value is missing between adjacent flow values, a suspected abnormal flow value is added to the second flow slice, and then abnormal recovery processing is performed on the missing flow value.

[0134] In step 440, the fitting value of the suspected abnormal flow value can be calculated according to the flow slice corresponding to the suspected abnormal flow value. Exemplarily, the flow mean of the flow slice corresponding to the suspected abnormal flow value is used as the first fitting value corresponding to the suspected abnormal flow value.

[0135] In step 450, the first model is obtained after training based on the autoencoder. Unlike the autoencoder, which is a fully connected network structure, the first model of the present application is a one-dimensional convolutional network structure, which is used to mine the regional spatial characteristics of the traffic slices of the time domain sequence and reproduce the normal traffic in the abnormal traffic slices.

[0136] See also Fig. 9 , Fig. 9 The following is a schematic diagram of an embodiment of the present application showing an example of determining an abnormal traffic value based on a first model. The input data is the abnormal traffic segment determined above, and the input data is the first traffic segment reproduced based on the first model. The difference between the first traffic segment and the abnormal traffic segment is determined by Fig. 9 It can be seen that the difference between the flows corresponding to the solid-line boxes is relatively large, so the suspected abnormal flow values ​​whose difference with the first flow segment is greater than the set abnormal value are determined as abnormal flow values.

[0137] In an implementable manner, based on the first model, it can be determined whether all flows in the abnormal flow segment are abnormal flows. If the difference between each flow value in the first flow segment and each flow value in the abnormal flow segment is greater than the set abnormal value, it is determined that each flow in the abnormal flow segment is abnormal flow.

[0138] In an embodiment of the present application, the first model can simultaneously detect 200 traffic segments per second, which improves data detection efficiency compared to the autoencoder.

[0139] In step 460, the input of the second model includes a preset number (e.g., 32) of traffic segments; any traffic segment includes each collection time point t, the traffic value f corresponding to the collection time point, and the digitized traffic value rec=log 10 f.

[0140] The processing logic of the second model includes the following steps:

[0141] Step 1010, normalize the collection time points and flow values ​​f, so that each collection time point is normalized to between 0 and 1, where 0 represents 00:00 and 1 represents 23:59. Each flow value f is normalized to between 0 and 1.

[0142] Step 1020, generating a tensor according to a preset number of traffic segments, and the normalized collection time points and traffic value f.

[0143] For example, if the preset number is 32, then the tensor x inp It is a 32×4 or 32×3 matrix.

[0144] Step 1030, perform embedding operation on the tensor to obtain an input matrix.

[0145] For example, x emb =x inp *M emb +b emb ; where x emb is the input matrix, M emb To set a 4×64 (or 3×64) dimensional matrix, b emb is a 64-dimensional vector.

[0146] Step 1040, use a 4-layer Transformer Encoder to transform x emb Transform to x out .

[0147] Among them, x out It is a 32×64 dimensional matrix, which can also be expressed as 32 64-dimensional vectors, where the vectors represent the features of rec.

[0148] Step 1050, based on the 32nd 64-dimensional vector, use a 2-layer feedforward neural network to predict the flow at the next moment to obtain a second fitting value.

[0149] See also Fig.10 , Fig.10 A schematic diagram of determining a second fitting value based on a second model is exemplified in an embodiment of the present application. A preset number of traffic segments are input data, and the input data is a second traffic segment reproduced based on the second model. It can be seen that the standard deviation of the second traffic segment obtained based on the second model is slightly different from the standard deviation of the input data, and the second fitting value corresponding to the abnormal traffic is determined in the second traffic segment. The accuracy of network data traffic processing can be improved.

[0150] In step 470, the flow recovery value of the abnormal flow value is determined according to the first fitting value of the abnormal flow value and the weight corresponding to the first fitting value, and the second fitting value of the abnormal flow value and the weight corresponding to the second fitting value.

[0151] In some embodiments, the flow recovery value of the abnormal flow value is determined by weighted summation, wherein the sum of the weight corresponding to the first fitting value and the weight corresponding to the second fitting value is 1; the weight corresponding to the first fitting value and the weight corresponding to the second fitting value can be values ​​preset based on experience, which are not specifically limited here, such as the weight corresponding to the first fitting value is 0.3, and the weight corresponding to the second fitting value is 0.7.

[0152] For example, the multiplication result of the first fitting value (z1) of the abnormal flow value and the first weight (0.3) is calculated to obtain 0.3*z1; the multiplication result of the second fitting value (z2) of the abnormal flow value and the second weight (0.7) is calculated to obtain 0.7*z2; and then the flow recovery value is calculated to be 0.3*z1+0.7*z2.

[0153] After the flow recovery value of the abnormal flow value is determined, the abnormal flow value is replaced with the flow recovery value, thereby achieving abnormal recovery of the abnormal flow value.

[0154] In the embodiment of the present invention, abnormal traffic is detected by Gaussian distribution and the first model, thereby improving the detection efficiency and universality of network data traffic. The method of detecting abnormal traffic based on the first model can solve the problem that the normal distribution and the box plot method cannot find abnormal traffic fragments, thereby improving the detection accuracy of network data traffic. By using the second model to calculate the second fitting value of abnormal traffic, the detection accuracy of network data traffic is improved.

[0155] In order to better explain the above technical solution, Fig.11 A schematic diagram of a method for processing network data traffic is shown as an example. Fig.11 As shown, the content includes:

[0156] The time domain sequence generated by the flow value of the network data collected at a set time interval is obtained, the flow mean of the time domain sequence is counted, and the time domain sequence is preprocessed according to the flow mean. Then, the processed time domain sequence is converted into a frequency domain sequence according to the fast Fourier transform.

[0157] After the frequency domain sequence is determined, the statistical interval time length used to divide the time domain sequence is calculated according to the frequency domain sequence, and then the time domain sequence is divided according to the statistical interval time length to obtain at least two traffic slices.

[0158] For any traffic slice, determine whether there is suspected abnormal traffic in the traffic slice based on Gaussian distribution. If so, determine the traffic slice as an abnormal traffic slice, and determine the traffic mean of the traffic slice as the first fitting value of the suspected abnormal traffic.

[0159] For any traffic slice, determine whether there is a missing traffic value in the traffic slice based on Gaussian distribution. If so, add suspected abnormal traffic to the traffic slice, determine the traffic slice as an abnormal traffic slice, and determine the traffic mean of the traffic slice as the first fitting value of the suspected abnormal traffic.

[0160] After the abnormal traffic slice is determined, the abnormal traffic slice is input into the first model, and the abnormal traffic is determined from the suspected abnormal traffic in the abnormal traffic slice according to the first traffic slice output by the first model.

[0161] In some embodiments, after abnormal traffic is determined, an alarm mechanism is triggered to display an alarm message.

[0162] After the abnormal flow is determined, a second fitting value corresponding to the abnormal flow is determined based on the second model. The second model can be trained by an external flow reference value to improve the data processing accuracy of the second model.

[0163] A weighted sum is performed on the first fitting value and the second fitting value of the abnormal flow to obtain a flow recovery value corresponding to the abnormal flow, and abnormal recovery is performed on the abnormal flow according to the flow recovery value.

[0164] In the embodiment of the present application, abnormal traffic is determined through the following two steps: the first step is to determine suspected abnormal traffic based on Gaussian distribution; the second step is to determine abnormal traffic from suspected abnormal traffic based on the first model. In this way, the error rate of determining abnormal traffic based on Gaussian distribution is reduced, and the accuracy of abnormal traffic detection is improved.

[0165] A first fitting value of the abnormal traffic is determined based on Gaussian distribution, a second fitting value of the abnormal traffic is determined based on a second model, and then abnormal recovery of the abnormal traffic is achieved based on the first fitting value and the second fitting value of the abnormal traffic, thereby increasing the credibility and accuracy of abnormal recovery and improving the accuracy of network data traffic processing.

[0166] The embodiment of the present application also proposes a network data traffic processing device for executing each step in the above method. The relevant features can be found in the introduction of the above embodiment and will not be repeated here. Fig.12 This is a schematic diagram of a network data flow processing device exemplarily shown in an embodiment of the present application, see Fig.12 , the device comprises:

[0167] The acquisition module 1210 is used to acquire a time domain sequence of network data flow; the time domain sequence includes flow values ​​of network data collected at set time intervals;

[0168] Processing module 1220 is used to divide the time domain sequence into at least two traffic slices; determine the abnormal traffic slice in the at least two traffic slices based on the Gaussian distribution of the traffic value of the network data, and the abnormal traffic slice includes at least one suspected abnormal traffic value; determine the first fitting value corresponding to the at least one suspected abnormal traffic value; determine the abnormal traffic value in the at least one suspected abnormal traffic value based on the first model; determine the second fitting value of the abnormal traffic value based on the second model; and perform abnormal recovery on the abnormal traffic value according to the first fitting value and the second fitting value of the abnormal traffic value.

[0169] In some embodiments, the processing module 1220 is specifically used to: convert the time domain sequence into a frequency domain sequence, the frequency domain sequence is used to represent the characteristics of the amplitude value of the network data traffic in the frequency domain changing with the frequency; determine the statistical interval time length corresponding to the traffic slice according to the frequency domain sequence; divide the time domain sequence into at least two traffic slices according to the statistical interval time length, each of the at least two traffic slices includes the traffic value of the network data collected within the statistical interval time length.

[0170] In some embodiments, the processing module 1220 is specifically used to: determine the ratio of the maximum amplitude value to the first value in the frequency domain sequence to obtain an amplitude threshold; determine a frequency threshold based on the amplitude threshold so that the amplitude value corresponding to the frequency value greater than the frequency threshold is less than the amplitude threshold; and use the inverse of the frequency threshold as the statistical interval time length corresponding to the traffic segment.

[0171] In some embodiments, the processing module 1220 is specifically used to: determine the normal traffic value range corresponding to each of the at least two traffic slices based on the Gaussian distribution of the traffic value of the network data; if any traffic value in the first traffic slice of the at least two traffic slices is outside the normal traffic value range corresponding to the first traffic slice, then determine the first traffic slice as an abnormal traffic slice; and / or, if the second traffic slice of the at least two traffic slices lacks a traffic value, then determine the second traffic slice as an abnormal traffic slice.

[0172] In some embodiments, the processing module 1220 is specifically used to: before determining the first fitting value corresponding to the at least one suspected abnormal traffic value, if the second traffic slice of the at least two traffic slices is missing a traffic value, then add the suspected abnormal traffic value to the second traffic slice.

[0173] In some embodiments, the processing module 1220 is specifically used to: for a suspected abnormal traffic value in any abnormal traffic slice, use the traffic mean of the abnormal traffic slice as a first fitting value corresponding to the suspected abnormal traffic value.

[0174] In some embodiments, the processing module 1220 is specifically used to: determine the flow recovery value of the abnormal flow value based on the first fitting value of the abnormal flow value and the weight corresponding to the first fitting value, as well as the second fitting value of the abnormal flow value and the weight corresponding to the second fitting value; and perform abnormal recovery on the abnormal flow value based on the flow recovery value of the abnormal flow value.

[0175] In a simple embodiment, those skilled in the art can imagine that the network data flow processing device in the above embodiment can be used. Fig.13The form shown.

[0176] like Fig.13 The device 1300 shown includes at least one processor 1310 , a memory 1320 , and optionally, a communication interface 1330 .

[0177] The specific connection medium between the processor 1310 and the memory 1320 is not limited in the present application embodiment. Fig.13 The device also includes a communication interface 1330. When the processor 1310 communicates with other devices, data can be transmitted through the communication interface 1330.

[0178] It should be understood that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0179] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0181] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for processing network data traffic, characterized in that: The method comprises: Acquire a time domain sequence of network data traffic, wherein the time domain sequence includes traffic values ​​of network data collected at set time intervals; Splitting the time domain sequence into at least two traffic slices; Determine an abnormal traffic slice in the at least two traffic slices based on the Gaussian distribution of the traffic value of the network data, wherein the abnormal traffic slice includes at least one suspected abnormal traffic value; Determine a first fitting value corresponding to the at least one suspected abnormal flow value; determining an abnormal flow value among the at least one suspected abnormal flow value based on a first model, wherein the first model is a self-supervised model; Determine a second fitting value of the abnormal flow value based on a second model, where the second model is a Transformer model; The abnormal flow value is abnormally recovered according to the first fitting value and the second fitting value of the abnormal flow value.

2. The method according to claim 1, characterized in that Dividing the time domain sequence into at least two traffic slices includes: Converting the time domain sequence into a frequency domain sequence, wherein the frequency domain sequence is used to represent the characteristic of the amplitude value of the flow of the network data in the frequency domain changing with the frequency; Determine the statistical interval time length corresponding to the traffic slice according to the frequency domain sequence; The time domain sequence is divided into at least two traffic slices according to the statistical interval time length, and each of the at least two traffic slices includes a traffic value of network data collected within the statistical interval time length.

3. The method according to claim 2, characterized in that Determining the statistical interval time length corresponding to the traffic slice according to the frequency domain sequence includes: Determine a ratio of a maximum amplitude value in the frequency domain sequence to the first value to obtain an amplitude threshold; Determine a frequency threshold according to the amplitude threshold, so that an amplitude value corresponding to a frequency value greater than the frequency threshold is less than the amplitude threshold; The reciprocal of the frequency threshold is used as the statistical interval time length corresponding to the traffic slice.

4. The method according to any one of claims 1 to 3, characterized in that: Determining an abnormal traffic slice in the at least two traffic slices based on a Gaussian distribution of a traffic value of the network data includes: Based on the Gaussian distribution of the traffic value of the network data, determine the normal traffic value range corresponding to each of the at least two traffic slices, if any traffic value in a first traffic slice of the at least two traffic slices is outside the normal traffic value range corresponding to the first traffic slice, determine the first traffic slice as an abnormal traffic slice; and / or If the second traffic slice among the at least two traffic slices lacks a traffic value, the second traffic slice is determined as an abnormal traffic slice.

5. The method according to claim 1, characterized in that Before determining the first fitting value corresponding to the at least one suspected abnormal flow value, the method further includes: If the second traffic slice of the at least two traffic slices lacks a traffic value, a suspected abnormal traffic value is added to the second traffic slice.

6. The method according to claim 1, characterized in that Determining a first fitting value corresponding to the at least one suspected abnormal flow value includes: For any suspected abnormal traffic value in any abnormal traffic slice, the traffic mean of the abnormal traffic slice is used as the first fitting value corresponding to the suspected abnormal traffic value.

7. The method according to claim 1, characterized in that The abnormal recovery of the abnormal flow value according to the first fitting value and the second fitting value of the abnormal flow value comprises: Determine a flow recovery value of the abnormal flow value according to a first fitting value of the abnormal flow value and a weight corresponding to the first fitting value, and a second fitting value of the abnormal flow value and a weight corresponding to the second fitting value; The abnormal flow value is abnormally recovered based on the flow recovery value of the abnormal flow value.

8. A network data flow processing device, characterized in that: include: An acquisition module, used to acquire a time domain sequence of network data flow, wherein the time domain sequence includes flow values ​​of network data collected at set time intervals; A processing module, configured to divide the time domain sequence into at least two traffic slices; Determine an abnormal traffic slice in the at least two traffic slices based on the Gaussian distribution of the traffic value of the network data, wherein the abnormal traffic slice includes at least one suspected abnormal traffic value; Determine a first fitting value corresponding to the at least one suspected abnormal flow value; determining an abnormal flow value among the at least one suspected abnormal flow value based on a first model, wherein the first model is a self-supervised model; Determine a second fitting value of the abnormal flow value based on a second model, where the second model is a Transformer model; The abnormal flow value is abnormally recovered according to the first fitting value and the second fitting value of the abnormal flow value.

9. The device according to claim 8, characterized in that The processing module is specifically used for: Converting the time domain sequence into a frequency domain sequence, wherein the frequency domain sequence is used to represent the characteristic of the amplitude value of the flow of the network data in the frequency domain changing with the frequency; Determine the statistical interval time length corresponding to the traffic slice according to the frequency domain sequence; The time domain sequence is divided into at least two traffic slices according to the statistical interval time length, and each of the at least two traffic slices includes a traffic value of network data collected within the statistical interval time length.

10. The device according to claim 9, characterized in that The processing module is specifically used for: Determine a ratio of a maximum amplitude value in the frequency domain sequence to the first value to obtain an amplitude threshold; Determine a frequency threshold according to the amplitude threshold, so that an amplitude value corresponding to a frequency value greater than the frequency threshold is less than the amplitude threshold; The reciprocal of the frequency threshold is used as the statistical interval time length corresponding to the traffic slice.

11. The device according to any one of claims 8 to 10, characterized in that: The processing module is specifically used for: Based on the Gaussian distribution of the traffic value of the network data, determine the normal traffic value range corresponding to each traffic slice of the at least two traffic slices, if any traffic value in the first traffic slice of the at least two traffic slices is outside the normal traffic value range corresponding to the first traffic slice, determine the first traffic slice as an abnormal traffic slice; and / or If the second traffic slice among the at least two traffic slices lacks a traffic value, the second traffic slice is determined as an abnormal traffic slice.

12. The device according to claim 8, characterized in that The processing module is also used for: Before determining the first fitting value corresponding to the at least one suspected abnormal traffic value, if a second traffic slice of the at least two traffic slices lacks a traffic value, then adding the suspected abnormal traffic value to the second traffic slice.

13. The device according to claim 8, characterized in that The processing module is specifically used for: For any suspected abnormal traffic value in any abnormal traffic slice, the traffic mean of the abnormal traffic slice is used as the first fitting value corresponding to the suspected abnormal traffic value.

14. The device according to claim 8, characterized in that The processing module is specifically used for: Determine a flow recovery value of the abnormal flow value according to a first fitting value of the abnormal flow value and a weight corresponding to the first fitting value, and a second fitting value of the abnormal flow value and a weight corresponding to the second fitting value; The abnormal flow value is abnormally recovered based on the flow recovery value of the abnormal flow value.

15. A readable storage medium, characterized in that: The readable storage medium includes a program, and when the program is executed on a device, the device is caused to execute the method according to any one of claims 1 to 7.

16. A device, characterized in that: include: One or more processors; wherein, when instructions of one or more computer programs are executed by the one or more processors, the device performs the method according to any one of claims 1 to 7.

17. A chip system, characterized in that: The device comprises at least one chip and a memory, wherein the at least one chip is coupled to the memory and is used to read and execute program instructions stored in the memory to implement the method according to any one of claims 1 to 7.

18. A computer program product, characterized in that When the computer program product is called by a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images