A network data analysis system based on deep learning

Abstract

The present application discloses a network data analysis system based on deep learning, which relates to the field of computer network technology, including: a data flow acquisition module, which is used to acquire real-time data flow; a primary classification module, which is used to determine whether the data point to be classified is abnormal data flow according to the distance between the data point to be classified and other data points corresponding to the real-time data flow; a secondary classification module, which is used to determine whether the real-time data flow from multiple API interfaces has reached the bandwidth upper limit after the primary classification module determines that the data point to be classified is abnormal data flow, and if so, correct the distance between the data point to be classified and other data points, and determine whether the data point to be classified is normal data flow according to the corrected distance. The present application identifies normal data flow with data flow fluctuations caused by bandwidth limitations from data flow considered abnormal, thereby improving the accuracy of data flow analysis results.

Description

Technical Field

[0001] The present application relates to the field of computer network technology, and in particular to a network data analysis system based on deep learning. Background Art

[0002] Computer networks have penetrated into people's lives. Various electronic devices are connected through computer networks to exchange data and information. Data traffic is generated during the transmission of data in the network. After a network attack occurs, such as DDoS (distributed denial of service attack) and virus infection, obvious traffic anomalies will occur after the computer equipment is connected to the network. By identifying these traffic anomalies, network anomalies can be discovered in a timely manner.

[0003] The identification of traffic anomalies was initially done manually. Network experts analyzed real-time network traffic data and based on their experience, they could draw conclusions on whether network traffic anomalies occurred. This method relies heavily on manual experience and is not satisfactory in terms of accuracy and efficiency. In response to this problem, a variety of automated analysis methods have been developed, and the most widely used is the identification method based on deep learning technology. Deep learning is a type of machine learning. By conducting in-depth analysis of data traffic, analysis conclusions can be obtained relatively quickly and accurately. For example, CN113378990A provides a traffic data anomaly detection method based on deep learning, which classifies data points according to their distance in the sample space and their density within a certain radius.

[0004] However, data traffic is closely related to the network bandwidth of computer devices. When data traffic suddenly increases or decreases, it does not necessarily mean that there is an abnormality in data traffic. It may also be caused by normal data traffic being limited by bandwidth. However, the existing technology does not take this situation into consideration. Summary of the invention

[0005] The embodiment of the present application provides a network data analysis system based on deep learning, which is used to solve the problem that the existing technology does not consider the impact of bandwidth limitations on data traffic.

[0006] The present application embodiment provides a network data analysis system based on deep learning, including:

[0007] A data flow acquisition module is used to acquire real-time data flow;

[0008] A primary classification module is used to establish a feature space, represent the real-time data flow as a data point to be classified in the feature space, and determine whether the data point to be classified is abnormal data flow according to the distance between the data point to be classified and other data points in the feature space;

[0009] The secondary classification module is used to determine whether the real-time data traffic from multiple API interfaces has reached the bandwidth limit after the primary classification module determines that the data point to be classified is abnormal data traffic. If so, the weight of the real-time data traffic of each API interface is determined according to the network speed of the multiple API interfaces at the same time, and the distance between the data point to be classified and other data points in the feature space is recalculated according to the weight, and whether the data point to be classified is normal data traffic is determined according to the recalculated distance.

[0010] A network data analysis system based on deep learning in this application has the following advantages:

[0011] First, determine whether there is a traffic anomaly based on the distance between data points in the feature space. If there is a traffic anomaly, determine whether the network speed at the time the data traffic occurs has reached the bandwidth upper limit. If so, correct the distance and determine whether there is a traffic anomaly again. This effectively distinguishes normal data traffic with data traffic fluctuations caused by bandwidth limitations from data traffic that is considered abnormal, thereby improving the accuracy of data traffic analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 A schematic diagram of the functional modules of a deep learning-based network data analysis system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0015] Figure 1 A functional module diagram of a network data analysis system based on deep learning provided in an embodiment of the present application. An embodiment of the present application provides a network data analysis system based on deep learning, including:

[0016] The data flow acquisition module 100 is used to acquire real-time data flow.

[0017] Exemplarily, the system of the present application is deployed on a computer device, which can be a desktop, notebook or tablet computer in the usual sense, or various portable smart devices. Since a computer device can receive or send data through multiple API (application programming interface) interfaces at the same time, when obtaining real-time data traffic, the data traffic acquisition module 100 needs to monitor the status of each API interface of the computer device and obtain the real-time data traffic of each API interface.

[0018] In the embodiment of the present application, the real-time data flow is multi-dimensional data. After acquiring the real-time data flow of the computer device, the data flow acquisition module 100 will represent and save the real-time data flow in the form of a multi-dimensional array.

[0019] The primary classification module 110 is used to establish a feature space, represent the real-time data flow as a data point to be classified in the feature space, and determine whether the data point to be classified is abnormal data flow according to the distance between the data point to be classified and other data points in the feature space.

[0020] Exemplarily, after the data traffic acquisition module 100 acquires the real-time data traffic of each API interface, the primary classification module represents the real-time data traffic of each API interface as data points to be classified in different feature spaces.

[0021] The feature space is a multidimensional space with the same dimension as the real-time data flow. In the feature space, the multidimensional data of the real-time data flow is reflected in the corresponding dimensions, forming data points whose positions can be reflected by coordinate points.

[0022] In the embodiment of the present application, before the real-time data flow is input into the feature space, there are already multiple data points in the feature space, and these data points are other data points. These other data points are previously determined to be normal data flows. Since they all have the same attributes, that is, normal attributes, all data points are gathered within a certain range in the feature space, and the distance between any two data points is not very large. If the distance between the newly added data point to be classified and any other data point is relatively small, it means that the data point to be classified can also be classified as a normal data point.

[0023] Furthermore, a classification module calculates the distance between the data point to be classified and multiple other data points, takes the average of the distances, compares the average distance with the distance threshold, and when the average distance is greater than the distance threshold, determines that the data point to be classified is abnormal data traffic. Specifically, the distance threshold is the average of the distances between all other data points. In practical applications, the distance between the data point to be classified and each other data point can be calculated in turn, and after obtaining the same number of distance values ​​as other data points, the average of all distance values ​​is calculated, and then the average distance is compared with the distance threshold. This calculation method takes into account the positions of all other data points. When there are other data points that are far away from the center, considering these data points may cause the average distance to be inaccurately expressed. Therefore, the present application can select a part of other data points to calculate the average distance.

[0024] Furthermore, when determining the number of other data points to be selected to calculate a more accurate average distance, a classification module performs a classification test on known data points in advance, and uses the number of other data points selected when the classification accuracy is the highest as the number of other data points when calculating the distance between the data point to be classified and the other data points. Specifically, the pre-prepared data flow can be divided into other data flow and known data flow according to a certain ratio, such as 7:3, and the other data flow and the known data flow are represented as other data points and known data points in the feature space. The other data points and the known data points have the same classification, so the other data flow can be first input into the feature space to form other data points, and then the corresponding known data points are classified after each known data flow is input. When all the known data points are classified, the current classification accuracy is calculated. Then repeat the classification operation, select a different number of other data points each time, so different numbers of other data points and corresponding classification accuracy will be obtained, and the number of other data points with the highest classification accuracy is selected as the number of other data points selected when the data points to be classified are subsequently classified.

[0025] The secondary classification module 120 is used to determine whether the real-time data traffic from multiple API interfaces has reached the bandwidth upper limit after the primary classification module determines that the data point to be classified is abnormal data traffic. If so, the weight of the real-time data traffic of each API interface is determined according to the network speed of the multiple API interfaces at the same time, and the distance between the data point to be classified and other data points in the feature space is recalculated according to the weight, and whether the data point to be classified is normal data traffic is determined according to the recalculated distance.

[0026] Exemplarily, the network to which the computer device is connected has a bandwidth upper limit, that is, the network speed of the computer network cannot exceed the bandwidth upper limit. For example, when the network to which the computer device is connected has a bandwidth upper limit of 100M, the sum of the network speeds of all API interfaces in the computer device can only reach 100M at most, and cannot exceed 100M. Based on this situation, if the computer device is connected to multiple API interfaces at the same time, these API interfaces will share the bandwidth. When a new API interface is suddenly connected, or the data transmission of a certain API interface is completed, the network speed of other API interfaces will suddenly decrease or increase, which is manifested in data traffic, that is, the data traffic of some API interfaces in a certain period of time suddenly decreases or increases. However, this sudden change is not necessarily caused by a network attack, but may also be caused by the limitation of the bandwidth upper limit. Therefore, when the primary classification module determines that the data point to be classified is abnormal, the secondary classification module is used to make a judgment again to minimize the misjudgment of data traffic.

[0027] In an embodiment of the present application, the secondary classification module detects the bandwidth upper limit of the computer device before determining whether the real-time data traffic from multiple API interfaces has reached the bandwidth upper limit. Specifically, when detecting the bandwidth upper limit, the secondary classification module first interrupts the data transmission of all API interfaces of the computer device, then establishes an API interface pointing to the specified link, and downloads a data packet of known size through the API interface. After the download is completed, the ratio of the data packet size to the download duration is calculated to obtain the most appropriate bandwidth upper limit. It should be understood that the bandwidth upper limit is usually a few fixed values, such as 100M, 300M, 500M, etc. After calculating the ratio of the data packet size to the download duration, the value closest to and greater than the ratio can be used as the bandwidth upper limit.

[0028] Furthermore, the bandwidth upper limit is the upper limit of the network speed, and the secondary classification module determines the network speed based on the data volume and transmission time in the real-time data flow. Since the real-time data flow already includes the start time, end time and data volume, the difference between the end time and the start time is the data transmission time. The ratio of the data volume and the transmission time can be calculated to obtain the network speed during the data transmission time. By summing up the network speeds of all API interfaces, it can be determined whether the network speed of the computer device at a certain moment or in a certain period of time has reached the bandwidth upper limit.

[0029] Furthermore, after determining the network speed, the secondary classification module decomposes the bandwidth upper limit according to the number of API interfaces whose network speed is not zero at the time when the real-time data flow occurs, obtains the corresponding theoretical speed value, calculates the difference between the network speed of each API interface and the theoretical speed, and determines the weight according to the speed difference of each API interface. Specifically, the bandwidth upper limit can be averaged according to the number of API interfaces whose network speed is not zero, that is, the ratio of the bandwidth upper limit to the number of API interfaces whose network speed is not zero is used as the theoretical speed value.

[0030] However, it is a common situation that not all API interfaces have the same network speed. Therefore, this application sets a weight factor for each API interface. The sum of the weight factors of all API interfaces is 1. After the bandwidth upper limit is decomposed, the proportion of the theoretical speed value corresponding to the API interface in the bandwidth upper limit is the corresponding weight factor. The weight factor can be set manually according to the actual situation of the API interface. The greater the maximum network speed that the API interface can reach, the greater the corresponding weight factor. After determining the theoretical speed value, the network speed of each API interface is subtracted from the corresponding theoretical speed value, and the weight for distance correction can be determined based on the difference. It should be understood that since it is impossible to determine in advance which API interfaces have a non-zero network speed, appropriate weight values ​​can be set manually for all API interfaces, regardless of whether the current network speed is zero. After a specific data transmission task is completed, the API interfaces with a non-zero network speed can be determined, and then the sum of the weight values ​​of these API interfaces is calculated. The ratio of the weight value of each API interface to the sum of the weight values ​​can be used as a weight factor.

[0031] Furthermore, whether the network speed decreases or increases due to the access of a new API interface or the release of resources by a certain API interface, it is reflected in the increase of distance in data flow. Therefore, the distance needs to be reduced during correction so that the abnormal data flow is as close to the normal data flow as possible. In the embodiment of the present application, a fuzzy control strategy is used to determine the weight. Specifically, the correspondence between the speed difference and the weight can be pre-set. When the speed difference is within a certain numerical range, the weight corresponding to the numerical range can be selected, and then the weight can be used to reduce the distance. The weight can use a value less than 1, so when reducing the distance, the product of the weight and the distance determined by the primary classification model can be used as the recalculated distance.

[0032] After the distance is recalculated, the recalculated distance is compared with the distance threshold. If the recalculated distance is still greater than the distance threshold, it indicates that the data flow corresponding to the data point to be classified is still abnormal data flow.

[0033] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0034] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A network data analysis system based on deep learning, characterized in that: include: A data flow acquisition module is used to acquire real-time data flow; A primary classification module, used to establish a feature space, represent the real-time data flow as a data point to be classified in the feature space, and determine whether the data point to be classified is abnormal data flow according to the distance between the data point to be classified and other data points in the feature space; A secondary classification module is used to determine whether the real-time data traffic from multiple API interfaces has reached the bandwidth upper limit after the primary classification module determines that the data point to be classified is abnormal data traffic, and if so, determine the weight of the real-time data traffic of each API interface according to the network speed of the multiple API interfaces at the same time, recalculate the distance between the data point to be classified and other data points in the feature space according to the weight, and determine whether the data point to be classified is normal data traffic according to the recalculated distance; The data flow acquisition module monitors the status of each API interface of the computer device, obtains the real-time data flow of each API interface, and then the primary classification module represents the real-time data flow of each API interface as different data points to be classified in the feature space; The primary classification module calculates the distance between the data point to be classified and the other data points, takes the average of the distances, compares the average distance with a distance threshold, and determines that the data point to be classified is abnormal data flow when the average distance is greater than the distance threshold; After determining the network speed, the secondary classification module decomposes the bandwidth upper limit according to the number of the API interfaces whose network speed is not zero at the time when the real-time data traffic occurs, obtains the corresponding theoretical speed value, calculates the difference between the network speed of each API interface and the theoretical speed, and determines the weight according to the speed difference of each API interface; Each of the API interfaces is provided with a weight factor, the sum of the weight factors of all the API interfaces is 1, and after the bandwidth upper limit is decomposed, the proportion of the theoretical speed value corresponding to the API interface in the bandwidth upper limit is the corresponding weight factor.

2. A network data analysis system based on deep learning according to claim 1, characterized in that: The distance threshold is the average value of the distances between all other data points.

3. A network data analysis system based on deep learning according to claim 1, characterized in that: The primary classification module performs a classification test on the known data points, and uses the number of the other data points selected when the classification accuracy is the highest as the number of the other data points when calculating the distance between the data point to be classified and the other data points.

4. A network data analysis system based on deep learning according to claim 1, characterized in that: The secondary classification module detects the bandwidth upper limit of the computer device before determining whether the real-time data traffic from multiple API interfaces has reached the bandwidth upper limit.

5. A network data analysis system based on deep learning according to claim 1, characterized in that: The bandwidth upper limit is the upper limit of the network speed, and the secondary classification module determines the network speed according to the data volume and transmission time in the real-time data traffic.

6. A network data analysis system based on deep learning according to claim 1, characterized in that: The weights are determined based on a fuzzy control strategy.

Images