Network traffic prediction and anomaly detection method and device, and readable storage medium

Abstract

The invention provides a network traffic prediction and anomaly detection method and device and a readable storage medium. The method comprises the following steps: acquiring real-time network traffic data; extracting a real-time feature vector of the real-time network flow data; inputting the real-time feature vector into a pre-trained network traffic prediction model to obtain a network traffic prediction result; and performing anomaly detection according to the real-time network traffic data, the network traffic prediction result and a preset data anomaly threshold to obtain an anomaly detection result. Complex network configuration and extra hardware equipment are not needed, the method is easy to understand and implement by network managers, and deployment and maintenance in an actual network environment are facilitated. The abnormal fluctuation in the network flow can be found in time, the potential network attack or fault can be effectively identified, and the security and stability of the network can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a network traffic prediction and anomaly detection method, device and readable storage medium. Background Art

[0002] In the era of rapid development of informatization, a huge amount of network information emerges every day. This information contains both normal and reasonable needs of people and a lot of junk information. These junk information accounts for a large proportion of the network load, which not only puts heavy pressure on network operation, but also threatens personal privacy and security. This type of network traffic is usually regarded as abnormal traffic. The factors that cause abnormal network traffic are complex and include malicious attacks, such as viruses, DoS and DDoS attacks, as well as illegal access behaviors, such as continuous port scanning and remote unauthorized access.

[0003] With the continuous expansion of the Internet and the rapid growth of business volume, the security challenges faced by the network are becoming more complex and changeable. Whether it is a network failure or a malicious attack, it may cause abnormal fluctuations in network traffic.

[0004] Therefore, achieving real-time and accurate monitoring of Internet traffic and timely discovering and reporting abnormal traffic are crucial for network managers to take prompt action and ensure the stable operation of the network. Summary of the invention

[0005] The technical problem to be solved by the present application is to provide a network traffic prediction and anomaly detection method, device and readable storage medium to solve the problems existing in the prior art in view of the above-mentioned deficiencies in the prior art.

[0006] In a first aspect, the present application provides a network traffic prediction and anomaly detection method, the method comprising:

[0007] S1. Obtain real-time network traffic data;

[0008] S2, extracting the real-time feature vector of the real-time network traffic data;

[0009] S3, inputting the real-time feature vector into a pre-trained network traffic prediction model to obtain a network traffic prediction result;

[0010] S4. Perform anomaly detection based on the real-time network traffic data, the network traffic prediction result and a preset data anomaly threshold to obtain an anomaly detection result.

[0011] In some embodiments, in S1, the real-time network traffic data includes transport layer feature data and application layer feature data;

[0012] The transport layer characteristic data includes at least one of the source IP address, the destination IP address, the transport protocol, the flow size and the data packet count;

[0013] The application layer characteristic data includes at least one of an application layer protocol, a domain name, and a URL.

[0014] In some embodiments, after S1 and before S2, the process further includes:

[0015] Preprocessing the real-time network traffic data;

[0016] The preprocessing includes data format conversion and denoising.

[0017] In some embodiments, in S2, the real-time feature vector includes at least one of a traffic feature, a time feature, a network feature, and a statistical feature;

[0018] The traffic characteristics include at least one of the traffic volume, the number of data packets, and the transmission time;

[0019] The time feature includes at least one of hour, day, and week;

[0020] The network feature includes at least one of a source IP, a destination IP, and a protocol type;

[0021] The statistical feature includes at least one of a mean value, a variance, a maximum value, and a minimum value of the flow.

[0022] In some embodiments, in S3, the training process of the network traffic prediction model includes:

[0023] Get historical network traffic data;

[0024] Extracting feature vectors of the historical network traffic data, and dividing the historical network traffic data into a training data set and a test data set according to the feature vectors;

[0025] An initial model is constructed by a logistic regression algorithm, the initial model is trained by a training data set, and the trained model is evaluated and parameter optimized by a test data set to obtain the network traffic prediction model.

[0026] In some embodiments, the network traffic prediction model is expressed as:

[0027] P(Y=1|X)=1 / (1+exp(-(wx+b)))

[0028] Among them, P(Y=1|X) represents the probability that the network traffic data is abnormal under the condition of the feature vector X, w represents the weight vector, x represents the feature vector extracted from the network traffic data, b represents the bias term, and exp represents the exponential function.

[0029] In some embodiments, S4 includes:

[0030] If the difference between the real-time network traffic data and the network traffic prediction result exceeds a preset data anomaly threshold, it is determined that abnormal traffic exists;

[0031] If the difference between the real-time network traffic data and the network traffic prediction result does not exceed a preset data anomaly threshold, it is determined that there is no abnormal traffic.

[0032] In a second aspect, the present application provides a network traffic prediction and anomaly detection device, the device comprising:

[0033] A traffic acquisition module configured to acquire real-time network traffic data;

[0034] A feature extraction module, which is configured to extract a real-time feature vector of the real-time network traffic data;

[0035] A traffic prediction module, which is configured to input the real-time feature vector into a pre-trained network traffic prediction model to obtain a network traffic prediction result;

[0036] The anomaly detection module is configured to perform anomaly detection based on the real-time network traffic data, the network traffic prediction result and a preset data anomaly threshold to obtain anomaly detection results.

[0037] In a third aspect, the present application provides a network traffic prediction and anomaly detection device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the network traffic prediction and anomaly detection method described in the first aspect above.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the network traffic prediction and anomaly detection method described in the first aspect is implemented.

[0039] The network traffic prediction and anomaly detection method, device and readable storage medium provided by the present application include: obtaining real-time network traffic data; extracting real-time feature vectors of the real-time network traffic data; inputting the real-time feature vectors into a pre-trained network traffic prediction model to obtain network traffic prediction results; performing anomaly detection according to the real-time network traffic data, the network traffic prediction results and the preset data anomaly threshold to obtain anomaly detection results. The present application constructs a network traffic prediction model by introducing a logistic regression algorithm and combines it with a training data set for training, which can accurately capture the changing trend of network traffic and achieve high-precision prediction of future network traffic, which helps network managers to understand the network traffic status in advance and provide strong support for the reasonable allocation and optimization of network resources. It is built based on machine learning technology and does not require complex network configuration and additional hardware equipment. It is easy to be understood and implemented by network managers and is easy to deploy and maintain in an actual network environment. The present application sets a data anomaly threshold based on the predicted network traffic results and network traffic data, and uses the threshold to perform anomaly detection on real-time network traffic data, which can timely discover abnormal fluctuations in network traffic, effectively identify potential network attacks or failures, and improve the security and stability of the network. By preprocessing and extracting features from network traffic data, the present application can remove noise and redundant information, extract feature vectors that play a key role in network traffic prediction and anomaly detection, which helps to reduce the complexity and computational complexity of data processing and improve the efficiency and accuracy of data processing; it can process a variety of network traffic data including transport layer feature data and application layer feature data, and has wide applicability; at the same time, by continuously optimizing the network traffic prediction model, it can adapt to the continuous changes in the network environment and maintain high prediction and detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] Figure 1 A flowchart of a network traffic prediction and anomaly detection method provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the structure of a network traffic prediction and anomaly detection device provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the structure of another network traffic prediction and anomaly detection device provided in an embodiment of the present application.

[0044] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solution of the present application, the implementation mode of the present application will be further described in detail below with reference to the accompanying drawings.

[0046] It should be understood that the specific embodiments and drawings described herein are only used to explain the present application, rather than to limit the present application.

[0047] It can be understood that, in the absence of conflict, the various embodiments in the present application and the various features in the embodiments can be combined with each other.

[0048] It can be understood that, for the convenience of description, the drawings of the present application only show the parts related to the present application, while the parts unrelated to the present application are not shown in the drawings.

[0049] It can be understood that each unit and module involved in the embodiments of the present application may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0050] It can be understood that the terms "first", "second", etc. in the embodiments of the present application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0051] It is understandable that, in the absence of conflict, the functions and steps marked in the flowcharts and block diagrams of the present application may occur in an order different from that marked in the drawings.

[0052] It is understood that the flowcharts and block diagrams of the present application illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present application. Among them, each box in the flowchart or block diagram may represent a unit, module, program segment, code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or by a combination of hardware and computer instructions.

[0053] It can be understood that the units and modules involved in the embodiments of the present application can be implemented by software or hardware, for example, the units and modules can be located in a processor.

[0054] This application proposes a network traffic prediction and anomaly detection method based on machine learning, which does not require complex network configuration and additional hardware equipment, is easy to be understood and implemented by network administrators, and is convenient for deployment and maintenance in actual network environments.

[0055] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0056] The present application provides a network traffic prediction and anomaly detection method, the working process of which can be implemented by electronic devices, such as computers, handheld smart terminals, etc. For ease of explanation, the embodiments of the present application are described with the method execution subject being a computer.

[0057] Figure 1 A schematic diagram of a network traffic prediction and anomaly detection method provided in an embodiment of the present application, such as Figure 1 As shown, the present application provides a network traffic prediction and anomaly detection method, the method includes S1-S4, which are as follows:

[0058] S1. Obtain real-time network traffic data;

[0059] In some embodiments, in S1, the real-time network traffic data includes transport layer feature data and application layer feature data;

[0060] The transport layer characteristic data includes at least one of the source IP address, the destination IP address, the transport protocol, the flow size and the data packet count;

[0061] The application layer characteristic data includes at least one of an application layer protocol, a domain name, and a URL.

[0062] In this step, real-time network traffic data is an important part of network monitoring and analysis. The following is an explanation of the above characteristics:

[0063] (1) Transport layer characteristic data:

[0064] Source IP address: The network address that identifies the sender of the data packet.

[0065] Destination IP address: Identifies the network address of the data packet recipient.

[0066] Transmission protocol: refers to the protocol used for data transmission, such as TCP (Transmission Control Protocol), UDP (User Datagram Protocol), etc.

[0067] Traffic size: refers to the total amount of data transmitted in a specific period of time, usually measured in bytes.

[0068] Packet Count: Refers to the number of packets transmitted in a specific period of time.

[0069] Optionally, the transport layer characteristic data may also include the following:

[0070] Port number: Source port number and destination port number, used to identify specific network services or applications.

[0071] TCP flags: such as SYN, ACK, FIN, etc., are used to control the transmission process of data packets.

[0072] Packet transmission time: The time it takes for a packet to be transmitted from the source address to the destination address.

[0073] Transmission rate: the speed of data transmission per unit time.

[0074] (2) Application layer feature data:

[0075] Application layer protocol: refers to the protocol used in the application layer, such as HTTP (Hypertext Transfer Protocol), HTTPS (Secure Hypertext Transfer Protocol), FTP (File Transfer Protocol), etc.

[0076] Domain name: refers to the name of a network server, which is used to identify a website on the Internet.

[0077] URL: Uniform Resource Locator, used to specify the location of a resource on the Internet.

[0078] Optionally, the application layer characteristic data may also include the following:

[0079] Request type: such as GET, POST, etc., indicating the interaction method between the client and the server.

[0080] User Agent: refers to the identifier of the browser or other access tool used by the client.

[0081] Content type: such as text / html, application / json, etc., indicating the type of transmitted data.

[0082] Status code: such as 200, 404, etc., indicating the status of the server response.

[0083] Cookies: text files used to store user session information.

[0084] Through real-time monitoring and analysis of characteristic data at the transport layer and application layer, abnormal behaviors, security threats, performance bottlenecks and other issues in network traffic can be effectively identified, thereby ensuring the stable operation and security of the network.

[0085] In some embodiments, after S1 and before S2, the process further includes:

[0086] Preprocessing the real-time network traffic data;

[0087] The preprocessing includes data format conversion and denoising.

[0088] Specifically, the acquired network traffic data is converted into a unified data format and denoised. In order to facilitate subsequent data analysis and processing, all network traffic data needs to be converted into a standardized data format. This step includes data parsing, format conversion, and integration. At the same time, since network traffic data may contain some noise and redundant information, denoising is also required to eliminate the impact of these unnecessary information on subsequent analysis. Denoising includes data cleaning, outlier detection and processing, etc.

[0089] S2, extracting the real-time feature vector of the real-time network traffic data;

[0090] In some embodiments, in S2, the real-time feature vector includes at least one of a traffic feature, a time feature, a network feature, and a statistical feature;

[0091] The traffic characteristics include at least one of the traffic volume, the number of data packets, and the transmission time;

[0092] The time feature includes at least one of hour, day, and week;

[0093] The network feature includes at least one of a source IP, a destination IP, and a protocol type;

[0094] The statistical feature includes at least one of a mean value, a variance, a maximum value, and a minimum value of the flow.

[0095] Specifically, the real-time feature vector is a key concept in network traffic analysis, which describes the behavior and pattern of network traffic through multi-dimensional features. The following is an explanation of the real-time feature vector:

[0096] (1) Traffic characteristics:

[0097] Traffic size: refers to the total amount of data transmitted within a specific period of time, which can be used to measure network usage.

[0098] Number of packets: The number of packets transmitted in a specific period of time, reflecting the frequency and intensity of network activity.

[0099] Transmission time: The time required for a data packet to travel from the source address to the destination address, which can be used to evaluate network latency.

[0100] Optionally, the traffic characteristics may also include the following:

[0101] Traffic rate: The data transmission rate per unit time, such as the number of bytes transmitted per second.

[0102] Traffic direction: The direction of data flow, such as ingress traffic, egress traffic, or bidirectional traffic.

[0103] Traffic pattern: The periodic or bursty pattern of traffic, such as periodic peak hours.

[0104] (2) Time characteristics:

[0105] Hour: The specific hour when the packet was transmitted, used to analyze network activity at different times of the day.

[0106] Day: The specific date on which the packet was transmitted, used to analyze network activity patterns on time scales such as weeks, months, and years.

[0107] Week: The specific day of the week when the packet was transmitted, used to analyze differences in network activity on different days of the week.

[0108] Optionally, the time feature may also include the following:

[0109] Minute: The specific minute when the packet was transmitted, for more detailed time analysis.

[0110] Month: The specific month in which the packet was transmitted, used for seasonal analysis.

[0111] Holidays: Time characteristics of specific holidays, used to analyze network activities on special dates.

[0112] (3) Network characteristics:

[0113] Source IP: The IP address of the sender of the data packet, used to track the source of network traffic.

[0114] Destination IP: The IP address of the recipient of the data packet, used to track the destination of network traffic.

[0115] Protocol type: The type of network protocol used, such as TCP, UDP, ICMP, etc.

[0116] Optionally, the network characteristics may also include the following:

[0117] Port number: A specific service or application usually uses a fixed port number to identify the network service.

[0118] IP geographic location: The geographic location information corresponding to the IP address, used for geographic distribution analysis.

[0119] Network path: The transmission path of a data packet in the network, used to analyze network topology and routing.

[0120] (4) Statistical characteristics:

[0121] Traffic mean: The average value of traffic volume over a period of time, used to measure the average level of network traffic.

[0122] Traffic variance: The degree of fluctuation of traffic volume over a period of time, used to measure the stability of network traffic.

[0123] Maximum traffic volume: The maximum value of the traffic volume within a period of time, used to identify traffic peaks.

[0124] Minimum flow rate: The minimum flow rate within a period of time, used to identify flow valleys.

[0125] Optionally, the statistical features may also include the following:

[0126] Median: The median value of traffic size, used to describe the central trend of network traffic.

[0127] Quartile: A statistic that describes traffic distribution and is used for more detailed traffic analysis.

[0128] Kurtosis: A statistic that describes the shape of traffic distribution and is used to analyze the sharpness of traffic distribution.

[0129] Skewness: A statistic that describes the symmetry of traffic distribution and is used to analyze the degree of skewness of traffic distribution.

[0130] Through the comprehensive analysis of these real-time feature vectors, network behavior patterns can be more accurately identified, abnormal traffic can be detected, network trends can be predicted, and corresponding management measures can be taken.

[0131] S3, inputting the real-time feature vector into a pre-trained network traffic prediction model to obtain a network traffic prediction result;

[0132] In this step, based on the trained network traffic prediction model, the real-time feature vector of the real-time network traffic data is input to predict the network traffic result at the next moment.

[0133] S4. Perform anomaly detection based on the real-time network traffic data, the network traffic prediction result and a preset data anomaly threshold to obtain an anomaly detection result.

[0134] In some embodiments, S4 includes:

[0135] If the difference between the real-time network traffic data and the network traffic prediction result exceeds a preset data anomaly threshold, it is determined that abnormal traffic exists;

[0136] If the difference between the real-time network traffic data and the network traffic prediction result does not exceed a preset data anomaly threshold, it is determined that there is no abnormal traffic.

[0137] Specifically, S4 includes the following steps:

[0138] S41. According to the predicted network traffic results and network traffic data, a data anomaly threshold is set, and the data anomaly threshold is backtested in combination with the network traffic data;

[0139] S42: input the real-time network traffic data into the network traffic prediction model, and perform anomaly detection on the real-time network traffic data in combination with a set data anomaly threshold.

[0140] The step of setting a data anomaly threshold according to the predicted network traffic results and network traffic data, and backtesting the data anomaly threshold in combination with the network traffic data includes the following steps:

[0141] S411, selecting a section of network traffic data containing normal data and abnormal data as a backtesting data set;

[0142] S412. Based on the set data anomaly threshold, mark all data points in the backtest data set that exceed the threshold, record the data points and compare them with the backtest data set.

[0143] The step of inputting the real-time network traffic data into the network traffic prediction model and performing anomaly detection on the real-time network traffic data in combination with the set data anomaly threshold comprises the following steps:

[0144] S421, predicting the real-time data based on the trained network traffic prediction model to obtain the predicted real-time network traffic result;

[0145] S422, comparing the real-time network traffic result with the set data anomaly threshold, if the real-time network traffic result is lower than the set data anomaly threshold, it is determined to be normal;

[0146] S423: If the real-time network traffic result reaches or exceeds the set data abnormality threshold, it is determined to be abnormal and the abnormal data is recorded.

[0147] It should be explained that the feature vector of the real-time traffic data is compared with the prediction result of the prediction model. If the difference between the actual traffic data and the prediction result exceeds the set threshold or meets the defined abnormal pattern, it indicates that abnormal traffic exists.

[0148] In network traffic prediction, it is necessary to determine whether the network traffic is abnormal. Specifically:

[0149] 1) First, extract the feature vectors of network traffic, which may include traffic rate, packet size distribution, etc.

[0150] 2) Use the training data set to train the logistic regression model and learn the mapping relationship between the feature vector and the label (normal traffic or abnormal traffic).

[0151] 3) In the prediction stage, the new network traffic data is input into the trained model to obtain the probability that the data is abnormal traffic.

[0152] 4) Based on the size of the probability value, it can be determined whether the network traffic is abnormal.

[0153] The following is an explanation of the training process of the network traffic prediction model used in this application.

[0154] In some embodiments, in S3, the training process of the network traffic prediction model includes:

[0155] Get historical network traffic data;

[0156] Extracting feature vectors of the historical network traffic data, and dividing the historical network traffic data into a training data set and a test data set according to the feature vectors;

[0157] An initial model is constructed by a logistic regression algorithm, the initial model is trained by a training data set, and the trained model is evaluated and parameter optimized by a test data set to obtain the network traffic prediction model.

[0158] Among them, the historical network traffic data includes transport layer characteristic data and application layer characteristic data. The transport layer characteristic data includes source IP address, destination IP address, transmission protocol, traffic size and data packet count; the application layer characteristic data includes application layer protocol, domain name and URL.

[0159] Optionally, the acquired historical network traffic data is converted into a unified data format and subjected to denoising. In order to facilitate subsequent data analysis and processing, all network traffic data needs to be converted into a standardized data format. This step includes data parsing, format conversion, and integration. At the same time, since network traffic data may contain some noise and redundant information, denoising is also required to eliminate the impact of these unnecessary information on subsequent analysis. Denoising includes data cleaning, outlier detection and processing, etc.

[0160] Then, feature vectors are extracted from the preprocessed historical network traffic data, and the network traffic data is divided into a training data set and a test data set according to the feature vectors; the network traffic prediction model constructed using the logistic regression algorithm is initialized, and the network traffic prediction model is trained in combination with the training data set; based on the trained network traffic prediction model, the test data set is used for evaluation, and the network traffic prediction model is optimized according to the evaluation results.

[0161] Among them, the feature vector includes traffic characteristics, time characteristics, network characteristics and statistical characteristics. The traffic characteristics include traffic size, number of data packets, and transmission time. The time characteristics include hours, days, and weeks. The network characteristics include source IP, destination IP, and protocol type. The statistical characteristics include the mean, variance, maximum, and minimum values ​​of the traffic.

[0162] In some embodiments, the network traffic prediction model is expressed as:

[0163] P(Y=1|X)=1 / (1+exp(-(wx+b)))

[0164] Among them, P(Y=1|X) represents the probability that the network traffic data is abnormal under the condition of the feature vector X, w represents the weight vector, x represents the feature vector extracted from the network traffic data, b represents the bias term, and exp represents the exponential function.

[0165] It should be explained that in feature extraction and data set partitioning, representative and predictive feature vectors are extracted based on preprocessed network traffic data. These feature vectors cover traffic features (such as traffic size, number of packets, transmission time, etc.), time features (such as time dimensions such as hours, days, and weeks), network features (such as source IP, destination IP, protocol type, and other network identifiers), and statistical features (such as traffic mean, variance, maximum, minimum, and other statistics).

[0166] According to the extracted feature vectors, the network traffic data is divided into a training data set and a test data set. The training data set is used to build and train the network traffic prediction model, while the test data set is used to evaluate the performance of the model. During model building and training, a network traffic prediction model based on the logistic regression algorithm is initialized. The logistic regression algorithm is a statistical method widely used in classification problems. It can predict the changing trend of network traffic by learning the relationship between feature vectors and output labels. Combined with the training data set, the network traffic prediction model is trained. During the training process, the model will continuously adjust the weight vector w and the bias term b to minimize the prediction error and improve the prediction accuracy.

[0167] In this application, a training data set is used to train a logistic regression model, and the goal is to distinguish normal traffic from abnormal traffic. During the training process, regularization techniques (such as L1 regularization and L2 regularization) can be used to prevent the model from overfitting.

[0168] Logistic regression is suitable for processing features with strong linear relationships, so select features that are linearly related to network traffic anomalies, such as traffic growth rate in a specific time period, traffic share of a specific protocol, etc. Remove highly correlated or redundant features to avoid model overfitting. Transform nonlinear features, such as using polynomial features, logarithmic transformation, etc., to make them more suitable for the linear assumption of logistic regression. Encode categorical features, such as using One-Hot Encoding or Label Encoding.

[0169] In addition, the model evaluation and optimization is based on the trained network traffic prediction model and the test data set is used for evaluation. The evaluation indicators include accuracy, recall, F1 score, etc. These indicators can fully reflect the performance of the model. According to the evaluation results, the network traffic prediction model is optimized. The optimization process may include adjusting model parameters, adding feature vectors, improving data preprocessing methods, etc., to improve the prediction ability and generalization performance of the model.

[0170] Logistic distribution: Logistic distribution is a continuous probability distribution. Its probability density function curve is S-shaped and symmetrical around a certain point. In logistic regression, the characteristics of the logistic distribution are used to map the output of the linear model to the interval (0, 1) to obtain the probability of an event occurring.

[0171] Logit: Logit is the logarithm of the ratio of the probability of an event occurring to the probability of the event not occurring. In logistic regression, the relationship between the linear model and probability is established through the logit function.

[0172] Assume there is a linear model z = w x, where w is the weight vector and x is the input feature vector. Map the output z of the linear model through the sigmoid function to get the value of P(Y = 1 | X). The form of the sigmoid function is 1 / (1+exp(-z)).

[0173] In summary, the network traffic prediction model constructed by the logistic regression algorithm can be used to determine whether the network traffic is abnormal, providing strong support for network traffic anomaly detection.

[0174] This application provides a network traffic prediction and anomaly detection method based on machine learning, which has the following beneficial effects:

[0175] 1. This application constructs a network traffic prediction model by introducing the logistic regression algorithm and training it in combination with a training data set. It can accurately capture the changing trend of network traffic and achieve high-precision prediction of future network traffic. This helps network managers understand the network traffic status in advance and provide strong support for the rational allocation and optimization of network resources. It is built based on machine learning technology, does not require complex network configuration and additional hardware equipment, is easy to be understood and implemented by network managers, and is convenient for deployment and maintenance in actual network environments.

[0176] 2. This application sets a data anomaly threshold based on the predicted network traffic results and network traffic data, and uses the threshold to perform anomaly detection on real-time network traffic data, which can timely discover abnormal fluctuations in network traffic, effectively identify potential network attacks or failures, and improve the security and stability of the network.

[0177] 3. This application can remove noise and redundant information by preprocessing and extracting features from network traffic data, and extract feature vectors that play a key role in network traffic prediction and anomaly detection. This helps to reduce the complexity and computational complexity of data processing and improve the efficiency and accuracy of data processing. It can process a variety of network traffic data including transport layer feature data and application layer feature data, and has wide applicability. At the same time, by continuously optimizing the network traffic prediction model, it can adapt to the continuous changes in the network environment and maintain a high prediction and detection performance.

[0178] It should be understood that, although the various steps in the flowcharts in the above-described embodiments are sequentially displayed according to the indications of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily to be carried out sequentially, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0179] Figure 2 A schematic diagram of a network traffic prediction and anomaly detection device provided in an embodiment of the present application, such as Figure 2 As shown, the present application provides a network traffic prediction and anomaly detection device, the device comprising:

[0180] A traffic acquisition module 11, which is configured to acquire real-time network traffic data;

[0181] A feature extraction module 12, which is configured to extract a real-time feature vector of the real-time network traffic data;

[0182] A traffic prediction module 13, which is configured to input the real-time feature vector into a pre-trained network traffic prediction model to obtain a network traffic prediction result;

[0183] The anomaly detection module 14 is configured to perform anomaly detection based on the real-time network traffic data, the network traffic prediction result and a preset data anomaly threshold to obtain an anomaly detection result.

[0184] Regarding the limitation of the network traffic prediction and anomaly detection device, reference may be made to the limitation of the network traffic prediction and anomaly detection method in the above-mentioned embodiments of the present application, which will not be repeated in this embodiment.

[0185] Figure 3 Another schematic diagram of the network traffic prediction and anomaly detection device provided in the embodiment of the present application is as follows Figure 3 As shown, the device includes a memory 22 and a processor 21, the memory stores a computer program, and the processor is configured to run the computer program to execute the methods in the above embodiments of the present application.

[0186] The memory is connected to the processor, the memory may be a flash memory or a read-only memory or other memory, and the processor may be a central processing unit or a single-chip microcomputer.

[0187] In some embodiments, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the methods in the above embodiments of the present application are implemented.

[0188] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0189] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. A network traffic prediction and anomaly detection method, characterized in that: The method comprises: S1. Obtain real-time network traffic data; S2, extracting the real-time feature vector of the real-time network traffic data; S3, inputting the real-time feature vector into a pre-trained network traffic prediction model to obtain a network traffic prediction result; S4. Perform anomaly detection based on the real-time network traffic data, the network traffic prediction result and a preset data anomaly threshold to obtain an anomaly detection result.

2. The network traffic prediction and anomaly detection method according to claim 1, characterized in that: In S1, the real-time network traffic data includes transport layer characteristic data and application layer characteristic data; The transport layer characteristic data includes at least one of the source IP address, the destination IP address, the transport protocol, the flow size and the data packet count; The application layer characteristic data includes at least one of an application layer protocol, a domain name, and a URL.

3. The network traffic prediction and anomaly detection method according to claim 1, characterized in that: After S1 and before S2, it also includes: Preprocessing the real-time network traffic data; The preprocessing includes data format conversion and denoising.

4. The network traffic prediction and anomaly detection method according to claim 1, characterized in that: In S2, the real-time feature vector includes at least one of a traffic feature, a time feature, a network feature, and a statistical feature; The traffic characteristics include at least one of the traffic volume, the number of data packets, and the transmission time; The time feature includes at least one of hour, day, and week; The network feature includes at least one of a source IP, a destination IP, and a protocol type; The statistical feature includes at least one of a mean value, a variance, a maximum value, and a minimum value of the flow.

5. The network traffic prediction and anomaly detection method according to claim 1, characterized in that: In S3, the training process of the network traffic prediction model includes: Get historical network traffic data; Extracting feature vectors of the historical network traffic data, and dividing the historical network traffic data into a training data set and a test data set according to the feature vectors; An initial model is constructed by a logistic regression algorithm, the initial model is trained by a training data set, and the trained model is evaluated and parameter optimized by a test data set to obtain the network traffic prediction model.

6. The network traffic prediction and anomaly detection method according to claim 1, characterized in that: The expression of the network traffic prediction model is: P(Y=1|X)=1 / (1+exp(-(wx+b))) Among them, P(Y=1|X) represents the probability that the network traffic data is abnormal under the condition of the feature vector X, w represents the weight vector, x represents the feature vector extracted from the network traffic data, b represents the bias term, and exp represents the exponential function.

7. The network traffic prediction and anomaly detection method according to claim 1, characterized in that: S4, including: If the difference between the real-time network traffic data and the network traffic prediction result exceeds a preset data anomaly threshold, it is determined that abnormal traffic exists; If the difference between the real-time network traffic data and the network traffic prediction result does not exceed a preset data anomaly threshold, it is determined that there is no abnormal traffic.

8. A network traffic prediction and anomaly detection device, characterized in that: The device comprises: A traffic acquisition module configured to acquire real-time network traffic data; A feature extraction module, which is configured to extract a real-time feature vector of the real-time network traffic data; A traffic prediction module, which is configured to input the real-time feature vector into a pre-trained network traffic prediction model to obtain a network traffic prediction result; The anomaly detection module is configured to perform anomaly detection based on the real-time network traffic data, the network traffic prediction result and a preset data anomaly threshold to obtain anomaly detection results.

9. A network traffic prediction and anomaly detection device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the network traffic prediction and anomaly detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the network traffic prediction and anomaly detection method according to any one of claims 1 to 7 is implemented.

Images