A method and apparatus for detecting network traffic

Abstract

The application relates to the technical field of network flow management, and discloses a network flow detection method and device. The method comprises the following steps: a gateway acquires to-be-detected network flow corresponding to a specified port on a server; the gateway classifies and processes to-be-detected flow data according to a preset rule to obtain flow data corresponding to each flow type, wherein the preset rule is a classification rule set for the flow type; the gateway performs abnormality detection on the flow data of each flow type to obtain to-be-distributed flow data; the gateway distributes the to-be-distributed flow data to a service node associated with the corresponding flow type; and the service node performs abnormality detection on the flow data received from the gateway when the micro service of the service node is started. Through implementation of the application, the gateway and each service node can directly perform independent abnormality detection on the flow data, and the detection time of malicious flow is saved, and the detection efficiency of the malicious flow is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network flow management, and in particular to a network flow detection method and device. BACKGROUND

[0002] Malicious software such as malicious crawlers, automata, and simulators is increasingly threatening the healthy development of the Internet. Malicious software usually attacks websites, APPs, or APIs at the application layer to obtain Internet data by initiating requests from fake users, in order to achieve the purpose of profit. In order to ensure that private data is not snooped, encryption means are usually used to protect private data. However, encryption can hide malicious software like other private information, thereby bringing a series of malicious flows such as worms, trojans, and viruses.

[0003] In order to deal with this problem, the common method at present is to decrypt the flow and use devices such as new-generation firewalls to view the flow. A detection center can also be set up in the cloud to forward the flow to be detected to the detection center, and a centralized flow management detection method is used for centralized detection, or a firewall is set up at the gateway of the flow entrance to process the flow with specific rules to identify potential security threats hidden in encrypted flow. The user secret key can also be authorized to decrypt the input flow, and then the decrypted data is detected to determine the security of each flow. However, decrypting the flow takes a long time and requires additional devices to be added to the network. All flows in the centralized flow management detection method need to be forwarded and detected by the central control, and the detection efficiency is low. In the authorized decryption method, the flow is decrypted for malicious flow analysis and then transmitted, resulting in a large transmission performance loss. SUMMARY

[0004] Therefore, the embodiments of the present application provide a network flow detection method and device to solve the problems of long encryption flow detection time, low efficiency, and large transmission loss.

[0005] According to a first aspect, the embodiments of the present application provide a network flow detection method, which is applied to a server, the server is provided with a gateway and service nodes corresponding to a plurality of microservices, different microservices correspond to different service nodes, and the method comprises the following steps: the gateway acquires network flow to be detected corresponding to a specified port on the server; the gateway classifies and processes the flow data to be detected according to a preset rule to obtain flow data corresponding to each flow type; the preset rule is a classification rule set for the flow type; the gateway performs anomaly detection on the flow data of each flow type to obtain flow data to be distributed; the gateway distributes the flow data to be distributed to the service node associated with the corresponding flow type; and the service node performs anomaly detection on the flow data received from the gateway when the microservice of the service node is started.

[0006] The method for detecting network flow provided by the embodiments of the present application comprises: a server gateway obtains to-be-detected network flow corresponding to a server specified port, and classifies the to-be-detected network flow according to classification rules set for flow types, to obtain flow data corresponding to each flow type, and performs abnormality detection on the flow data of each flow type, and distributes to-be-distributed flow data obtained through the abnormality detection to service nodes associated with the flow types, and the service nodes perform abnormality detection on the flow data received by the service nodes when the micro services of the service nodes are started. In the method, the gateway in the server and each service node can directly perform independent abnormality detection on the flow data, without the need for detection by other means, thereby saving the detection time of malicious flow, improving the detection efficiency of malicious flow, and the service nodes automatically perform abnormality detection on the corresponding flow data along with the start of the micro services, thereby realizing the automation of abnormality detection, without the need for manual operation, and avoiding the problem of large transmission performance loss caused by the need for subsequent transmission of the flow data after the completion of abnormality detection, thereby ensuring the transmission performance of the flow data.

[0007] In combination with the first aspect, in a first implementation manner of the first aspect, the gateway performs abnormality detection on the flow data of each flow type to obtain to-be-distributed flow data, comprising: the gateway extracts first flow features corresponding to the flow data of each flow type; inputs the first flow features into a flow abnormality detection model to obtain a flow detection result; and determines the to-be-distributed flow data based on the flow detection result; wherein the flow abnormality detection model is obtained by training a preset classification model based on abnormal flow feature samples.

[0008] In a second implementation form of the first aspect, in conjunction with the first implementation form of the first aspect, the gateway extracts the first traffic feature corresponding to the traffic data of each traffic type, including: extracting the data packet of the flow corresponding to the traffic data of each traffic type, the data packet of the flow including flow metadata, and the flow metadata including the number of bytes, the number of packets, port information, and the duration of the flow; analyzing the data packet of the flow based on a preset frequency to determine the distribution feature corresponding to the data packet of the flow, the distribution feature being used to represent the distribution state of each item of metadata in the data packet of the flow; or extracting the data packet of the flow corresponding to the traffic data of each traffic type, the data packet of the flow including flow metadata, and the flow metadata including the number of bytes, the number of packets, port information, and the duration of the flow; analyzing the data packet of the flow based on a preset frequency to determine the distribution feature corresponding to the data packet of the flow, the distribution feature being used to represent the distribution state of each item of metadata in the data packet of the flow; and extracting the handshake data packet of the security transmission layer protocol corresponding to the traffic data of each traffic type; parsing the handshake data packet to obtain the handshake information corresponding to the handshake data packet.

[0009] The network traffic detection method provided in the embodiments of the present application can ensure the accuracy of the abnormal detection of the traffic data by the gateway extracting the traffic features corresponding to the traffic data of each traffic type and inputting the traffic features into the traffic anomaly detection model to obtain the traffic detection result.

[0010] In a third implementation form of the first aspect, in conjunction with the first aspect, the service node performs abnormal detection on the traffic data received from the gateway when the micro service of the service node is started, including: the service node extracting the second traffic feature corresponding to the traffic data received from the gateway; inputting the second traffic feature into the traffic anomaly detection model to obtain the traffic detection result; and the traffic anomaly detection model being obtained by training a preset classification model based on abnormal traffic feature samples.

[0011] In a fourth implementation form of the first aspect, in combination with the third implementation form of the first aspect, the service node extracting the second traffic feature corresponding to the traffic data received from the gateway comprises: the service node extracting data packets of a flow corresponding to the traffic data received from the gateway, the data packets of the flow comprising flow metadata, the flow metadata comprising a byte number, a packet number, port information, and a flow duration; analyzing the data packets of the flow based on a preset frequency to determine a distribution feature corresponding to the data packets of the flow, the distribution feature being used to represent a distribution state of each item of metadata in the data packets of the flow; or the service node extracting data packets of a flow corresponding to the traffic data received from the gateway, the data packets of the flow comprising flow metadata, the flow metadata comprising a byte number, a packet number, port information, and a flow duration; analyzing the data packets of the flow based on a preset frequency to determine a distribution feature corresponding to the data packets of the flow, the distribution feature being used to represent a distribution state of each item of metadata in the data packets of the flow; and extracting handshake data packets of a secure transmission layer protocol corresponding to the traffic data received from the gateway; and parsing the handshake data packets to obtain handshake information corresponding to the handshake data packets.

[0012] The method for detecting network traffic provided in the embodiments of the present application can ensure that each service node can directly detect encrypted malicious traffic by extracting traffic features corresponding to traffic data from a gateway and inputting the traffic features into a traffic anomaly detection model to obtain a traffic detection result. The traffic anomaly detection model is deployed on each service node to detect traffic data, and the traffic anomaly detection model can be started along with the start of a service, thereby further ensuring that each service node can directly detect encrypted malicious traffic. The service node extracts multiple dimensions of traffic features corresponding to each type of traffic data, thereby ensuring the accuracy of traffic data anomaly detection.

[0013] In combination with the first aspect, in a fifth implementation form of the first aspect, the gateway obtaining the to-be-detected network traffic corresponding to a specified port of a server comprises: the gateway obtaining all traffic data packets received by a specified port on a server; determining whether there is an incomplete traffic data packet in the all traffic data packets; when there is an incomplete traffic data packet, filtering out the incomplete traffic data packet to obtain complete traffic data packets, and determining the complete traffic data packets as the to-be-detected network traffic.

[0014] The network flow detection method provided by the embodiment of the present application, the gateway receives all flow data packets on a specified port of a server, when there is an incomplete flow data packet in the received flow data packet, the incomplete flow data packet is filtered out, the complete flow data packet is obtained, the complete flow data packet is determined as the network flow to be detected, the gateway can extract the complete network flow from the flow data, and the detection efficiency and detection accuracy of the flow data are further ensured.

[0015] According to a second aspect, the embodiment of the present application provides a network flow detection device, the device is applied to a server, the device comprises: a gateway, configured to obtain network flow to be detected corresponding to a specified port on the server; perform classification processing on the network flow to be detected according to a preset rule, and obtain flow data corresponding to each flow type; the preset rule is a classification rule set for the flow type; perform anomaly detection on the flow data of each flow type, and obtain network flow to be distributed; distribute the network flow to be distributed to a service node associated with the corresponding flow type; a service node, configured to perform anomaly detection on flow data received from the gateway when the micro service of the service node is started.

[0016] In combination with the second aspect, in a first implementation manner of the second aspect, the gateway is further configured to extract first flow features corresponding to the flow data of each flow type; input the first flow features into a flow anomaly detection model to obtain a flow detection result; and determine the network flow to be distributed based on the flow detection result; wherein the flow anomaly detection model is obtained by training a preset classification model based on abnormal flow feature samples.

[0017] In combination with the second aspect, in a second implementation manner of the second aspect, the service node is further configured to extract second flow features corresponding to the flow data received from the gateway; input the second flow features into a flow anomaly detection model to obtain a flow detection result; wherein the flow anomaly detection model is obtained by training a preset classification model based on abnormal flow feature samples.

[0018] In combination with the second aspect, in a third implementation manner of the second aspect, the gateway is further configured to obtain all flow data packets received on a specified port of a server; determine whether there is an incomplete flow data packet in the all flow data packets; when there is an incomplete flow data packet, filter out the incomplete flow data packet, obtain complete flow data packets, and determine the complete flow data packets as the network flow to be detected.

[0019] It should be noted that the corresponding beneficial effects of the network flow detection provided by the embodiment of the present application are described in the description of the network flow detection method, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0021] Figure 1 The structural schematic diagram of the server in the embodiment of the present application is shown;

[0022] Figure 2 The flow chart of the network traffic detection method according to the embodiment of the present application is shown;

[0023] Figure 3 The flow chart of the network traffic detection method according to the embodiment of the present application is shown;

[0024] Figure 4 The flow chart of the network traffic detection method according to the embodiment of the present application is shown;

[0025] Figure 5 The test result diagram of the network traffic detection method according to the embodiment of the present application is shown;

[0026] Figure 6 The structural block diagram of the network traffic detection device according to the embodiment of the present application is shown;

[0027] Figure 7 The structural block diagram of the network traffic detection system according to the embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the present application.

[0029] Due to the encryption can hide malware like other private information, thus bring a series of worm, Trojan and virus and other malicious traffic. In order to deal with this problem, the current common method is to decrypt the traffic, and use the device such as the next generation firewall to view the traffic, can also set a detection center in the cloud, the traffic to be detected is forwarded to the detection center, using centralized traffic management detection method for centralized detection, or in the gateway of the traffic entrance, set a firewall, with specific rules to deal with the traffic, to identify the security threat hidden in the encrypted traffic, can also be authorized to obtain the user secret key, the input traffic is first decrypted, and then the decrypted data is detected to judge the security of each traffic. But the above method has certain defects.

[0030] Based on this, the technical scheme of the present application sets up network traffic detection devices in the gateway and service nodes of the server to simultaneously detect the external traffic and internal traffic, and the gateway and each service node can directly detect the traffic data independently without the help of other methods, saving the detection time of malicious traffic, improving the detection efficiency of malicious traffic, and ensuring the transmission performance of traffic data.

[0031] According to the embodiment of the present application, an embodiment of a network traffic detection method is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from here.

[0032] In this embodiment, a network traffic detection method is provided, which can be used in a server, the server is provided with a gateway and a plurality of service nodes corresponding to microservices, different microservices correspond to different service nodes, so that the network traffic detection method can be used for detecting the network traffic of the server. Figure 1As shown in the server architecture diagram, a lightweight traffic detection device is deployed in the gateway and each microservice in the form of a sidecar proxy. The traffic detection device can be configured on demand through the proxy configuration file of the traffic control middleware, without frequent manual operation. After the configuration of the traffic detection device is completed, it can be started along with the start of the microservice. Specifically, a gateway proxy is arranged at the gateway traffic entrance to receive external network traffic and input it to the network traffic detection device arranged in the gateway for traffic anomaly detection. A proxy is also arranged at the traffic entrance of each service node to receive the traffic data distributed by the gateway, and a network traffic detection device is arranged in each microservice corresponding service node to detect the traffic data received by each service node (e.g., service node 1, service node 2, and service node 3). In addition, each service node can also communicate with each other, and the network traffic detection device arranged in the service node can also detect the traffic data communicated between the service nodes. The traffic data anomaly detection can be started along with the start of the microservice of the service node.

[0033] Figure 2 is a flowchart of network traffic detection according to an embodiment of the present application, as shown in the flowchart, the flow includes the following steps: Figure 2

[0034] S11, the gateway obtains the to-be-detected network traffic corresponding to the specified port on the server.

[0035] The to-be-detected network traffic is external encrypted traffic received by the specified port on the server, and the specified port is the port of the server connected to the gateway. The to-be-detected network traffic can be determined by capturing the traffic data packet of the specified port on the server through the network traffic detection device arranged in the gateway. The network traffic detection device can be arranged in the proxy module of the gateway, and the to-be-detected network traffic is forwarded and detected through the proxy module.

[0036] S12, the gateway classifies the to-be-detected traffic data according to a preset rule to obtain traffic data corresponding to each traffic type, wherein the preset rule is a classification rule set for the traffic type.

[0037] When the gateway in the server obtains the to-be-detected network traffic, the classification rule set for the traffic type can be called to classify the to-be-detected network traffic to obtain traffic data of each traffic type. The classification rule can be a traffic feature or a service identifier corresponding to the traffic data, and the preset rule is not limited here.

[0038] S13, the gateway detects the traffic data of each traffic type for anomaly to obtain to-be-distributed traffic data.

[0039] ​The gateway in the server can perform anomaly detection on the traffic data of each traffic type to determine whether the traffic data of each traffic type currently contains malicious traffic, and when the gateway determines that the traffic data of each traffic type contains malicious traffic, the malicious traffic is discarded to obtain the to-be-distributed traffic data, and if the gateway determines that the traffic data of each traffic type does not contain malicious traffic, the traffic data corresponding to each traffic type is determined as the to-be-distributed traffic data.

[0040] In S14, the gateway distributes the to-be-distributed traffic data to the service node associated with the corresponding traffic type.

[0041] Different microservices correspond to different service nodes, and the traffic types corresponding to different microservices are different. The gateway can distribute the to-be-distributed traffic data determined by the gateway to the service nodes associated with each traffic type. The network traffic detection device arranged in the gateway of the server forwards the traffic data of each traffic type to the corresponding service node after determining the traffic data.

[0042] In S15, the service node performs anomaly detection on the traffic data received from the gateway when the microservice of the service node is started.

[0043] Each service node is provided with a network traffic detection device to perform anomaly detection on the traffic data received by the service node, and to forward or discard the traffic data according to the detection result of the traffic data. The network traffic detection device is started in the process of starting the microservice of the service node, and is simultaneously exited with the exit of the microservice. After receiving the traffic data from the gateway, each service node in the server can directly perform anomaly detection on the traffic data through the network traffic detection device arranged in the service node to determine whether the traffic data is malicious traffic or contains malicious traffic. The anomaly detection is automatically generated along with the start of the microservice, thereby realizing independent and automated detection of the traffic data by each service node, and the entire detection process does not require human operation.

[0044] The network flow detection method provided in the embodiment comprises the following steps:

[0045] A network flow detection method is provided in the embodiment, which can be used for a server. Figure 1 The server is provided with a gateway and service nodes corresponding to a plurality of microservices. Figure 3 The flowchart of the network flow detection according to the embodiment of the application is shown in Figure 3 The flowchart comprises the following steps:

[0046] In S21, the gateway obtains the network flow to be detected corresponding to a specified port on the server. For details, refer to the related description of the corresponding step S11 in the above embodiment, which will not be repeated here.

[0047] In S22, the gateway classifies the flow data to be detected according to a preset rule to obtain flow data corresponding to each flow type, wherein the preset rule is a classification rule set for the flow type. For details, refer to the related description of the corresponding step S12 in the above embodiment, which will not be repeated here.

[0048] In S23, the gateway performs abnormality detection on the flow data of each flow type to obtain flow data to be distributed.

[0049] Specifically, the above step S23 can comprise the following steps:

[0050] In S231, the gateway extracts a first flow feature corresponding to the flow data of each flow type.

[0051] The first traffic feature is a feature corresponding to the traffic data of each traffic type determined by the gateway. The gateway can extract features from the traffic data obtained by the traffic analysis tool to determine whether malicious traffic exists in the traffic data. Specifically, the first traffic feature can include associated packet attribute information, domain name resolution attribute information, and Transport Layer Security (TLS) metadata. The domain name resolution attribute information can include associated Hyper Text Transfer Protocol (HTTP) attributes and associated Domain Name System (DNS) attributes, and the TLS metadata can include TLS flow statistical features and TLS flow handshake information.

[0052] Specifically, the above step S241 can include:

[0053] (1) Extracting the data packets of the flow corresponding to the traffic data of each traffic type, the data packets of the flow including flow metadata, the flow metadata including byte number, packet number, port information, and duration.

[0054] The flow data packets can be parsed from the traffic data NetFlow of each traffic type by the traffic analysis tool to determine the flow metadata Flow Metadata, i.e., the function independent of TLS. Specifically, the flow metadata can include byte number, packet number, port information, and duration. The byte number is the number of inbound and outbound bytes Nb of inbound bytes & Nb of outbound bytes, i.e., the number of bytes transmitted in and out in this session; the packet number is the number of inbound and outbound packets Nb of inbound packets & Nb of outbound packets, i.e., the number of packets transmitted in and out in this session; the port information includes an ephemeral source port Ephemeral src port, which is usually randomly selected by the operating system in the range of 49152-65535, and a TLS destination port Usual TLS destination port, which is used for the server to listen to the TLS connection on a specific port, and any content in the port can be considered suspicious; and the duration is the duration of the flow Flow duration, i.e., the duration of this session.

[0055] (2) Analyzing the data packets of the flow based on a preset frequency to determine the distribution feature corresponding to the data packets of the flow, the distribution feature being used to represent the distribution state of each item of metadata in the data packets of the flow.

[0056] The preset frequency is used to represent the time interval of extracting the packet related information, and the distribution feature is the result of analyzing the flow packets at the preset frequency, which is determined according to the extracted packet related information and cannot be directly extracted from the packet. The preset frequency can be determined according to an empirical value, which is not limited here.

[0057] Specifically, the related information of the flow packets can include a sequence of packet length, a sequence of packet time, a byte distance mean, a byte distance standard deviation, and a byte entropy. The sequence of packet length refers to that, for a packet, its length is counted in an interval, for example, the length of a packet is 100, and the interval interval is set to 150, so the packet with a length of 100 will be counted in the packet length interval [0, 150], and so on, the second interval is [151, 300]…; the sequence of packet time is consistent with the sequence of packet length, and the sequence of packet length is replaced by the arrival time of the packet; the byte distance mean is an array with a length of 256, which is used to count each byte value encountered in the payload of the packet and calculate the mean; the byte distance standard deviation is an array with a length of 256, which is used to count each byte value encountered in the payload of the packet and calculate the standard deviation; and the byte entropy is an array with a length of 256, which is used to count each byte value encountered in the payload of the packet and calculate the entropy.

[0058] After obtaining the metadata of the flow and the distribution feature corresponding to the flow packets, the port information and the duration in the metadata of the flow are used to represent the domain name resolution attribute information, and the number of bytes, the number of packets, and the distribution feature corresponding to the flow packets in the metadata of the flow are used to represent the associated packet attribute.

[0059] Optionally, the above step S241 can further include:

[0060] (3) Extracting the security transmission layer protocol handshake packet corresponding to the flow data of each flow type.

[0061] The flow data is composed of multiple packets, and the TLS metadata corresponds to the TLS handshake packet. The gateway can extract the corresponding TLS handshake packet from the flow data of each flow type based on the flow analysis tool, wherein the TLS handshake packet includes ClientHello, ServerHello, and Certificate.

[0062] (4) Analyzing the handshake data packet to obtain handshake information corresponding to the handshake data packet.

[0063] The TLS handshake data packet is analyzed to extract the handshake information corresponding to the TLS handshake data packet, which includes TLS flow statistical features and TLS flow handshake information, and the TLS flow statistical features and TLS flow handshake information are used as characteristic information representing the TLS metadata. Specifically, the TLS flow statistical features and TLS flow handshake information are included in the following characteristic information: Ciphersuites, which includes asymmetric encryption algorithms, symmetric encryption algorithms, and data digest algorithms; Extensions, which are used to declare support for certain new functions of the protocol or carry additional data required during handshake; the number of extensions Nb of extensions, i.e. the number of extensions supported by the TLS certificate; Supported Groups, which belongs to the extension of TLS, mainly used to identify the application program of the client and notify the server on the elliptic curve supported by the client; Point Formats, which belongs to the extension of TLS, mainly used to notify the server of the compact representation of the elliptic curve point; Client’s key length; Certificate’s validity, which is the number of days from the expiration time of the certificate to the effective time of the certificate; Certificate’s nb of SAN, which represents the number of domain names allowed to be covered by the certificate; Self-signed certificate, which is used to indicate whether the certificate is a self-signed certificate.

[0064] S232, input the first flow feature into the flow anomaly detection model to obtain a flow detection result, wherein the flow anomaly detection model is obtained by training a preset classification model based on abnormal flow feature samples.

[0065] The first flow features extracted from each type of flow data are combined into a feature vector, which is input into the anomaly detection model. The anomaly detection model can output a corresponding numerical value, such as a value between 0 and 1, based on the input feature vector. The numerical value is used to determine whether the flow data corresponding to the feature vector is malicious flow.

[0066] Specifically, the flow anomaly detection model can be an online machine learning model XGBoost[2], whose objective function consists of two parts: the first part is the loss of gradient boosting algorithm, and the second part is the regularization term. Wherein. The loss function is defined as:

[0067]

[0068] Wherein, L(φ) is a loss function, n is the number of training function samples, that is, the number of feature vectors composed of traffic characteristics, and l is a loss function for a single sample, such as a convex function, y i is a predicted value of the model for the training sample, y i is a true label value of the training sample, and Ω(f) is a regularization term.

[0069] Wherein, the regularization term defines the complexity of the model:

[0070]

[0071] Wherein, γ and λ are pre-set parameters, w is a vector formed by all leaf node values of the decision tree, and T is the number of leaf nodes in the decision tree.

[0072] In training the traffic anomaly detection model, the extracted traffic characteristic samples are sequentially subjected to feature verification, feature selection, and feature standardization. The feature verification mainly checks the legality and effectiveness of the feature information, such as whether the length, type, and value range of the data are reasonable. The feature selection eliminates less important traffic characteristics and retains important traffic characteristics, which on the one hand reduces the training data, helping to speed up the model training, and on the other hand reduces redundant features, improving the accuracy. The feature standardization performs hash processing on character and string type traffic characteristics to unify them into corresponding dimensions, and performs standardization on numerical type traffic characteristics according to their maximum and minimum values. Finally, the traffic characteristics subjected to feature verification, feature selection, and feature standardization are combined into a feature vector to train an online machine learning model, obtaining a traffic anomaly detection model. After the traffic anomaly detection model is trained, the joblib library is used to persist the model, and the server is run to load the traffic anomaly detection model to the gateway.

[0073] S233, based on the traffic detection result, determining the to-be-distributed traffic data.

[0074] The gateway performs anomaly detection on the traffic data of each traffic type to obtain a traffic detection result, and determines whether the traffic data of each traffic type contains malicious traffic based on the traffic detection result. If it contains malicious traffic, the malicious traffic is discarded, and the traffic data after removing the malicious traffic is determined as the to-be-distributed traffic data; if it does not contain malicious traffic, the traffic data of each traffic type is determined as the to-be-distributed traffic data.

[0075] S24, the gateway distributes the to-be-distributed traffic data to the service node associated with the corresponding traffic type. For details, see the related description of step S14 in the above embodiment, which will not be repeated here.

[0076] S25, the service node performs anomaly detection on the traffic data received from the gateway when the microservice of the service node starts. For details, refer to the description of step S15 in the above embodiment, which will not be repeated here.

[0077] Specifically, the above step S25 can include:

[0078] S251, the service node extracts the second traffic feature corresponding to the traffic data received from the gateway.

[0079] The second traffic feature is a feature corresponding to the traffic data received from the gateway by the service node. The service node can extract the feature from the traffic data received from the gateway by the traffic analysis tool to determine whether there is malicious traffic in the traffic data.

[0080] Specifically, the above step S251 can include:

[0081] (1) The service node extracts the data packet of the flow corresponding to the traffic data received from the gateway, and the data packet of the flow includes flow metadata, and the flow metadata includes byte number, packet number, port information, and flow duration.

[0082] (2) Analyze the data packet of the flow based on a preset frequency to determine the distribution feature corresponding to the data packet of the flow, and the distribution feature is used to represent the distribution state of each metadata in the data packet of the flow.

[0083] Optionally, the above step S251 can include:

[0084] (3) Extract the handshake data packet of the secure transmission layer protocol corresponding to the traffic data received from the gateway.

[0085] (4) Analyze the handshake data packet to obtain the handshake information corresponding to the handshake data packet.

[0086] The service node extracts the second traffic feature in the same way as the first traffic feature. For details, refer to the description of the extraction of the first traffic feature, which will not be repeated here.

[0087] S252, input the second traffic feature into the traffic anomaly detection model to obtain a traffic detection result, wherein the traffic anomaly detection model is obtained by training a preset classification model based on abnormal traffic feature samples.

[0088] The service node inputs the second traffic feature into the traffic anomaly detection model for anomaly detection in the same way as the gateway inputs the first traffic feature into the traffic anomaly detection model for anomaly detection. For details, refer to the above description, which will not be repeated here.

[0089] The network flow detection method provided in the embodiment of the application, the gateway extracts the flow characteristics corresponding to the flow data of each flow type, inputs the flow characteristics into the flow anomaly detection model to obtain a flow detection result. By deploying the flow anomaly detection model on the gateway for anomaly detection of flow data, the malicious flow detection pressure of subsequent service nodes is further reduced. The gateway extracts the flow characteristics of multiple dimensions corresponding to the flow data of each flow type, thereby ensuring the accuracy of anomaly detection of flow data.

[0090] The service node extracts the flow characteristics corresponding to the flow data from the gateway, inputs the flow characteristics into the flow anomaly detection model to obtain a flow detection result. By deploying the flow anomaly detection model on each service node for anomaly detection of flow data, and by deploying the flow anomaly detection model in the service node, the service node can be started along with the start of the service, thereby further ensuring that each service node can directly detect encrypted malicious flow. The service node extracts the flow characteristics of multiple dimensions corresponding to the flow data of each flow type, thereby ensuring the accuracy of anomaly detection of flow data.

[0091] In the embodiment, a network flow detection method is provided, which can be used in a server. The server is provided with a gateway and service nodes corresponding to multiple microservices. Different microservices correspond to different service nodes, as shown in Figure 1 . Figure 4 The flowchart of network flow detection according to the embodiment of the application is shown in Figure 4 , and the flowchart includes the following steps:

[0092] S31, the gateway obtains the network flow to be detected corresponding to a specified port on the server.

[0093] Specifically, the above step S31 can include:

[0094] S311, the gateway obtains all flow data packets received by the specified port on the server.

[0095] The gateway extracts all flow data packets received by the specified port on the server through a packet capturing tool. For example, the server gateway can capture the flow data packets received by the specified port on the server through wireshark.

[0096] S312, it is judged whether there is an incomplete flow data packet in all flow data packets.

[0097] Since there can be incomplete data packets in the traffic data packets received by the gateway, the gateway can discard the incomplete data packets, and the traffic information carried by the incomplete data packets is also incomplete. Therefore, when the traffic data packets are captured, the gateway can perform integrity detection on each traffic data packet to determine whether there is an incomplete traffic data packet. When there is an incomplete traffic data packet, step S313 is performed, otherwise all traffic data packets are determined as network traffic to be detected.

[0098] S313, filtering out incomplete traffic data packets to obtain complete traffic data packets, and determining the complete traffic data packets as network traffic to be detected.

[0099] The preset rule is a filtering rule for incomplete traffic data packets. The server gateway can filter the incomplete traffic data packets captured by the server gateway according to the set filtering rule, so as to ensure that the traffic data packets captured by the server gateway are complete data packets, and then analyze the complete data packets by using a traffic analysis tool to extract network traffic corresponding to each complete traffic data packet.

[0100] S32, the gateway classifies the traffic data to be detected according to a preset rule to obtain traffic data corresponding to each traffic type, wherein the preset rule is a classification rule set for the traffic type. For details, refer to the related description of step S12 in the above embodiment, which will not be repeated here.

[0101] S33, the gateway performs anomaly detection on the traffic data of each traffic type to obtain traffic data to be distributed.

[0102] S34, the gateway distributes the traffic data to be distributed to a service node associated with the corresponding traffic type. For details, refer to the related description of step S14 in the above embodiment, which will not be repeated here.

[0103] S35, the service node performs anomaly detection on the traffic data received from the gateway when the microservice of the service node is started. For details, refer to the related description of step S15 in the above embodiment, which will not be repeated here.

[0104] The network traffic detection method provided in the embodiment can ensure that the gateway can extract complete network flows from the traffic data, and further ensure the detection efficiency and detection accuracy of the traffic data.

[0105] The network traffic detection method is used for testing samples from a Huangyun environment, wherein the number of positive samples includes 234 locally collected samples, 7229 open source papers, 28149 IXIA tester traffic, and a total of 35612; the number of negative samples includes 349 Lastline (a security company), 8083 MTA (a website for analyzing malicious traffic), 7848 Stratoshpere (a security company), 4854 IXIA tester traffic, and a total of 21134. The network traffic detection method is compared with the existing disclosed technical solutions, and the experimental results are shown in Figure 5 The False Positive Rate indicates that the model is misclassified and is a negative example itself, and the True Positive Rate indicates that the model is correctly classified and is a positive example itself. According to the experimental results, the network traffic detection method provided in the technical solution can obviously ensure the accuracy of the abnormal detection of the traffic data.

[0106] In the embodiment, a network traffic detection device is also provided, which is used to implement the above-described embodiments and preferred embodiments, and will not be described again. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and is conceived.

[0107] The embodiment provides a network traffic detection device, which is used for a server as shown in Figure 1 , and the device includes Figure 6 .

[0108] A gateway is configured to obtain the network traffic to be detected corresponding to a specified port on the server, perform classification processing on the traffic data to be detected according to a preset rule to obtain traffic data corresponding to each traffic type, perform abnormal detection on the traffic data of each traffic type to obtain traffic data to be distributed, and distribute the traffic data to be distributed to a service node associated with the corresponding traffic type. For details, refer to the related description of the above-described embodiments, which will not be described again.

[0109] The service node 42 is configured to perform abnormal detection on the traffic data received from the gateway when the microservice of the service node 42 is started. For details, refer to the related description of the above-described embodiments, which will not be described again.

[0110] The network traffic detection device provided in the embodiment can directly perform independent abnormality detection on the traffic data by each service node in the server, without the need of detection by other means, thereby saving the detection time of malicious traffic, improving the detection efficiency of malicious traffic, and automatically performing the abnormality detection of the corresponding traffic data by the service node along with the start of the micro service, thereby realizing the automation of the abnormality detection, without the need of manual operation, and avoiding the problem of large transmission performance loss caused by the subsequent transmission of the traffic data after the completion of the abnormality detection, thereby guaranteeing the transmission performance of the traffic data.

[0111] Optionally, the gateway 41 is further configured to extract the first traffic feature corresponding to the traffic data of each traffic type, input the first traffic feature into the traffic abnormality detection model to obtain a traffic detection result, and determine the to-be-distributed traffic data based on the traffic detection result, wherein the traffic abnormality detection model is obtained by training a preset classification model based on abnormal traffic feature samples. For details, refer to the related description of the above embodiment, which will not be repeated here.

[0112] Optionally, the service node 42 is further configured to extract the second traffic feature corresponding to the traffic data received from the gateway, input the second traffic feature into the traffic abnormality detection model to obtain a traffic detection result, and wherein the traffic abnormality detection model is obtained by training a preset classification model based on abnormal traffic feature samples. For details, refer to the related description of the above embodiment, which will not be repeated here.

[0113] Optionally, the gateway is further configured to obtain all traffic data packets received by a specified port on the server, determine whether there is an incomplete traffic data packet in all the traffic data packets, filter out the incomplete traffic data packet when there is an incomplete traffic data packet, obtain complete traffic data packets, and determine the complete traffic data packets as the to-be-detected network traffic. For details, refer to the related description of the above embodiment, which will not be repeated here.

[0114] The further function of the above gateway and service node is the same as that of the above corresponding embodiment, which will not be repeated here.

[0115] The embodiment provides a network traffic detection system, as shown in Figure 7As shown, it comprises: a server 51 and a control device 52, the server 51 and the control device 52 are in communication connection; the server 51 is provided with a gateway 511 and a plurality of service nodes (service node 1, service node 2 and service node 3, etc.) in communication connection with the gateway 511, and different service nodes correspond to different types of microservices. The proxy in the gateway 511 can obtain the network traffic to be detected corresponding to the specified port on the server, and the gateway 511 can classify the to-be-detected traffic data according to the preset rules to obtain traffic data corresponding to each traffic type, and input the traffic data of each traffic type into the network traffic detection device provided in the gateway 511, so that the network traffic detection device detects the traffic data of each traffic type, that is, the external network traffic received by the server 51 through the router is detected. The gateway 511 takes the number of traffic detected by the abnormality as the to-be-distributed traffic data, and distributes it to the associated service node 1, service node 2 and service node 3 according to the traffic type. The service node 1, service node 2 and service node 3 are respectively provided with a proxy to receive the traffic data from the gateway, and input the received traffic data into the network traffic detection device provided in each service node. The network traffic detection device detects the received traffic data, that is, different service nodes 1, service node 2 and service node 3 can independently detect network traffic anomalies. When malicious traffic is detected in the traffic data, an alarm information is generated according to the malicious traffic, and the alarm information is fed back to the control device 52, and the control device 52 interrupts the process or reminds the technical personnel to take timely response measures according to the alarm information.

[0116] The network traffic detection system provided by the embodiment can not only detect external traffic data for abnormality, but also detect the traffic data of each service node for abnormality, and simultaneously realize independent detection of external traffic and internal traffic corresponding to each service node, without the need for detection by other means, saving the detection time of malicious traffic, improving the detection efficiency of malicious traffic, avoiding the problem of large transmission performance loss caused by subsequent transmission of traffic data after completing abnormal detection, and ensuring the transmission performance of traffic data.

[0117] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and variations can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method of detecting network traffic, characterized by, The method is applied to a server, wherein a gateway and service nodes corresponding to a plurality of microservices are arranged in the server, different microservices correspond to different service nodes, and the method comprises the following steps of: The gateway obtains to-be-detected network traffic corresponding to a specified port on the server; The gateway performs classification processing on the to-be-detected traffic data according to a preset rule to obtain traffic data corresponding to each traffic type; the preset rule is a classification rule set for the traffic type; The gateway performs anomaly detection on the traffic data of each traffic type to obtain to-be-distributed traffic data, including: the gateway extracts first traffic features corresponding to the traffic data of each traffic type; the first traffic features are input into a traffic anomaly detection model to obtain a traffic detection result; and the to-be-distributed traffic data is determined based on the traffic detection result; wherein the traffic anomaly detection model is obtained by training a preset classification model based on abnormal traffic feature samples; and the first traffic features include flow metadata and secure transmission layer protocol metadata; The gateway distributes the to-be-distributed traffic data to a service node associated with a corresponding traffic type, and different microservices correspond to different traffic types; The service node performs anomaly detection on the traffic data received from the gateway when the microservice of the service node is started.

2. The method of claim 1, wherein, The gateway extracts the first traffic features corresponding to the traffic data of each traffic type, including: The flow data packets corresponding to the traffic data of each traffic type are extracted, and the flow data packets include flow metadata, and the flow metadata includes byte number, packet number, port information, and flow duration; The flow data packets are analyzed based on a preset frequency to determine distribution features corresponding to the flow data packets, and the distribution features are used to represent the distribution state of each item of metadata in the flow data packets; Or, The flow data packets corresponding to the traffic data of each traffic type are extracted, and the flow data packets include flow metadata, and the flow metadata includes byte number, packet number, port information, and flow duration; The flow data packets are analyzed based on a preset frequency to determine distribution features corresponding to the flow data packets, and the distribution features are used to represent the distribution state of each item of metadata in the flow data packets; and The handshake data packets of a secure transmission layer protocol corresponding to the traffic data of each traffic type are extracted; The handshake data packets are parsed to obtain handshake information corresponding to the handshake data packets.

3. The method of claim 1, wherein, The service node performs anomaly detection on the traffic data received from the gateway when the microservice of the service node is started, including: The service node extracts second traffic features corresponding to the traffic data received from the gateway; The second traffic features are input into a traffic anomaly detection model to obtain a traffic detection result; The traffic anomaly detection model is obtained by training a preset classification model based on abnormal traffic feature samples.

4. The method of claim 3, wherein, The service node extracts the second traffic features corresponding to the traffic data received from the gateway, including: The service node extracts the flow data packets corresponding to the traffic data received from the gateway, the flow data packets comprising flow metadata, the flow metadata comprising byte number, packet number, port information, and flow duration; The flow data packets are analyzed based on a preset frequency to determine distribution characteristics corresponding to the flow data packets, the distribution characteristics being used to represent the distribution state of each item of metadata in the flow data packets; Or, The service node extracts the flow data packets corresponding to the traffic data received from the gateway, the flow data packets comprising flow metadata, the flow metadata comprising byte number, packet number, port information, and flow duration; The flow data packets are analyzed based on a preset frequency to determine distribution characteristics corresponding to the flow data packets, the distribution characteristics being used to represent the distribution state of each item of metadata in the flow data packets; and The handshake data packets of the secure transmission layer protocol corresponding to the traffic data received by the service node from the gateway are extracted; The handshake information corresponding to the handshake data packets is obtained by analyzing the handshake data packets.

5. The method of claim 1, wherein, The gateway obtains the network traffic to be detected corresponding to a specified port of a server, comprising: The gateway obtains all traffic data packets received by a specified port on a server; It is judged whether there is an incomplete traffic data packet in all traffic data packets; When there is an incomplete traffic data packet, the incomplete traffic data packet is filtered out to obtain a complete traffic data packet, and the complete traffic data packet is determined as the network traffic to be detected.

6. An apparatus for detecting network traffic, characterized by The device is applied to a server, and the device comprises: A gateway is configured to obtain network traffic to be detected corresponding to a specified port of a server, and perform classification processing on the network traffic to be detected according to a preset rule to obtain traffic data corresponding to each traffic type; the preset rule is a classification rule set for the traffic type; and the traffic data of each traffic type is subjected to anomaly detection to obtain network traffic data to be distributed; and the network traffic data to be distributed is distributed to a service node associated with the corresponding traffic type, different microservices corresponding to different traffic types. The service node is configured to perform anomaly detection on the network traffic data received from the gateway when the microservice of the service node is started. The gateway is further configured to extract first traffic characteristics corresponding to the network traffic data of each traffic type; input the first traffic characteristics into a traffic anomaly detection model to obtain a traffic detection result; and determine the network traffic data to be distributed based on the traffic detection result; wherein the traffic anomaly detection model is obtained by training a preset classification model based on abnormal traffic characteristic samples; and the first traffic characteristics comprise flow metadata and secure transmission layer protocol metadata.

7. The apparatus of claim 6, wherein, The service node is further configured to extract second traffic characteristics corresponding to the network traffic data received from the gateway; The second traffic characteristics are input into the traffic anomaly detection model to obtain a traffic detection result; wherein the traffic anomaly detection model is obtained by training a preset classification model based on abnormal traffic characteristic samples.

8. The apparatus of claim 6, wherein, The gateway is also used to acquire all traffic data packets received by a specified port on a server; judge whether there is an incomplete traffic data packet in all traffic data packets; when there is an incomplete traffic data packet, filter out the incomplete traffic data packet to obtain a complete traffic data packet, and determine the complete traffic data packet as the network traffic to be detected.

Images