Large-flow analysis method and device, storage medium and electronic equipment

Abstract

The invention discloses a mass flow analysis method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining each flow data packet, receiving risk information and sampling information of an analysis platform, carrying out the matching of each flow data packet with the risk information, determining the flow data packet matched with the risk information as key flow, and carrying out the analysis of the key flow. And matching each traffic data packet with the sampling information, determining the traffic data packet matched with the sampling information as edge traffic, and sampling the traffic data packets in the edge traffic to obtain sampling traffic. In the method, before the traffic probe performs risk detection on the traffic data packet, the traffic data packet is divided into the key traffic and the edge traffic, and the data volume of the edge traffic needing security detection is reduced in a sampling manner, so that the data volume of the traffic data packet for risk detection is reduced; and the number of hardware equipment needing to be deployed is correspondingly reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, storage medium and electronic device for analyzing large volumes of traffic. Background Art

[0002] With the popularization of the Internet and the rapid development of various applications, security threats such as network attacks, malware, and data leakage are becoming more and more serious. Monitoring and analyzing traffic data packets can timely identify potential security threats in the network and take corresponding measures in a timely manner.

[0003] Traffic probes are a common network security device used for traffic capture and analysis, and are connected in series to the links where traffic needs to be monitored. The data processing capacity of a single traffic probe is limited, and it is unable to capture and analyze all traffic packets in a timely manner. In large traffic scenarios, a large number of traffic probes are often required to share the load in order to perform real-time and comprehensive detection of traffic packets. However, as the number of traffic probes increases, hardware costs and operation and maintenance costs increase.

[0004] Therefore, the present specification provides a large flow analysis method. Summary of the invention

[0005] The present specification provides a large flow analysis method, device, storage medium and electronic device to at least partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions: This specification provides a large flow analysis method, which is applied to a flow probe, including: Collect each traffic data packet and receive risk information and sampling information sent by the analysis platform; Matching each of the traffic data packets with the risk information, determining the traffic data packets matching the risk information as key traffic, matching each of the traffic data packets with the sampling information, determining the traffic data packets matching the sampling information as edge traffic; Sampling each flow data packet in the edge flow to obtain sampled flow; The flow to be analyzed is determined according to the key flow and the sampled flow, and the log information of the flow to be analyzed is sent to the analysis platform.

[0007] Optionally, the sampling information includes a plurality of safety flow features and a sampling ratio of each safety flow feature; Sampling each flow data packet in the edge flow to obtain sampled flow, specifically including: For each traffic data packet in the edge traffic, determining a security traffic feature corresponding to the traffic data packet; In the sampling information, a sampling ratio of the security traffic feature corresponding to the traffic data packet is determined as a target sampling ratio; According to the target sampling ratio, each traffic data packet in the edge traffic is sampled to obtain sampled traffic.

[0008] Optionally, sending the log information of the traffic to be analyzed to the analysis platform specifically includes: By using a preset risk detection algorithm, the data packet information of each flow data packet contained in the flow to be analyzed is analyzed, and the flow data packets with risks in the flow to be analyzed are determined as risky flow, and the flow data packets without risks in the flow to be analyzed are determined as safe flow; The log information of the risky traffic and the log information of the safe traffic are sent to the analysis platform.

[0009] This specification also provides another large flow analysis method, which is applied to the analysis platform, including: Receive the log information of the traffic to be analyzed sent by the traffic probe; According to the log information of the traffic to be analyzed, feature analysis is performed on the traffic to be analyzed to obtain risk information and sampling information; The risk information and the sampling information are sent to the traffic probe.

[0010] Optionally, the log information of the traffic to be analyzed includes log information of risky traffic and log information of safe traffic; According to the log information of the traffic to be analyzed, feature analysis is performed on the traffic to be analyzed to determine risk information and sampling information, specifically including: Extracting first field values ​​corresponding to preset key fields from the log information of the risky traffic; extracting second field values ​​corresponding to the key fields from the log information of the safe traffic; Performing feature analysis on the first field value corresponding to each key field to obtain each risk flow feature; performing feature analysis on the second field value corresponding to each key field to obtain each safety flow feature; The risk information is determined according to the risk flow characteristics, and the sampling information is determined according to the safety flow characteristics.

[0011] Optionally, the sampling information is determined according to the characteristics of each security flow, specifically including: Determining each pre-specified network resource and the importance of each network resource; For each security traffic feature, when the access target of the traffic data packet with the security traffic feature is any pre-set network resource, the analysis platform determines the sampling ratio of the security traffic feature based on the importance of any pre-set network resource, and the importance is negatively correlated with the sampling ratio.

[0012] This specification provides a large flow analysis device, which is applied to a flow probe, and the device includes: The collection module collects each traffic data packet and receives the risk information and sampling information sent by the analysis platform; A matching module, matching each of the traffic data packets with the risk information, determining the traffic data packets matching the risk information as key traffic, matching the traffic data packets in each of the traffic data packets with the sampling information, determining the traffic data packets matching the sampling information as edge traffic; A sampling module samples each flow data packet in the edge flow to obtain a sampled flow; The transmission module determines the traffic to be analyzed according to the key traffic and the sampled traffic, and sends the log information of the traffic to be analyzed to the analysis platform.

[0013] This specification provides a large flow analysis device, which is applied to an analysis platform, and the device includes: The log receiving module receives the log information of the traffic to be analyzed sent by the traffic probe; An analysis module performs feature analysis on the traffic to be analyzed according to the log information of the traffic to be analyzed to obtain risk information and sampling information; The transmission module sends the risk information and the sampling information to the traffic probe.

[0014] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned large-flow analysis method is implemented.

[0015] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned large flow analysis method when executing the program.

[0016] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects: In the large traffic analysis method provided in this specification, each traffic data packet is obtained, and the risk information and sampling information of the receiving analysis platform are obtained, each traffic data packet is matched with the risk information, and the traffic data packet matching the risk information is determined as the key traffic, each traffic data packet is matched with the sampling information, and the traffic data packet matching the sampling information is determined as the edge traffic, and the traffic data packets in the edge traffic are sampled to obtain the sampled traffic. In this method, before the traffic probe performs risk detection on the traffic data packet, the traffic data packet is first divided into key traffic and edge traffic, and the data volume of the edge traffic that needs to be security detected is reduced by sampling, thereby reducing the data volume of the traffic data packet for risk detection, and correspondingly reducing the number of hardware devices that need to be deployed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings: Figure 1 is a schematic diagram of a large flow analysis system in this specification; Figure 2 A data transmission relationship diagram of a flow probe and an analysis platform provided in an embodiment of this specification; Figure 3 A schematic diagram of a large flow analysis method provided in this specification; Figure 4 A schematic diagram of a large flow analysis method provided in this specification; Figure 5 A schematic diagram of a large flow analysis device provided in this specification; Figure 6 A schematic diagram of a large flow analysis device provided in this specification; Figure 7 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0020] Figure 1 is a schematic diagram of a large flow analysis system in this specification, such as Figure 1 As shown, the system includes a flow probe and an analysis platform. The flow probe is used to collect the flow data packets flowing through the link in real time, identify the risks of each flow data packet, and generate flow logs to report to the analysis platform. The analysis platform performs feature analysis, threat cause tracing and other data analysis based on the flow log. The analysis platform can be a terminal or server with data analysis capabilities.

[0021] The traffic probe is connected in series on the link node where the traffic data packets need to be monitored. The traffic data packets flowing through the link node can be copied to the traffic probe through the traffic mirroring of the switch. The traffic data packets obtained by the traffic probe are all the traffic data packets flowing through the link node.

[0022] With the development of Internet technologies such as big data, the Internet of Things, and artificial intelligence, network resources are becoming more and more abundant, and the amount of data contained in traffic packets in the network is also increasing. In the existing technology, in order to ensure that risk traffic can be accurately identified through traffic analysis, risk detection is generally performed on all traffic packets. In this way, the amount of data that the traffic probe needs to process is extremely large. Because the data processing capacity of a traffic probe is limited, in order to speed up the processing efficiency, it is necessary to deploy a large number of traffic probes.

[0023] In the large-flow analysis method provided in this specification, the flow probe and the analysis platform work together to filter the flow data packets collected by the flow probe through the risk information and sampling information sent by the analysis platform, thereby reducing the amount of flow data packets that the flow probe needs to process, and correspondingly reducing the number of flow probes that need to be deployed in the application scenario of flow analysis, thereby reducing hardware costs and operation and maintenance costs.

[0024] Figure 1 The traffic probe and analysis platform shown in the figure work together to complete the analysis process of network traffic, which is a periodic feedback process. The traffic probe periodically reports the traffic log to the analysis platform, and the analysis platform performs data analysis on the traffic log received in the current cycle to generate risk information and sampling information, which will be used in the risk identification process of the traffic probe in the next cycle.

[0025] The processing flow of this system in each cycle is the same. The following will explain the large flow analysis method in this manual for the flow analysis process of one cycle. The length of one cycle can be set according to the analysis requirements and is not specifically limited in this manual.

[0026] like Figure 1 As shown, the process may include the following steps: S100: Collect each traffic data packet and receive risk information and sampling information sent by the analysis platform.

[0027] The traffic probe obtains all traffic data packets passing through the link nodes where the traffic probe is deployed through the traffic mirroring of the switch.

[0028] At the same time, the traffic probe receives the risk information and sampling information output by the analysis platform of the previous cycle. The risk information includes the risk traffic features obtained by the analysis platform based on the analysis of each traffic data packet collected in the previous cycle. In the current cycle, it can be used to identify traffic data packets that may be at risk from the traffic data packets obtained in the current cycle. The sampling information includes the safety traffic features obtained by the analysis platform based on the analysis of each traffic data packet collected in the previous cycle for traffic data packets that do not pose a risk. It can be used to identify relatively safe traffic data packets from the traffic data packets obtained in the current cycle.

[0029] S102: Match each of the traffic data packets with the risk information, determine the traffic data packets matching the risk information as key traffic, match each of the traffic data packets with the sampling information, determine the traffic data packets matching the sampling information as edge traffic.

[0030] The risk information contains multiple risk traffic features, which are obtained by the analysis platform based on the analysis of each traffic data packet collected in the previous cycle. This specification does not limit the specific form of the risk traffic feature, which can be composed of one or more of the source IP, destination IP, source port, destination port, specific payload content, etc. of the risky traffic. Among them, the specific payload content is the specific content carried in the payload information of the data packet information of the traffic data packet of different network attack methods, and the known network attack methods can be identified according to the specific payload content.

[0031] The traffic probe matches each traffic data packet with each risk traffic feature in the risk information, and determines the traffic data packet that matches the risk information as the key traffic.

[0032] Specifically, the traffic probe can parse the data packet information of each traffic data packet and extract the field value of the preset key field in the data packet information. The preset key field can include the target IP, source IP, target port, source port, protocol type, etc.

[0033] According to the field values ​​of each key field extracted, it is determined whether the traffic data packet matches any risk traffic feature. If so, the traffic data packet is determined to be a key traffic. The target IP, source IP, target port, source port, protocol type, etc. represented in the data packet information of the traffic data packet are the same as the risk traffic feature.

[0034] Key traffic refers to the traffic data packets in the current cycle that match the risk traffic characteristics obtained by the analysis platform in the previous cycle. In the current cycle, these key traffic may also pose security threats, so it is necessary to analyze all traffic data packets in the key traffic to accurately determine whether the key traffic in the current cycle really poses a threat.

[0035] The sampling information contains multiple safe traffic features, which are obtained by the analysis platform based on the traffic data packets collected in the previous cycle and analyzed for traffic data packets without risks. Similar to risk traffic features, safe traffic features can also be composed of one or more of the source IP, destination IP, source port, destination port, etc. of traffic data packets without risks.

[0036] The traffic probe matches each acquired traffic data packet with each risk traffic feature in the risk information, and determines the traffic data packet that matches the sampling information as the edge traffic.

[0037] Edge traffic refers to the traffic data packets in the current cycle that match the security traffic characteristics obtained by the analysis platform in the previous cycle. These edge traffic are less likely to pose a security threat in the current cycle.

[0038] S104: Sampling data packets in the edge traffic to obtain sampled traffic.

[0039] Because the possibility of security threats in edge traffic is small, in order to reduce the amount of data required for risk identification and speed up the analysis rate, the traffic probe needs to reduce the amount of data required for risk identification and speed up the analysis rate. After determining that the traffic data packet is edge traffic, the traffic probe samples the edge traffic to obtain the sampled traffic.

[0040] The specific sampling method is not limited in this specification, and it can be performed according to a preset sampling ratio.

[0041] When a traffic data packet is determined to be edge traffic, the traffic probe determines whether to drop the data packet based on a certain sampling ratio for each traffic data packet. The traffic probe only identifies risks for the retained traffic data packets.

[0042] Through this method, before the traffic probe identifies the risk of the traffic data packet, it pre-screens the traffic data packet according to the risk information and sampling information of the analysis platform, thereby reducing the amount of traffic data packets that need to be further processed.

[0043] S106: Determine the flow to be analyzed according to the key flow and the edge flow obtained after sampling.

[0044] S108: Sending the log information of the traffic to be analyzed to the analysis platform.

[0045] S110: receiving log information of the traffic to be analyzed sent by the traffic probe.

[0046] Because the large flow analysis method in this specification needs to update the risk information and sampling information in real time in each cycle, after the flow probe determines the key flow and the edge flow after sampling, the key flow and the sampled flow continue to be used as the flow to be analyzed.

[0047] The traffic probe sends the log information of the traffic to be analyzed to the analysis platform, which performs feature analysis on the traffic to be analyzed based on the log information of the traffic to be analyzed, and generates new risk features and sampling information for use by the traffic probe in the next cycle of traffic analysis.

[0048] The log information may include the timestamp, target IP, source IP, target port, source port, protocol type, request count, and other information of each traffic data packet in the traffic to be analyzed.

[0049] S112: Perform feature analysis on the traffic to be analyzed according to the log information of the traffic to be analyzed, and determine risk information and sampling information.

[0050] S114: Send the risk information and the sampling information to the traffic probe.

[0051] The analysis platform can store the log information of the traffic to be analyzed, and perform feature analysis on the log information of the traffic to be analyzed, extracting the risk traffic features of the risky traffic and the security traffic features of the security traffic.

[0052] This specification does not limit the specific method of feature analysis. The analysis platform can analyze the log information of the traffic to be analyzed as a whole. It can also distinguish between key traffic and edge traffic in the traffic to be analyzed, and analyze the key traffic and edge traffic separately. Specific feature analysis methods can use clustering, statistics, behavior modeling, machine learning, etc.

[0053] If the log information of the traffic to be analyzed is taken as a whole, the analysis platform needs to perform risk detection on each traffic data packet contained in the traffic to be analyzed based on the log information of the traffic to be analyzed. This specification does not limit the specific method used for risk monitoring, and any existing network threat identification algorithm based on traffic data packets can be used, such as rule-based detection algorithms, machine learning detection algorithms, protocol analysis algorithms, sandboxes, etc.

[0054] The analysis platform obtains the common content in the log information of the risky traffic data packets detected by deduplication and aggregation, such as the request time period indicated by the timestamp, the target IP segment, the source IP segment, the source port number, etc., as the characteristics of each risky traffic.

[0055] For the log information of the traffic data packets detected to be non-risky, the common content in the log information of the traffic data packets without risk is obtained by deduplication and aggregation, such as the request time period indicated by the timestamp, the target IP segment, the source IP segment, the source port number, etc., as the characteristics of each risky traffic.

[0056] The analysis platform uses the risk traffic features determined in the current cycle as risk information and the safety traffic features determined in the current cycle as sampling information. The risk information and sampling information of the current cycle are sent to the traffic probe, so that the traffic probe can distinguish the key traffic and edge traffic in each traffic data packet collected by the traffic probe in the next cycle through the risk information and sampling information of the current cycle in the traffic analysis of the next cycle.

[0057] based on Figure 1 In the large-traffic analysis system shown, before the traffic probe performs risk detection on each traffic data packet, the traffic probe first divides each traffic data packet into key traffic and edge traffic, and reduces the data volume of edge traffic that requires security detection through sampling, thereby reducing the data volume of traffic data packets for risk detection, and correspondingly reducing the number of traffic probes that need to be deployed.

[0058] In one embodiment of the present specification, in the above step S102, the traffic data packets are matched with the risk information and the sampling information respectively. In some cases, there may be traffic data packets in the traffic data packets that do not match the risk information and the sampling information, which are regarded as unmatched traffic.

[0059] Unmatched traffic may have a source IP address or a destination IP address that has never appeared before. The traffic probe cannot classify the unmatched traffic based on the risk information or sampling information sent by the analysis platform. The traffic probe can set a default processing method for risk detection of unmatched traffic. This manual does not specify the specific default processing method, which can be security detection of a specified number of packets of unmatched traffic, or direct discard, etc.

[0060] In one embodiment of the present specification, the sampling information may include multiple security flow features and a sampling ratio corresponding to each security flow feature.

[0061] Then in the above step S104, the traffic probe downsamples the edge traffic according to the sampling information.

[0062] Specifically, the traffic probe can determine the security traffic features corresponding to each traffic data packet in the edge traffic, and determine the sampling ratio of the security traffic features corresponding to the traffic data packet in the sampling information as the target sampling ratio. According to the target sampling ratio, the data packets in the traffic data packet are sampled, that is, the target sampling ratio is used to randomly determine whether the data packets in the traffic data packet need to be retained for subsequent risk identification.

[0063] The traffic probe samples edge traffic. Each traffic packet in the edge traffic does not match the risk information, but the edge traffic in the current cycle may also contain security risks. Edge traffic may contain different types of traffic packets, such as traffic for media signal transmission and traffic for website access. Different types of traffic have different possibilities for security threats.

[0064] This embodiment can sample different types of traffic data packets at different sampling ratios, so that for traffic data packet types with a high possibility of security threats, a smaller sampling ratio is sampled, and more original traffic data packets are retained. For traffic data packet types with a low possibility of security threats, a larger sampling ratio is used. While reducing the amount of analysis data of the traffic probe, different sampling ratios are used for different types of traffic data packets according to the possibility of security threats in different traffic data packets, to prevent data packets with security threats from being mistakenly lost, so as to increase the accuracy of threat traffic identification.

[0065] In one embodiment of this specification, a risk detection algorithm may be built into the flow probe, and the flow data packets with risks in the flow to be analyzed may be further identified through the line detection algorithm to determine the risk flow. This specification does not limit the specific algorithm type of the risk detection algorithm, and any existing network threat identification algorithm based on network flow may be used, such as a rule-based detection algorithm, a machine learning detection algorithm, a protocol analysis algorithm, a sandbox, etc.

[0066] In step S102, the traffic probe matches key traffic in the traffic data packet based on the risk information of the analysis platform. The key traffic is determined based on the traffic data packets that had security risks in the previous cycle, and may have security risks in this cycle.

[0067] However, not all the key traffic in the current cycle may be traffic packets with security risks, and the edge traffic in the current cycle that matches the sampling information may also contain traffic with security risks. The traffic probe uses the built-in risk detection algorithm to accurately conduct risk and security analysis on the traffic to be analyzed, and determine the traffic packets that actually pose security risks in the traffic to be analyzed.

[0068] Because the traffic to be analyzed consists of key traffic and sampled traffic, the amount of traffic data packets that the traffic probe needs to analyze is greatly reduced. Compared with the method of not sampling and performing full risk identification on all acquired traffic data packets, fewer traffic probes are needed to complete large traffic analysis tasks.

[0069] In this embodiment, the flow probe analyzes the data packet information of each flow data packet contained in the flow to be analyzed through a preset risk detection algorithm, and determines the flow data packets with risks in the flow to be analyzed as risky flow. And determines the flow data packets without risks in the flow to be analyzed as safe flow. The flow probe sends the log information of the risky flow and the log information of the safe flow to the analysis platform.

[0070] The risk traffic determined here is different from the key traffic mentioned above. The key traffic is the traffic data packets that may pose a security threat, and the risk traffic is the traffic data packets that are detected by the risk detection algorithm and are determined to pose a security risk.

[0071] Based on this embodiment, in the above step S110, the analysis platform can perform feature analysis on the log information of risk traffic and the log information of security traffic in the log information of the traffic to be analyzed, so as to determine the characteristics of each risk traffic for the log information of risk traffic, and determine the characteristics of each security traffic for the log information of security traffic.

[0072] The analysis platform extracts the first field value corresponding to each preset key field in the log information of the risk flow, performs feature analysis on the first field value corresponding to each key field, and obtains the features of each risk flow.

[0073] Specifically, the analysis platform can obtain the common content in the log information of the risky traffic by deduplication and aggregation for the first field value corresponding to each key field, such as the request time period indicated by the timestamp, the target IP segment, the source IP segment, the source port number, etc., as the characteristics of each risky traffic.

[0074] For example, if the analysis platform determines that the source IP of each flow packet in the risky flow is port 80 and the source IP exists in the ab address segment, the analysis platform can use port 80 and source IP segment ab as a risky flow feature. Then, when the flow probe matches that the source port of a flow packet is port 80 and the source IP belongs to the ab address segment, the flow packet is determined to be a key flow.

[0075] The analysis platform extracts the second field value corresponding to each key field in the log information of the security flow, performs feature analysis on the second field value of each key field, and obtains the features of each security flow.

[0076] Similarly, the analysis platform can obtain common content in the log information of security traffic by deduplication and aggregation for the second field value corresponding to each key field, such as the request time period indicated by the timestamp, the target IP segment, the source IP segment, the source port number, etc., as the characteristics of each security traffic.

[0077] In one embodiment of the present specification, when the sampling information includes a sampling ratio, the analysis platform may also set the sampling ratio according to the importance of the network resources that each traffic data packet needs to access.

[0078] Then in the above step S112, the analysis platform determines the sampling information according to the characteristics of each security flow, and may also include the following steps.

[0079] The analysis platform determines the pre-specified network resources and the importance of each network resource. For each security traffic feature, when the access target of the traffic data packet with the security traffic feature is any pre-set network resource, the analysis platform determines the sampling ratio of the security traffic feature according to the importance of any pre-set network resource.

[0080] Specifically, the analysis platform can pre-set the importance level of various network resources, and different importance levels correspond to a weight coefficient, and different weight coefficients correspond to a sampling ratio. Then the analysis platform can determine a sampling ratio according to the importance of the network resources.

[0081] The importance is negatively correlated with the sampling ratio, that is, the higher the importance of the network asset, the smaller the sampling ratio, which means the lower the degree of downsampling and the more complete the retention of the original network traffic data packets.

[0082] For example, network resources are pre-divided into three levels, 1 to 3, with weight coefficients corresponding to levels 1, 1 to 3 being 0.2, 0.5, and 1, respectively; a sampling ratio corresponding to a weight of 0.2 is 10000:1; a sampling ratio corresponding to a weight of 0.5 is 1000:1; and a sampling ratio corresponding to a weight of 1 is 1:1.

[0083] Then, the analysis platform sends each security traffic feature and the sampling ratio of each security traffic feature as sampling information to the traffic probe, and the traffic probe is used for the traffic analysis process of the next cycle.

[0084] When the sampling ratio is 1:1, it means that no downsampling is performed and all traffic packets are identified for risk. That is, in the next cycle, the traffic probe can perform full detection on traffic packets with the security traffic characteristics, more accurately identify the security risks that may exist in the traffic packets, and prevent important network resources from being threatened.

[0085] Figure 2This is a data transmission relationship diagram of a flow probe and an analysis platform provided in an embodiment of this specification, such as Figure 2 As shown, the risk information includes various risk flow characteristics, and the sampling information includes various safety flow characteristics and the sampling ratio corresponding to each safety flow characteristic.

[0086] After the traffic probe obtains each traffic data packet, it matches each traffic data packet with each risk traffic feature in the risk information to obtain the key traffic, and matches each traffic data packet with each security traffic feature in the sampling information to obtain the edge traffic. Then, according to the sampling ratio in the sampling information, each traffic data packet in the edge traffic is sampled at a different sampling ratio to obtain the sampled traffic, and the key traffic and sampled traffic are used as the traffic to be analyzed. The traffic probe uses the built-in risk detection algorithm to perform risk detection on the traffic to be analyzed, and reclassifies the traffic to be analyzed into risk traffic with security threats and security traffic without security threats, and sends the log information of the risk traffic and the log information of the security traffic to the analysis platform for feature analysis.

[0087] The analysis platform performs feature analysis on the log information of risky traffic to obtain risk information, and performs feature analysis on the log information of safe traffic to obtain sampling information. The risk information and sampling information will be used for traffic analysis of traffic data packets obtained by the traffic probe in the next cycle.

[0088] This manual also provides Figure 1 The large flow analysis method corresponding to the system, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a large flow analysis method provided in this specification, which is applied to a flow probe.

[0089] S200: Collect each traffic data packet and receive risk information and sampling information sent by the analysis platform.

[0090] S202: Match each of the traffic data packets with the risk information, determine the traffic data packets that match the risk information as key traffic, match each of the traffic data packets with the sampling information, and determine the traffic data packets that match the sampling information as edge traffic.

[0091] S204: Sampling each traffic data packet in the edge traffic to obtain sampled traffic.

[0092] S206: Determine the traffic to be analyzed according to the key traffic and the sampled traffic, and send log information of the traffic to be analyzed to the analysis platform.

[0093] Figure 3 The specific contents of the corresponding steps S200 to S206 can refer to the above Figure 1The corresponding contents in the manual will not be elaborated in this manual.

[0094] This manual also provides Figure 1 The large flow analysis method corresponding to the system, such as Figure 4 As shown, Figure 4 A schematic diagram of a large flow analysis method provided in this specification, which is applied to an analysis platform.

[0095] S300: receiving log information of the traffic to be analyzed sent by the traffic probe.

[0096] S302: Perform feature analysis on the traffic to be analyzed according to the log information of the traffic to be analyzed to obtain risk information and sampling information.

[0097] S304: Send the risk information and the sampling information to the traffic probe.

[0098] Figure 4 The specific contents of the corresponding steps S300 to S304 can refer to the above Figure 1 The corresponding contents in the manual will not be elaborated in this manual.

[0099] This specification also provides a large flow analysis device, which is applied to flow probes, such as Figure 5 As shown, Figure 5 This is a schematic diagram of a large flow analysis device provided in this specification, which is applied to a flow probe, and includes: The collection module 400 is used to collect each flow data packet and receive the risk information and sampling information sent by the analysis platform; A matching module 402 is used to match each of the traffic data packets with the risk information, determine the traffic data packets matching the risk information as key traffic, match the traffic data packets in each of the traffic data packets with the sampling information, and determine the traffic data packets matching the sampling information as edge traffic; The sampling module 404 is used to sample each flow data packet in the edge flow to obtain a sampled flow; The transmission module 406 is used to determine the traffic to be analyzed according to the key traffic and the sampled traffic, and send the log information of the traffic to be analyzed to the analysis platform.

[0100] Optionally, the sampling information includes multiple security traffic features and a sampling ratio of each security traffic feature. The sampling module 404 is specifically used to determine, for each traffic data packet in the edge traffic, the security traffic feature corresponding to the traffic data packet. In the sampling information, the sampling ratio of the security traffic feature corresponding to the traffic data packet is determined as the target sampling ratio. According to the target sampling ratio, each traffic data packet in the edge traffic is sampled to obtain the sampled traffic.

[0101] Optionally, the transmission module 406 is specifically used to analyze the data packet information of each traffic data packet contained in the traffic to be analyzed through a preset risk detection algorithm, determine the traffic data packets with risks in the traffic to be analyzed as risky traffic, determine the traffic data packets without risks in the traffic to be analyzed as safe traffic, and send the log information of the risky traffic and the log information of the safe traffic to the analysis platform.

[0102] This specification also provides a large flow analysis device, which is applied to an analysis platform, such as Figure 6 As shown, Figure 6 This is a schematic diagram of a large flow analysis device provided in this specification, which is applied to an analysis platform and includes: The log receiving module 500 is used to receive the log information of the traffic to be analyzed sent by the traffic probe; The analysis module 502 is used to perform feature analysis on the traffic to be analyzed according to the log information of the traffic to be analyzed, and obtain risk information and sampling information; The transmission module 504 is used to send the risk information and the sampling information to the traffic probe.

[0103] Optionally, the log information of the traffic to be analyzed includes log information of risk traffic and log information of safe traffic, and the analysis module 502 is specifically used to extract the first field value corresponding to each preset key field in the log information of the risk traffic, extract the second field value corresponding to the key field in the log information of the safe traffic, perform feature analysis on the first field value corresponding to each key field to obtain each risk traffic feature, perform feature analysis on the second field value corresponding to each key field to obtain each safe traffic feature, determine the risk information based on the each risk traffic feature, and determine the sampling information based on the each safe traffic feature.

[0104] Optionally, the analysis module 502 is specifically used to determine each pre-specified network resource and the importance of each network resource. For each security traffic feature, when the access target of the traffic data packet with the security traffic feature is any pre-set network resource, the analysis platform determines the sampling ratio of the security traffic feature based on the importance of any pre-set network resource, and the importance is negatively correlated with the sampling ratio.

[0105] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provides a large flow analysis method.

[0106] This manual also provides Figure 7 The schematic structure diagram of the electronic device shown in FIG. Figure 7 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0107] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0108] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0109] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0110] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0111] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0112] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0115] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0116] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0117] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0118] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0119] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0121] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0122] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.

Claims

1. A method for analyzing large flows, applied to a flow probe, characterized in that: include: Collect each traffic data packet and receive risk information and sampling information sent by the analysis platform; Matching each of the traffic data packets with the risk information, determining the traffic data packets matching the risk information as key traffic, matching each of the traffic data packets with the sampling information, determining the traffic data packets matching the sampling information as edge traffic; Sampling each flow data packet in the edge flow to obtain sampled flow; The flow to be analyzed is determined according to the key flow and the sampled flow, and the log information of the flow to be analyzed is sent to the analysis platform.

2. The method according to claim 1, characterized in that The sampling information includes a plurality of safety flow characteristics and a sampling ratio of each safety flow characteristic; Sampling each flow data packet in the edge flow to obtain sampled flow, specifically including: For each traffic data packet in the edge traffic, determining a security traffic feature corresponding to the traffic data packet; In the sampling information, a sampling ratio of the security traffic feature corresponding to the traffic data packet is determined as a target sampling ratio; According to the target sampling ratio, each traffic data packet in the edge traffic is sampled to obtain sampled traffic.

3. The method according to claim 1, characterized in that Sending the log information of the traffic to be analyzed to the analysis platform specifically includes: By using a preset risk detection algorithm, the data packet information of each flow data packet contained in the flow to be analyzed is analyzed, and the flow data packets with risks in the flow to be analyzed are determined as risky flow, and the flow data packets without risks in the flow to be analyzed are determined as safe flow; The log information of the risky traffic and the log information of the safe traffic are sent to the analysis platform.

4. A large flow analysis method, applied to an analysis platform, characterized in that: include: Receive the log information of the traffic to be analyzed sent by the traffic probe; According to the log information of the traffic to be analyzed, feature analysis is performed on the traffic to be analyzed to obtain risk information and sampling information; The risk information and the sampling information are sent to the traffic probe.

5. The method according to claim 4, characterized in that The log information of the traffic to be analyzed includes log information of risky traffic and log information of safe traffic; According to the log information of the traffic to be analyzed, feature analysis is performed on the traffic to be analyzed to determine risk information and sampling information, specifically including: Extracting first field values ​​corresponding to preset key fields from the log information of the risky traffic; extracting second field values ​​corresponding to the key fields from the log information of the safe traffic; Performing feature analysis on the first field value corresponding to each key field to obtain each risk flow feature; performing feature analysis on the second field value corresponding to each key field to obtain each safety flow feature; The risk information is determined according to the risk flow characteristics, and the sampling information is determined according to the safety flow characteristics.

6. The method according to claim 5, characterized in that According to the characteristics of each security flow, sampling information is determined, specifically including: Determining each pre-specified network resource and the importance of each network resource; For each security traffic feature, when the access target of the traffic data packet with this security traffic feature is any pre-set network resource, the analysis platform determines the sampling ratio of the security traffic feature based on the importance of any pre-set network resource, and the importance is negatively correlated with the sampling ratio.

7. A large flow analysis device, applied to a flow probe, characterized in that: include: The collection module collects each traffic data packet and receives the risk information and sampling information sent by the analysis platform; A matching module, matching each of the traffic data packets with the risk information, determining the traffic data packets matching the risk information as key traffic, matching the traffic data packets in each of the traffic data packets with the sampling information, determining the traffic data packets matching the sampling information as edge traffic; A sampling module samples each flow data packet in the edge flow to obtain a sampled flow; The transmission module determines the traffic to be analyzed according to the key traffic and the sampled traffic, and sends the log information of the traffic to be analyzed to the analysis platform.

8. A large flow analysis device, applied to an analysis platform, characterized in that: include: The log receiving module receives the log information of the traffic to be analyzed sent by the traffic probe; An analysis module performs feature analysis on the traffic to be analyzed according to the log information of the traffic to be analyzed to obtain risk information and sampling information; The transmission module sends the risk information and the sampling information to the traffic probe.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 6 is implemented.

Images