Network asset processing method, system, device and medium based on traffic analysis

Abstract

The present invention relates to a network asset processing method, system, device and medium based on traffic analysis, the method comprising: collecting network traffic to be analyzed, filtering the collected network traffic; performing stream session management on the filtered network traffic to obtain a network stream session; identifying the application layer protocol from the network stream session through built-in protocol rules or custom protocol rules, and parsing the corresponding field features according to the identified application layer protocol type; when the corresponding application layer protocol cannot be identified from the network stream session based on the built-in protocol rules or custom protocol rules, identifying the application layer protocol through a pre-trained supervised machine learning algorithm model; based on the identified application layer protocol and field features, performing asset identification processing of a specified network segment, and generating a corresponding asset log, so as to perform corresponding network asset management for the asset log. The present invention can effectively collect and safely manage network assets of an electric energy system.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a network asset processing method, system, electronic device and computer-readable storage medium based on traffic analysis. Background Art

[0002] With the rapid development of computer network technology, the network has been deeply integrated into all aspects of social life. At present, the construction of the power energy Internet is in full swing, and the number and types of network assets are increasing. Key infrastructure such as energy and electricity are the top priorities of network security. In order to effectively defend against power system network attacks and ensure the security of critical information infrastructure, how to effectively collect and manage network assets is an important issue facing the current power energy industry.

[0003] However, there are the following problems in the prior art:

[0004] The existing network asset detection methods generally adopt active detection scanning methods, such as obtaining asset feedback information by sending preset data packets to the device where the target network is located. This method first requires the target network to be identified, and then the target network is detected by constructing a data packet or directly using a scanning tool. However, for the power energy system, firstly, its network space assets are widely distributed, the network address space is huge, and it is difficult to select and confirm the target network; secondly, it attaches great importance to network security, and the active detection method is not suitable.

[0005] Some existing technologies use highly complex artificial intelligence algorithm models (such as neural network models), which have high requirements for hardware resources, relatively high power consumption, and increased costs. For power energy companies that have always adhered to the "dual carbon" goal, these models may not be suitable. Summary of the invention

[0006] The purpose of the present invention is to provide a network asset processing method, system, electronic device and computer-readable storage medium based on traffic analysis, aiming to achieve effective collection and safe management of network assets in electric power energy systems.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a network asset processing method based on traffic analysis, comprising:

[0008] Collect network traffic to be analyzed and filter the collected network traffic;

[0009] Performing stream session management on the filtered network traffic to obtain a network stream session;

[0010] Identify the corresponding application layer protocol from the network stream session by using built-in protocol rules or user-defined protocol rules, and parse the field features of the corresponding protocol according to the identified application layer protocol type; wherein, when the corresponding application layer protocol cannot be identified from the network stream session based on the built-in protocol rules or user-defined protocol rules, identify the application layer protocol by using a pre-trained supervised machine learning algorithm model;

[0011] Based on the identified application layer protocol and field characteristics, asset identification processing is performed on the specified network segment, and a corresponding asset log is generated to perform corresponding network asset management for the asset log.

[0012] Preferably, after the step of performing asset identification processing of the designated network segment based on the identified application layer protocol and field characteristics and generating the corresponding asset log, the method further includes:

[0013] The anomaly detection algorithm model based on the unsupervised mode performs anomaly detection on the asset log field, and reports the abnormal log when an anomaly is found.

[0014] Preferably, the identifying the corresponding application layer protocol from the network stream session by using built-in protocol rules or custom protocol rules, and parsing the field characteristics of the corresponding protocol according to the identified application layer protocol type, specifically includes:

[0015] Identify the application layer protocol in the network stream session by matching based on a single field, matching based on a logical combination of multiple field values, or matching based on a feature string, and identify and parse field features related to the application layer protocol from the network stream session according to a preset plurality of types of protocol parsing plug-ins;

[0016] When the corresponding application layer protocol cannot be identified from the network flow session based on the built-in protocol rule or the custom protocol rule, identifying the application layer protocol through a pre-trained supervised machine learning algorithm model specifically includes:

[0017] Based on the traffic behavior characteristics corresponding to the network flow session, the application layer protocol is identified by a pre-trained supervised machine learning algorithm model; wherein the training process of the supervised machine learning algorithm model is as follows:

[0018] Collect a specified number of application layer protocol traffic samples of various types;

[0019] For each sample, parse and extract basic features of the streaming session, wherein the basic features of the streaming session include: transmission protocol type, streaming start time, streaming end time, packet length (byte count) sequence, packet arrival time sequence and / or TTL sequence;

[0020] For each sample, based on the basic characteristics of the streaming session, statistical methods are used to complete feature calculation and storage, and hundreds of dimensions of calculation features corresponding to each sample are obtained;

[0021] The LightGBM model is used to train the hundreds of dimensional computing features of each sample. After the model training is completed, the supervised machine learning algorithm model is obtained;

[0022] The identification of the application layer protocol by using a pre-trained supervised machine learning algorithm model specifically includes:

[0023] For the network flow session, hundreds of dimensions of computing features that are the same as those in model training are extracted, and the trained supervised machine learning algorithm model is used to predict and classify the hundreds of dimensions of computing features corresponding to the network flow session to identify the application layer protocol corresponding to the network flow session.

[0024] Preferably, the step of performing asset identification processing on a specified network segment based on the identified application layer protocol and field features and generating a corresponding asset log includes:

[0025] For the identified specified application layer protocol, a new built-in rule for identifying the specified server type asset in the intranet is created. When a specified application layer protocol stream session matches the built-in rule, the resolved server asset features and / or client basic resolution features are merged to finally form an asset log; wherein the specified server type asset includes HTTP server asset, DNS server asset or FTP server asset;

[0026] Based on the field characteristics corresponding to different application layer protocols, the log format of the corresponding protocol type is designed, and the log encapsulation implementation process is integrated into the corresponding application layer protocol parsing plug-in to report the asset log to the designated module for display.

[0027] Preferably, the anomaly detection algorithm model based on the unsupervised mode performs anomaly detection on the asset log field, and when an anomaly is found, reports the anomaly log, including:

[0028] By analyzing the asset log fields and performing statistics, the statistical feature data of each type of asset is portrayed in the form of a portrait, wherein the statistical feature data includes: the number of assets, the total frequency of asset access, the number of source access objects, the time information of being accessed by each source object or the number of times being accessed by each source object;

[0029] Based on the statistical feature data of the assets, extract the relevant access logs of the assets within the specified time, and parse and store the specified access feature information in the logs;

[0030] Converting the specified access feature information into an access feature information sequence;

[0031] Using probability distribution statistical analysis algorithms, learn and extract normal behavior pattern libraries from a given observation data sequence;

[0032] Based on the normal behavior pattern library, anomaly detection is performed on the asset log field, and when an anomaly is found, the abnormal log is reported.

[0033] Preferably, the step of collecting the network traffic to be analyzed and filtering the collected network traffic includes:

[0034] Collect network traffic to be analyzed through traffic mirroring or end node deployment;

[0035] Filter collected network traffic using custom configured rules based on IP address, IP address range, port and / or protocol type.

[0036] Preferably, the step of performing stream session management on the filtered network traffic to obtain a network stream session includes:

[0037] A flow session management operation including flow establishment, flow update and / or flow release is performed on the filtered network traffic to obtain a corresponding network flow session.

[0038] In a second aspect, the present invention provides a network asset processing system based on traffic analysis, the network asset processing system comprising:

[0039] A collection module is used to collect network traffic to be analyzed and filter the collected network traffic;

[0040] A management module, used for performing flow session management on the filtered network traffic to obtain a network flow session;

[0041] An identification module, used to identify the corresponding application layer protocol from the network flow session through built-in protocol rules or custom protocol rules, and parse the field characteristics of the corresponding protocol according to the identified application layer protocol type; wherein, when the corresponding application layer protocol cannot be identified from the network flow session based on the built-in protocol rules or custom protocol rules, the application layer protocol is identified through a pre-trained supervised machine learning algorithm model;

[0042] The generation module is used to perform asset identification processing of the specified network segment based on the identified application layer protocol and field characteristics, and generate corresponding asset logs to perform corresponding network asset management for the asset logs.

[0043] In a third aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the computer program is executed by the processor, the steps of the network asset processing method based on traffic analysis as described above are implemented.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a network asset processing system based on traffic analysis. The network asset processing system based on traffic analysis can be executed by at least one processor so that the at least one processor performs the steps of the network asset processing method based on traffic analysis as described above.

[0045] The beneficial effects of adopting the above embodiment are:

[0046] This embodiment performs stream session management on the network traffic to be analyzed to obtain a network stream session; identifies the corresponding application layer protocol from the network stream session through built-in protocol rules or custom protocol rules, and when the corresponding application layer protocol cannot be identified based on the built-in protocol rules or custom protocol rules, identifies the application layer protocol through a pre-trained supervised machine learning algorithm model to parse the field characteristics of the corresponding protocol according to the identified application layer protocol type; finally, based on the identified application layer protocol and field characteristics, performs asset identification processing on the specified network segment, and generates a corresponding asset log to perform corresponding network asset management for the asset log. This embodiment adopts a passive traffic analysis and identification method to achieve effective collection and security management of network assets in the power energy system. In addition, considering the problem of misidentification or missed identification in the rules, this embodiment adopts machine learning algorithm technology to assist in solving the problem of misidentification or missed identification, which is more conducive to the security analysis and management of network assets. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A method flow chart of an embodiment of a network asset processing method based on traffic analysis provided by the present application;

[0048] Figure 2 A block diagram of a network asset collection and management system based on traffic analysis in an embodiment of a network asset processing method based on traffic analysis provided by the present application;

[0049] Figure 3 A schematic diagram of a protocol parsing plug-in library in an embodiment of a network asset processing method based on traffic analysis provided by the present application;

[0050] Figure 4 A protocol identification framework diagram based on a supervised AI algorithm model in an embodiment of a network asset processing method based on traffic analysis provided by the present application;

[0051] Figure 5 This is a functional framework diagram of asset management in an embodiment of a network asset processing method based on traffic analysis provided by the present application;

[0052] Figure 6 A schematic diagram of the working of an anomaly detection algorithm model based on unsupervised AI in an embodiment of a network asset processing method based on traffic analysis provided by the present application;

[0053] Figure 7 A flow chart of a normal behavior pattern extraction algorithm based on probability distribution statistical analysis in an embodiment of a network asset processing method based on traffic analysis provided by the present application;

[0054] Figure 8 A principle block diagram of an embodiment of a network asset processing system based on traffic analysis provided by the present application;

[0055] Fig. 9 A functional block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0057] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0058] The inventors of the present invention have studied and found that network asset identification methods are divided into active detection and identification and passive traffic analysis and identification. The active detection and identification method is to send a constructed data packet to a known target network and extract assets from the returned data packet information (such as: the content of each layer of the protocol, packet retransmission time, etc.); while the passive traffic analysis and identification method is to collect network traffic and identify asset-related data through protocol identification and parsing and rule matching methods. Compared with the active detection and identification method, the passive traffic analysis method only analyzes the existing network traffic, will not cause network noise due to the active sending of data packets, and will not affect the network. Therefore, the passive traffic analysis and identification method used in the present invention can achieve effective collection and safe management of network assets in the electric energy system.

[0059] Furthermore, the present application provides a network asset processing method, system, electronic device and storage medium based on traffic analysis, which are described in detail below.

[0060] Reference Figure 1 A method flow chart of an embodiment of a network asset processing method based on traffic analysis provided by the present application, the method comprising the following steps:

[0061] S1. Collect the network traffic to be analyzed and filter the collected network traffic;

[0062] S2, performing stream session management on the filtered network traffic to obtain a network stream session;

[0063] S3, identifying the corresponding application layer protocol from the network flow session through built-in protocol rules or custom protocol rules, and parsing the field characteristics of the corresponding protocol according to the identified application layer protocol type; wherein, when the corresponding application layer protocol cannot be identified from the network flow session based on the built-in protocol rules or custom protocol rules, identifying the application layer protocol through a pre-trained supervised machine learning algorithm model;

[0064] S4. Based on the identified application layer protocol and field characteristics, perform asset identification processing on the specified network segment and generate a corresponding asset log to perform corresponding network asset management for the asset log.

[0065] This embodiment performs stream session management on the network traffic to be analyzed to obtain a network stream session; identifies the corresponding application layer protocol from the network stream session through built-in protocol rules or custom protocol rules, and when the corresponding application layer protocol cannot be identified based on the built-in protocol rules or custom protocol rules, identifies the application layer protocol through a pre-trained supervised machine learning algorithm model to parse the field characteristics of the corresponding protocol according to the identified application layer protocol type; finally, based on the identified application layer protocol and field characteristics, performs asset identification processing on the specified network segment, and generates a corresponding asset log to perform corresponding network asset management for the asset log. This embodiment adopts a passive traffic analysis and identification method to achieve effective collection and security management of network assets in the power energy system. In addition, considering the problem of misidentification or missed identification in the rules, this embodiment adopts machine learning algorithm technology to assist in solving the problem of misidentification or missed identification, which is more conducive to the security analysis and management of network assets.

[0066] Specifically, in some embodiments, a network asset collection and management method based on traffic analysis is proposed. Through passive traffic analysis means, rules and AI (Artificial Intelligence) technical methods are combined to solve the problem of effective collection and security management of network assets in the power energy industry. Figure 2 As shown, specifically including:

[0067] The step of traffic collection is to collect the network traffic to be analyzed by traffic mirroring or end node deployment. Among them, traffic mirroring uses technologies such as SPAN (Switched Port Analyzer) to copy a message passing through a specified source (such as a port) to a specified destination (such as a port) of another connected device (such as a network monitoring device) to achieve traffic collection. End node collection is to achieve traffic collection by deploying nodes (such as client software) on the end-side network device. Compared with the end node deployment method, the traffic mirroring method has the characteristics of easy operation and high efficiency, so this embodiment preferably adopts the traffic mirroring method.

[0068] The traffic filtering step is to configure and use custom rules as needed to filter and analyze the collected network traffic. Through custom rule configuration, the traffic screening and filtering function is completed based on IP address or IP address segment, port, protocol type, etc., so as to improve the traffic processing performance. For example, if there is no need to process the source IP (10.12.1.11), then the IP is newly configured in the IP filter list library. When the source IP obtained during traffic analysis is in the filter list library, the traffic will be filtered without subsequent analysis. Similarly, you can create a new IP address segment filter list library, port filter list library, protocol type filter list library, etc.

[0069] The steps of traffic management include stream session management for the network traffic after on-demand filtering, including: stream establishment, stream update and stream release operations. Stream establishment refers to the completion of preliminary protocol parsing of network traffic. The protocol layers to be parsed mainly include the data link layer, network layer and transport layer in the OSI (Open System Interconnection) reference model. The establishment of the stream session is completed through the five-tuple information (source IP, destination port, protocol type, destination IP and destination port) extracted by parsing. Different five-tuples correspond to different streams. Among them, the protocol type includes two transport layer protocols: TCP (Transmission Control Protocol) and UDP (User Datagram Protocol). At the same time, the basic session parsing features of the stream are stored, mainly including: five-tuple, source MAC (Media Access Control) and destination MAC addresses, stream start time, stream end time, packet length (byte count) sequence, packet arrival time sequence, TTL (Time To Live) sequence, etc. Flow update is to match a newly input data packet and complete the update of the basic characteristic data of the corresponding flow session, such as: updating and replacing the flow end time, adding storage packet length, packet arrival time and TTL information to the corresponding sequence, etc. Flow release means that when the flow session times out or the communication between the source and destination ends normally, it is necessary to release the flow session, such as: clearing the memory space of the flow characteristic data, etc.

[0070] The steps of traffic analysis include protocol identification and parsing. Based on the principle analysis of the application layer protocol of the OSI reference model, and considering the large number of application layer protocols, this system uses a plug-in protocol parsing method to complete the protocol decoding operation and extract the corresponding feature information on demand for the sake of scalability. It should be noted that: in different operating system environments, the plug-in format may be different, such as .dll and .lib library files under Windows, .so and .a library files under Linux, etc., which can be selected as needed.

[0071] like Figure 3 The figure shows the adopted protocol plug-in library. For example, based on the analysis of the application layer HTTP (Hypertext Transfer Protocol), an HTTP protocol parsing plug-in is designed on demand to identify the HTTP protocol from the network flow session and parse the field features related to the protocol.

[0072] The first method of protocol identification is to use built-in or custom rules for protocol identification. Built-in rules are fixed in the plug-in library and cannot be edited or modified, while custom rules are flexible and configurable. For example, a configurable file is provided to complete operations such as creating, editing and deleting custom rules. For the HTTP protocol, it uses the TCP protocol for transmission. The standard port of HTTP service is generally 80, and the request methods include GET, POST, PUT, HEAD, etc. The request method is in the first few bytes of the TCP Payload header. Therefore, the built-in rules for HTTP protocol identification that can be designed may include: Rule 1 is described as "the source or destination port is 80", and Rule 2 is described as "the flow direction is the client (Client) to the server (Server) request direction (abbreviated as: C->S) and the first few bytes of the TCP payload header contain any one of the string set {GET, POST, PUT, DELETE, HEAD, etc.}". When the source or destination port of a flow session hits Rule 1 or the payload request method matches and hits Rule 2, then the flow represents the HTTP protocol. Considering that some HTTP services do not use the standard port 80 (for example, the known destination port is 12345), the above built-in rule 1 may have a problem of missed identification. Therefore, a custom rule configuration method is provided, such as designing a custom port rule as "destination port is 12345" to further identify the HTTP protocol.

[0073] Secondly, considering that rules are matched based on a single field, a logical combination of multiple field values, or a feature string, and when the number of rules increases, the rule matching operation can easily become a performance bottleneck. Therefore, after comparative experimental analysis, it is recommended to use the high-performance feature matching library Hyperscan.

[0074] Finally, according to the identified application layer protocol, the field characteristics of the corresponding protocol are parsed, such as: HTTP protocol request method, protocol version, user agent (User-Agent), request URI (Uniform Resource Identifier), request parameters, request body and response body, etc. Similarly, other known types of protocol parsing plug-ins can be designed respectively, mainly including: DNS (Domain Name System) protocol parsing plug-in, FTP (File Transfer Protocol) protocol parsing plug-in, SMTP (Simple Mail Transfer Protocol) protocol parsing plug-in, POP3 (Post Office Protocol 3, Post Office Protocol Version 3) protocol parsing plug-in, IMAP (Internet Message Access Protocol, Interactive Mail Access Protocol) protocol parsing plug-in, Telnet protocol parsing plug-in, database (such as: MySQL, Oracle, MongoDB, PostgreSQL, etc.) protocol parsing plug-in and other types of protocol parsing plug-ins.

[0075] Furthermore, when the application layer protocol cannot be effectively identified based on built-in rules and custom rules, this system additionally provides a protocol identification framework based on AI technology to assist in solving the problem of missed identification of rules, such as Figure 4 As shown. Considering that different application layer protocols have different traffic behavior characteristics, the final feature calculation can be completed based on the basic features of the stream session extracted in the above process, and the identification (or prediction, classification) process of unknown application layer protocols can be completed through the trained supervised ML (Machine Learning) algorithm model.

[0076] The model training steps are described as follows:

[0077] 1. Sample collection: Capture a large number of known application layer protocol traffic samples of various types.

[0078] 2. Extraction of basic features of streaming sessions: For each sample, the analysis method of basic features of streaming sessions in the traffic management stage is used to complete feature extraction, including: transmission protocol type, stream start time, stream end time, packet length (byte count) sequence, packet arrival time sequence, TTL sequence and other basic features.

[0079] 3. Feature calculation: For each sample, based on the basic features of the flow session, the final feature calculation and storage are completed using statistical methods for the model training step. The main statistical feature values ​​and their descriptions or meanings are shown in Table 1. The calculated features include: transmission protocol type (using binary form 0 and 1 to represent TCP and UDP respectively), flow duration (flow end time minus flow start time), packet length sequence statistical feature values ​​(maximum, minimum, average, variance, standard deviation, median, mode, coefficient of variation, skewness, kurtosis, information entropy and transition probability matrix), packet arrival time interval sequence statistical feature values ​​(maximum, minimum, average, variance, standard deviation, median, mode, coefficient of variation, skewness, kurtosis and transition probability matrix) and TTL sequence statistical feature values ​​(maximum, minimum, average, variance, standard deviation, median, mode, coefficient of variation, skewness, kurtosis). Among them: the packet arrival time interval sequence is composed of the next value in the packet arrival time sequence minus its adjacent previous value.

[0080] Table 1

[0081] Feature Name Characteristic description or meaning Maximum The maximum value in a series of data Minimum The minimum value in a series of data average value The arithmetic mean of a series of data variance An indicator of the average dispersion of all observed values ​​of a random variable (data series) from the overall mean Standard Deviation Arithmetic mean root of variance median In a set of data sequences, arranged in ascending order, the number in the middle position Majority The value that appears most frequently in a set of data sequences Coefficient of variation Ratio of standard deviation to mean Skewness Used to measure the asymmetry of the probability distribution of a random variable (data sequence) Kurtosis Used to measure the steepness of the probability distribution of a random variable (data sequence) Information Entropy Used to measure the expected amount of information brought by all possible occurrence times (values) of a random variable (data sequence) Transition probability matrix The probability of each value in a data sequence transferring to its next adjacent value (matrix)

[0082] Considering that the maximum length of the link layer data frame is 1500 bytes, that is, the MTU (Maximum Transmission Unit) value, the packet length sequence value range is 1~1500, so the dimension of the packet length transition probability matrix will reach 1500×1500, which is 2250000 when converted into a one-dimensional vector. High-dimensional data will make the model complex, the amount of calculation is large, and it is easy to cause overfitting and other problems. Generally speaking, a dimensionality reduction algorithm can be used to reduce the feature dimension of the data, such as the PCA (Principal Component Analysis) algorithm. However, in order to more effectively reduce resource usage and improve performance, this system refers to the histogram statistical idea, firstly divides the maximum packet length (such as: 1500) into 15 adjacent interval cells according to the preset packet length interval value (such as: 100) (such as: using serial numbers 0, 1, ..., 14 to represent different cells), and then converts the packet length sequence data to different cell numbers, such as: the packet length range value (1~99) corresponds to cell 0, the packet length range value (100~199) corresponds to cell 1, ..., the packet length range value (1400~1500) corresponds to cell 14. Therefore, through this method, the packet length transition probability matrix can be finally obtained, which contains 225 (15×15) dimensional data features, greatly reducing the feature dimension and improving the algorithm performance.

[0083] Considering that the time interval between two adjacent packets is generally very small, such as the order of ms (millisecond), referring to the packet length transition probability matrix calculation method, the maximum value of the packet arrival time interval is first limited to 1000ms (i.e., if the packet arrival time interval value is greater than 1000, it is calculated according to the maximum value), and it can be divided into 10 adjacent interval cells (with serial numbers 0, 1, ..., 9 to represent different cells) according to the preset time interval value (100ms), and then the packet arrival time interval sequence data is converted to different cell numbers, such as: the packet arrival time interval range value (0~99) corresponds to cell 0, the packet arrival time interval range value (100~199) corresponds to cell 1, ... the packet arrival time interval range value (greater than or equal to 900) corresponds to cell 9. Therefore, through this method, the packet arrival time interval transition probability matrix can be finally obtained, which contains 100 (10×10) dimensional data features. That is, for each sample, based on the basic characteristics of the stream session, the feature calculation and storage are completed using statistical methods, and hundreds of dimensional calculation features corresponding to each sample are obtained.

[0084] 4. Model training: Through the above steps, hundreds of dimensional computing features corresponding to each sample have been obtained. At this time, a suitable AI algorithm model is used to train it. Considering the large amount of sample data, in order to improve the efficiency of algorithm execution, after comprehensive experimental analysis, it is recommended to use the LightGBM model, which has the characteristics of fast training speed and low memory usage. When the model training is completed, the generated model is saved for subsequent prediction process.

[0085] The model prediction steps are described as follows:

[0086] 1. Load model: Load the offline model saved during the training phase.

[0087] 2. Prediction and classification: Refer to the feature extraction and calculation process in the model training step, extract the same hundreds of dimensions of calculation features for the newly input streaming session samples, and use the loaded model to complete the prediction and classification (i.e. protocol identification) process.

[0088] The asset identification step is to create a built-in rule for identifying the specified server type assets in the intranet for the identified specified application layer protocol. When a specified application layer protocol stream session matches the built-in rule, the resolved server-side asset features and / or the client basic resolution features are merged to finally form an asset log; wherein, the specified server type assets include HTTP server assets, DNS server assets or FTP server assets; based on the field features corresponding to different application layer protocols, the log format of the corresponding protocol type is designed, and the log encapsulation implementation process is integrated in the corresponding application layer protocol resolution plug-in, so as to report the asset log to the specified module for display.

[0089] Specifically, after completing the protocol identification and field feature extraction based on traffic analysis, the asset identification process of the internal and external network segments can be implemented by using built-in or custom rules. For example, considering that the intranet IP address range usually includes 10 segments (10.0.0.0~10.255.255.255), 172 segments (172.16.0.0~172.31.255.255) and 192 segments (192.168.0.0~192.168.255.255), for the parsed HTTP protocol, a new built-in rule for identifying intranet HTTP server type assets can be created, which is described as "the server IP address is within the intranet IP address range". When the server IP address of an HTTP protocol stream session hits the rule, the resolved server asset features (such as server MAC address and IP address, service port number, domain name HOST, asset type tag [intranet HTTP service identifier], request URI, etc.) and client basic resolution features (such as client MAC address and IP address, user agent, access start time, end time, etc.) are merged to form an asset log and report it to the asset management module for display. Similarly, asset identification corresponding to other application layer protocol types can be completed, such as DNS server assets, FTP server assets, etc. In addition, considering that different application layer protocols have different field characteristics, it is necessary to design the log format of the corresponding protocol type when reporting, and the log encapsulation implementation process can be integrated into the corresponding application layer protocol parsing plug-in to ensure that the plug-in is independent, decoupled and extensible.

[0090] Furthermore, in some embodiments, after performing asset identification processing of a specified network segment based on the identified application layer protocol and field features and generating a corresponding asset log in the above step S4, the following is further included:

[0091] The anomaly detection algorithm model based on the unsupervised mode performs anomaly detection on the asset log field, and reports the anomaly log when an anomaly is found. The statistical feature data of each type of asset can be portrayed in the form of a portrait by analyzing the asset log field and performing statistics, wherein the statistical feature data includes: the number of assets, the total frequency of asset access, the number of source access objects, the time information of access by each source object or the number of accesses by each source object; based on the statistical feature data of the asset, extract the relevant access logs of the asset within the specified time, and parse and store the specified access feature information in the log; convert the specified access feature information into an access feature information sequence; use the probability distribution statistical analysis algorithm to learn and extract the normal behavior pattern library from the given observation data sequence; based on the normal behavior pattern library, perform anomaly detection on the asset log field, and report the anomaly log when an anomaly is found.

[0092] Specifically, after generating the asset log, this embodiment also includes the steps of asset management, such as Figure 5 As shown, it is an asset management framework diagram provided by this embodiment. The reported asset logs are stored and managed in the warehouse, which is specifically manifested in supporting configuration management functions such as adding, deleting, modifying, querying, importing or exporting various types of network assets; at the same time, the security monitoring function of assets is realized through statistical analysis and anomaly detection technology in the field of AI, which is specifically manifested in discovering abnormal values ​​of asset feature attribute fields, thereby discovering abnormal behaviors. Among them, abnormal values ​​are also called abnormalities, inconsistencies or deviations, that is, values ​​that deviate from their normal behavior patterns. Asset management includes configuration management and security monitoring. Configuration management: provides asset addition, deletion, modification, query, import or export functions. Security monitoring: includes statistical analysis and anomaly discovery functions. Statistical analysis is to analyze the asset log fields and perform statistics to depict the statistical feature data of each type of network assets in the form of portraits, such as: the number of assets, the total frequency of asset access, the number of source access objects (source IP), the time information of access by each source object, the number of accesses by each source object, etc.; and anomaly discovery is to monitor the security of assets based on the anomaly detection algorithm model.

[0093] Attached Figure 6 The figure shows a framework diagram of an unsupervised anomaly detection algorithm model proposed in this embodiment. For example, consider a scenario where multiple source IP objects access a certain asset (represented by asset A) in different time periods. Taking how to perform anomaly detection on the asset feature "time information accessed by the source object" as an example, the anomaly detection method adopted in this embodiment is described:

[0094] 1. Data extraction: Based on the reported asset logs, extract the access logs related to asset A within a configurable time period (e.g., 7 days), and parse and store the access time information of each source IP object in the log. For each source IP object, the access time information is given in the form of a sequence, such as: {time1, time2, time3, ...}.

[0095] 2. Data preprocessing: Considering that time information (such as time1) is generally given in the form of a specific moment (such as 07:56:30), the moment is converted into a time period for representation. For example, a day is divided into 4-hour intervals, forming a total of 6 time periods, namely: time period 0 corresponds to the time range [00:00:00-03:59:59], time period 1 corresponds to the time range [04:00:00-07:59:59], ..., time period 5 corresponds to the time range [20:00:00-23:59:59]. At this time, the example time point 14:56:30 is converted to time period 1. After this conversion, the access time information sequence of each source IP object will be converted into the form of a time period number, such as: {T0, T2, T1, ...}, with 'T' representing the time period, and 'T2' represents the time period number 2, and so on.

[0096] 3. Normal behavior establishment: Generally speaking, abnormal data is often rare. In order to reduce system resource usage, this embodiment uses a simple and efficient probability distribution statistical analysis algorithm to learn and extract the frequently occurring or habitual normal behavior patterns from a given observation data sequence, such as the following: Figure 7 The process is shown in the description.

[0097] First, for each source IP object, based on the obtained access time information sequence (in the form of time period number), the frequency corresponding to each time period number is obtained through statistical analysis and the final probability result is calculated, which is in the form of a dictionary structure: {T0:p0, T1:p1, T2:p2, ...}, where 'p' represents the probability, and 'p1' represents the probability of time period number 1, and so on.

[0098] Then, the above dictionary structure data is sorted in descending order according to the probability corresponding to each time period serial number, and new data is obtained: {T2: p2, T1: p1, T0: p0, ...}, where p2 is the maximum probability, followed by p1, and so on.

[0099] Furthermore, according to the descending results, a normal behavior pattern is established by iteratively calculating and determining whether the probability accumulation sum is greater than a preset configurable normal behavior probability threshold value (such as thrd=0.8). For example, when p2>=0.8, it means that the time period sequence number set {T2} is the normal behavior pattern of the access time attribute characteristics of the source object in asset A; when p2+p1>=0.8, it means that the time period sequence number set {T2, T1} is the normal behavior pattern of the access time attribute characteristics of the source object in asset A. At this point, the normal behavior pattern library of asset A for all source object access time attribute characteristics is obtained, such as: {IP1:N1, IP2:N2,...}, with 'N' representing the normal behavior pattern, then 'N1' represents the normal behavior pattern of source object IP1, and so on.

[0100] 4. Anomaly detection: Combine the established normal behavior pattern library of each source access object to detect anomalies for the new input samples. Assuming that a new access log of asset A is input through the asset identification process, refer to the above steps to extract the source object IP and its access time period sequence number features. At this time, different anomaly detection rules can be defined to detect anomalies. The example is as follows:

[0101] When the source object IP is not in the access source object set {IP1, IP2, ...} of asset A, it indicates an exception.

[0102] When the source object IP is in the access source object set {IP1, IP2, ...} of asset A, but its access time period sequence number feature is not within the normal behavior pattern of the access time attribute feature of the source object IP in asset A, it indicates an abnormality.

[0103] 5. Exception handling: When an exception is found, the exception log is reported and safety measures can be taken as needed to respond.

[0104] This embodiment proposes a network asset collection and management method based on passive traffic analysis. Through passive traffic analysis, protocol identification and asset identification are performed based on the combined built-in rules and custom rules, and protocol identification can also be performed based on a supervised AI algorithm model. Then, by jointly adopting rules and AI (Artificial Intelligence) technology, the problem of effective collection and safe management of network assets in the power energy industry is solved.

[0105] This embodiment also proposes a plug-in-based application layer protocol parsing concept, so that the system functions have good independence, decoupling and scalability.

[0106] This embodiment also proposes an asset security monitoring method based on an unsupervised AI anomaly detection algorithm model, which performs key feature processing and normal behavior pattern extraction calculation. The algorithm is easy to implement, efficient and has parallel computing capabilities.

[0107] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0108] This embodiment also provides a network asset processing system based on traffic analysis, which corresponds one-to-one to the network asset processing method based on traffic analysis in the above embodiment. Figure 8 As shown, the network asset processing system based on traffic analysis includes:

[0109] The collection module 100 is used to collect the network traffic to be analyzed and filter the collected network traffic;

[0110] The management module 110 is used to perform stream session management on the filtered network traffic to obtain a network stream session;

[0111] An identification module 120, configured to identify a corresponding application layer protocol from the network flow session by using a built-in protocol rule or a user-defined protocol rule, and parse field features of the corresponding protocol according to the identified application layer protocol type; wherein, when the corresponding application layer protocol cannot be identified from the network flow session based on the built-in protocol rule or the user-defined protocol rule, the application layer protocol is identified by using a pre-trained supervised machine learning algorithm model;

[0112] The generation module 130 is used to perform asset identification processing of a specified network segment based on the identified application layer protocol and field characteristics, and generate a corresponding asset log to perform corresponding network asset management for the asset log.

[0113] For the specific definition of each module of the network asset processing system based on traffic analysis, please refer to the definition of the network asset processing method based on traffic analysis above, which will not be repeated here. Each module in the above-mentioned network asset processing system based on traffic analysis can be implemented in whole or in part through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0114] Reference Fig. 9 , is a schematic diagram of the operating environment of a preferred embodiment of the network asset processing system 10 based on traffic analysis of the present invention.

[0115] In this embodiment, the network asset processing system 10 based on traffic analysis is installed and runs in an electronic device 1. The electronic device 1 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The electronic device 1 can be a computer, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers, where cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computers. (The electronic device 1 can be a terminal device with storage and computing functions, such as a server, a smart phone, a tablet computer, a portable computer, a desktop computer, etc. In one embodiment, when the electronic device 1 is a server, the server can be one or more of a rack server, a blade server, a tower server, a cabinet server, etc.)

[0116] In this embodiment, the electronic device 1 may include, but is not limited to, a memory 11, a processor 12, and a network interface 13 that can be interconnected via a system bus, and the memory 11 stores a network asset processing system 10 based on traffic analysis that can be run on the processor 12. It should be noted that Figure 1 The electronic device 1 is shown only with components 11 - 13 , but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0117] The memory 11 includes a memory and at least one type of readable storage medium. The memory provides a cache for the operation of the electronic device 1; the readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1; in other embodiments, the non-volatile storage medium may also be an external storage device of the electronic device 1, such as a plug-in hard disk equipped on the electronic device 1, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. In this embodiment, the readable storage medium of the memory 11 is generally used to store an operating system and various application software installed on the electronic device 1, such as storing a network asset processing system 10 based on traffic analysis in an embodiment of the present invention. In addition, the memory 11 can also be used to temporarily store various types of data that have been output or are to be output.

[0118] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as the network asset processing system 10 based on traffic analysis.

[0119] The network interface 13 may include a wireless network interface or a wired network interface. The network interface 13 is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0120] The network asset processing system 10 based on traffic analysis includes at least one computer-readable instruction stored in the memory 11 , and the at least one computer-readable instruction can be executed by the processor 12 to implement various embodiments of the present application.

[0121] In addition, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a network asset processing system based on traffic analysis, and the network asset processing system based on traffic analysis can be executed by at least one processor, so that the at least one processor performs the following steps:

[0122] Collect network traffic to be analyzed and filter the collected network traffic;

[0123] Performing stream session management on the filtered network traffic to obtain a network stream session;

[0124] Identify the corresponding application layer protocol from the network stream session by using built-in protocol rules or user-defined protocol rules, and parse the field features of the corresponding protocol according to the identified application layer protocol type; wherein, when the corresponding application layer protocol cannot be identified from the network stream session based on the built-in protocol rules or user-defined protocol rules, identify the application layer protocol by using a pre-trained supervised machine learning algorithm model;

[0125] Based on the identified application layer protocol and field characteristics, asset identification processing is performed on the specified network segment, and a corresponding asset log is generated to perform corresponding network asset management for the asset log.

[0126] The specific implementation of the computer-readable storage medium of the present invention is substantially the same as the above-mentioned electronic device 1 and each embodiment of the method, and will not be described in detail here.

[0127] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0129] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the present invention is not limited thereto. The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than that shown here.

[0130] Those skilled in the art may implement the present invention in a variety of variations without departing from the scope and essence of the present invention, such as using a feature of one embodiment to obtain another embodiment. Any modification, equivalent substitution and improvement made within the technical concept of the present invention shall be within the scope of the present invention.

Claims

1. A network asset processing method based on traffic analysis, characterized in that: The network asset processing method based on traffic analysis includes: Collect network traffic to be analyzed and filter the collected network traffic; Performing stream session management on the filtered network traffic to obtain a network stream session; Identify the corresponding application layer protocol from the network stream session by using built-in protocol rules or user-defined protocol rules, and parse the field features of the corresponding protocol according to the identified application layer protocol type; wherein, when the corresponding application layer protocol cannot be identified from the network stream session based on the built-in protocol rules or user-defined protocol rules, identify the application layer protocol by using a pre-trained supervised machine learning algorithm model; Based on the identified application layer protocol and field characteristics, asset identification processing is performed on the specified network segment, and a corresponding asset log is generated to perform corresponding network asset management for the asset log.

2. The network asset processing method based on traffic analysis according to claim 1, characterized in that: After the step of performing asset identification processing on the specified network segment based on the identified application layer protocol and field features and generating a corresponding asset log, the method further includes: The anomaly detection algorithm model based on the unsupervised mode performs anomaly detection on the asset log field, and reports the abnormal log when an anomaly is found.

3. The network asset processing method based on traffic analysis according to claim 1 or 2, characterized in that: The identifying the corresponding application layer protocol from the network stream session by using the built-in protocol rule or the custom protocol rule, and parsing the field characteristics of the corresponding protocol according to the identified application layer protocol type, specifically includes: Identify the application layer protocol in the network stream session by matching based on a single field, matching based on a logical combination of multiple field values, or matching based on a feature string, and identify and parse field features related to the application layer protocol from the network stream session according to a preset plurality of types of protocol parsing plug-ins; When the corresponding application layer protocol cannot be identified from the network flow session based on the built-in protocol rule or the custom protocol rule, identifying the application layer protocol through a pre-trained supervised machine learning algorithm model specifically includes: Based on the traffic behavior characteristics corresponding to the network flow session, the application layer protocol is identified by a pre-trained supervised machine learning algorithm model; wherein the training process of the supervised machine learning algorithm model is as follows: Collect a specified number of application layer protocol traffic samples of various types; For each sample, parse and extract basic features of the streaming session, wherein the basic features of the streaming session include: transmission protocol type, streaming start time, streaming end time, packet length sequence, packet arrival time sequence and / or TTL sequence; For each sample, based on the basic characteristics of the streaming session, statistical methods are used to complete feature calculation and storage, and hundreds of dimensions of calculation features corresponding to each sample are obtained; The LightGBM model is used to train the hundreds of dimensional computing features of each sample. After the model training is completed, the supervised machine learning algorithm model is obtained; The identification of the application layer protocol by using a pre-trained supervised machine learning algorithm model specifically includes: For the network flow session, hundreds of dimensions of computing features that are the same as those in model training are extracted, and the trained supervised machine learning algorithm model is used to predict and classify the hundreds of dimensions of computing features corresponding to the network flow session to identify the application layer protocol corresponding to the network flow session.

4. The network asset processing method based on traffic analysis according to claim 1 or 2, characterized in that: The step of performing asset identification processing on a specified network segment based on the identified application layer protocol and field features and generating a corresponding asset log includes: For the identified specified application layer protocol, a new built-in rule for identifying the specified server type asset in the intranet is created. When a specified application layer protocol stream session matches the built-in rule, the resolved server asset features and / or client basic resolution features are merged to finally form an asset log; wherein the specified server type asset includes HTTP server asset, DNS server asset or FTP server asset; Based on the field characteristics corresponding to different application layer protocols, the log format of the corresponding protocol type is designed, and the log encapsulation implementation process is integrated into the corresponding application layer protocol parsing plug-in to report the asset log to the designated module for display.

5. The network asset processing method based on traffic analysis according to claim 2, characterized in that: The anomaly detection algorithm model based on the unsupervised mode performs anomaly detection on the asset log field. When an anomaly is found, the anomaly log is reported, including: By analyzing the asset log fields and performing statistics, the statistical feature data of each type of asset is portrayed in the form of a portrait, wherein the statistical feature data includes: the number of assets, the total frequency of asset access, the number of source access objects, the time information of being accessed by each source object or the number of times being accessed by each source object; Based on the statistical feature data of the assets, extract the relevant access logs of the assets within the specified time, and parse and store the specified access feature information in the logs; Converting the specified access feature information into an access feature information sequence; Using probability distribution statistical analysis algorithms, learn and extract normal behavior pattern libraries from a given observation data sequence; Based on the normal behavior pattern library, anomaly detection is performed on the asset log field, and when an anomaly is found, the abnormal log is reported.

6. The network asset processing method based on traffic analysis according to claim 1 or 2, characterized in that: The step of collecting the network traffic to be analyzed and filtering the collected network traffic includes: Collect network traffic to be analyzed through traffic mirroring or end node deployment; Filter collected network traffic using custom configured rules based on IP address, IP address range, port and / or protocol type.

7. The network asset processing method based on traffic analysis according to claim 1 or 2, characterized in that: The step of performing stream session management on the filtered network traffic to obtain a network stream session includes: A flow session management operation including flow establishment, flow update and / or flow release is performed on the filtered network traffic to obtain a corresponding network flow session.

8. A network asset processing system based on traffic analysis, characterized in that: The network asset processing system comprises: A collection module is used to collect network traffic to be analyzed and filter the collected network traffic; A management module, used for performing flow session management on the filtered network traffic to obtain a network flow session; An identification module, used to identify the corresponding application layer protocol from the network flow session through built-in protocol rules or custom protocol rules, and parse the field characteristics of the corresponding protocol according to the identified application layer protocol type; wherein, when the corresponding application layer protocol cannot be identified from the network flow session based on the built-in protocol rules or custom protocol rules, the application layer protocol is identified through a pre-trained supervised machine learning algorithm model; The generation module is used to perform asset identification processing of the specified network segment based on the identified application layer protocol and field characteristics, and generate corresponding asset logs to perform corresponding network asset management for the asset logs.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the computer program is executed by the processor, the steps of the network asset processing method based on traffic analysis as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the network asset processing method based on traffic analysis according to any one of claims 1 to 7 are implemented.

Images