Flow metadata extraction method based on damping increment statistics and Selenium, electronic equipment and storage medium
Through the method based on damping increment statistics and Selenium, DNS traffic metadata is extracted, which solves the problem of DNS traffic feature extraction in the existing technology, and realizes efficient and accurate traffic data acquisition to meet real-time and low throughput requirements.
Patent Information
- Application Number
- CN202411234272.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-04
AI Technical Summary
The prior art is difficult to accurately, comprehensively and efficiently extract DNS traffic metadata characteristics, especially when distinguishing between normal traffic and tunnel traffic, it is difficult to meet the requirements of real-time and low throughput detection performance.
Using the traffic metadata extraction method based on damping increment statistics and Selenium, the original traffic is generated through Selenium, and the traffic acquisition and depth packet detection are used using Scapy. The session metadata is generated in combination with the damping increment statistics algorithm to simulate user behavior to obtain closer to reality. traffic data.
It realizes accurate, comprehensive and efficient extraction of DNS traffic metadata, can effectively distinguish between normal traffic and tunnel traffic, meet the needs of real-time and low throughput detection performance, and the generated traffic data is closer to the actual traffic.
Smart Images

Figure CN119071197A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of traffic metadata extraction, and in particular relates to a traffic metadata extraction method based on damped incremental statistics and Selenium. Background Art
[0002] Domain Name System (DNS), as a hierarchical distributed database system, is one of the most critical infrastructures of the Internet and an indispensable cornerstone of most network activities. The original intention of the DNS protocol was to convert complex IP addresses into domain names that are easy to remember and use, rather than for data transmission. Therefore, firewalls usually allow DNS services through port 53 in the default configuration, which allows DNS traffic to spread freely in the network. For the above reasons, the DNS protocol has become an ideal choice for tunneling technology.
[0003] Due to their different uses, there are significant differences in the characteristics of normal DNS traffic and DNS tunnel traffic. Normal DNS traffic is mainly used for domain name resolution. The data exchange between the two communicating parties is usually intermittent, non-continuous connection, and the data packet size is relatively fixed and short. DNS tunnel traffic is mainly used for data transmission. Compared with normal traffic, the two communicating parties show continuity, high frequency of queries and responses, and larger data packet length. In view of the real-time requirements and low-throughput tunnel detection performance requirements, the selected features must cover load characteristics (i.e., packet-level features) and traffic characteristics (i.e., session-level features), so accurate, comprehensive and efficient feature extraction is crucial to data analysis technology. Summary of the invention
[0004] The problem to be solved by the present invention is to generate original traffic and extract traffic metadata features accurately, comprehensively and efficiently, and propose a traffic metadata extraction method based on damped incremental statistics and Selenium.
[0005] To achieve the above object, the present invention is implemented through the following technical solutions:
[0006] A traffic metadata extraction method based on damped incremental statistics and Selenium includes the following steps:
[0007] S1. Generate original traffic based on Selenium;
[0008] S2. Collect the original traffic generated in step S1 based on Scapy to obtain the original traffic file;
[0009] S3. The original traffic file obtained in step S2 is first preprocessed, and then the data packet metadata is extracted based on deep packet inspection to obtain data packet-level metadata;
[0010] S4. The packet-level metadata obtained in step S3 is first preprocessed, and then session metadata is generated based on metadata aggregation and damped incremental statistics.
[0011] Furthermore, the specific implementation method of step S1 includes the following steps:
[0012] S1.1. Parameter setting: Set program-related parameters, including the number of search pages, waiting time after recommendation jump, waiting time after page turning, web browsing time, sleep time before clicking, and the number of keywords extracted from each word library;
[0013] S1.2.Drive selection: Select the browser drive, such as ChromeDrive;
[0014] S1.3. Search engine selection: Select a search engine as the basis for subsequent searches, such as Baidu, Sohu, and Bing;
[0015] S1.4. Keyword word library selection: Select the keyword word library as the search seed;
[0016] S1.5. Determine whether there are archive files: Use the interrupt-continuation mechanism to determine whether there are archive files left and whether the program has crashed, and then decide whether this run is a new round of crawling or a continuation of the previous round of crawling; the archive files are divided into a total keyword set and a completed keyword set; the keyword archive records all the keywords used in each round of crawling, and the completed keyword archive records the keywords that have been used;
[0017] S1.6. Random extraction of keywords: If there are no archived files left, this operation will be regarded as a new round of crawling. Randomly extract a number of data from the keyword thesaurus files in each field;
[0018] S1.7. Keyword archiving: storing the keywords extracted in this round of operation into the total keyword set;
[0019] S1.8. Keyword file reading: If there are archive files left, this run will be regarded as a continuation of the previous round of crawling. The keywords that have not been used are obtained by reading the total set of keywords and the complete set of keywords and taking the difference between them;
[0020] S1.9. Keyword traversal search: perform traversal search on the keywords used in this round. The keywords are regarded as seed keywords. After the keyword search, they are stored in the keyword completion set;
[0021] S1.10. Pre-avoidance of anti-crawler measures: Pre-avoid possible crawler detection measures by disabling the automatic control features of the Blink rendering engine, setting the User-Agent option, modifying the experimental options excludeSwitches, useAutomationExtension, and using JavaScript to overwrite the navigator.webdriver property;
[0022] S1.11. Determine whether the crawler is countered: Determine whether the search engine's anti-crawler measures are effective by detecting the structure of the web page obtained by the search;
[0023] S1.12. Introduce a recommended jump strategy. When the current keyword can no longer produce search results, it will jump through the relevant recommendations of the Baidu search engine to ensure that the number of browsed topics in each field is roughly the same;
[0024] S1.13. Search result traversal: After obtaining the search results, in order to simulate user behavior and circumvent anti-crawler measures, the Selenium crawler will randomly stay on the page for a period of time;
[0025] S1.14. Search page jump: select sequential page jump or random page jump according to the parameters set by the user;
[0026] S1.15. Delete archive: After all the keywords used in this round of operation are traversed, the archive file recording the total set of keywords and the completed set of keywords is deleted to obtain the original traffic.
[0027] Furthermore, the specific implementation method of step S2 includes the following steps:
[0028] S2.1. Monitoring protocol selection: Select the type of network protocol to be collected, including DNS and HTTP;
[0029] S2.2.BPF filter condition setting: Set BPF filter conditions to quickly filter the underlying protocols such as transport layer, network layer, data link layer, physical layer, etc.
[0030] S2.3. Packet sniffing: Use Scapy to monitor the network traffic of the specified network adapter;
[0031] S2.4.BPF fast filtering: Fast filtering of network traffic based on BPF filtering conditions;
[0032] S2.5. Deep filtering: Filter network traffic at the application layer that meets the requirements of the underlying protocol;
[0033] S2.6. Traffic saving: Save the traffic that meets the conditions by adding them to obtain the original traffic file.
[0034] Furthermore, the specific implementation method of step S3 includes the following steps:
[0035] S3.1. Original traffic file reading: All original traffic files are stored in the same specified folder path. By specifying the folder path, the file names in the folder are extracted and saved in a list form to obtain a file name list;
[0036] S3.2. File traversal: Use the file name and path provided by the file name list to read the traffic data stored in the file one by one through the iterator;
[0037] S3.3. Perform the following processing on the traffic data of a single file obtained in step S3.2:
[0038] S3.3.1. Database initialization: Create the corresponding data packet metadata table in the specified database pcap_data.db by file name and path;
[0039] S3.3.2. Packet filtering: Set the filtering conditions to DNS packets with transport layer UDP or TCP, network layer IPv4 or IPv6, and packet types A, AAAA, CNAME, MX, TXT, HTTPS, filter the data in the traffic file read in step S3.2, and obtain the filtered DNS packets;
[0040] S3.3.3. Aggregation key extraction: extract the address information and protocol of the filtered DNS data packet to form a five-tuple aggregation key, including source IP, source port, protocol, destination port, and destination IP;
[0041] The tuple order is normalized according to the port number size, and the address with smaller port number is placed at the front. This is convenient for subsequent aggregation calculations. The existence of the protocol attribute in the aggregation key is intended to leave room for the expansion of the detection range in the future.
[0042] S3.3.4. Packet-level metadata extraction: Extract and calculate packet-level metadata from the filtered DNS packets, including domain name entropy, timestamp, transport layer payload length, number of domain name labels, and ratio of non-lowercase characters in the domain name;
[0043] S3.4. Database update: Insert the aggregation key and package-level metadata of a single data packet into the table. Set it to submit an update to the database every time 10,000 data are inserted. Use this strategy as a heartbeat mechanism to observe whether the program has memory explosion.
[0044] Furthermore, the specific implementation method of step S4 includes the following steps:
[0045] S4.1. Set the extraction related parameters, including session timeout length Timeout_Num, time window sliding step Time_Step, time window length Time_Window, maximum number of empty windows Max_Null_Num, and attenuation factor Weak_Factor;
[0046] S4.2. Read the package-level metadata table obtained in step S3.4, read the names of all tables created by the user in the specified database, and store them in a list form to obtain a table name list;
[0047] S4.3. Table traversal: The data packet metadata table name provided by the table name list obtained in step S4.2 is used to traverse the data packet metadata table;
[0048] S4.4. Processing a separate data packet metadata table includes the following steps:
[0049] S4.4.1. Database initialization: Create the corresponding session metadata table and session timetable in the specified database dns_tunnel.db using the metadata table name provided in the table name list. The session timetable records the start and end time of each session, using the five-tuple aggregate key as the primary key; the session metadata table records the session metadata after damped increment calculation;
[0050] S4.4.2. Configure the time window: retrieve the timestamp of the data in the data packet metadata table, and use the minimum value as the starting time first_time and the maximum value as the ending time last_time; based on the time window length Time_Window, the time window value interval is from the starting window [0, Time_Window] to the ending window [last_time, last_time+Time_Window] through the window sliding / jumping;
[0051] When the window slides normally, the relevant calculation formulas of the time window are shown in formulas 1, 2, and 3:
[0052] window_right 0 =first_time+Time_Window (1)
[0053] window_right i =window_right i-1 +Time_Step (2)
[0054] window_left i=window_right i -Time_Window (3)
[0055] Among them, window_right 0 The right border of the initial window, window_right i is the right edge of the i-th time window, window_left i is the left edge of the i-th time window, window_right i-1 is the right boundary of the i-1th time window, Time_Window is the time window size, Time_Step is the time window sliding step, and first_time is the starting time;
[0056] S4.4.3. Extracting packet-level metadata: Search the data packet metadata table and extract the metadata in the time window interval configured in step S4.4.2;
[0057] S4.4.4. Metadata aggregation: The set of packets with the same aggregation key is regarded as a session, and the five-tuple aggregation key is used to aggregate the metadata of the packets under the same key;
[0058] During the calculation process, the packet timestamp is separated from other metadata, and an integer 1 representing the number of individual packets is inserted into the metadata header. After the aggregation calculation is completed, the session metadata dictionary and session time dictionary of the current window are obtained, which correspond to the session metadata table and session time table created in step S4.4.1;
[0059] S4.4.5. Damping increment calculation: damp the historical session metadata in the database and perform incremental calculation on the session metadata in the current window; introduce the concept of session timeout length Timeout_Num, that is, if the time interval between a new data packet and the previous data packet between the communicating parties is greater than the timeout length Timeout_Num, then this communication is considered to be a new session; the session metadata is updated by inserting instead of updating, and the session metadata table is changed, and the damping increment d λ The calculation formula of (t) is shown in Formula 4:
[0060] d λ (t) = 2 -λt (4)
[0061] Among them, λ is the attenuation factor, t is the interval time;
[0062] The calculation formula of the interval time t is shown in Formula 5:
[0063] t=cur_time-session_last_time (5)
[0064] Among them, session_last_time is the last update time of the session in the historical data, and cur_time is the current time window time;
[0065] S4.4.5. Database update: Update the metadata calculated in step S4.4.4 in the database;
[0066] S4.4.6. Window sliding or jumping: The sliding step of the time window is Time_Step, that is, when there is still no traffic in the window after multiple window sliding, the timestamp closest to the current window and greater than the current window is retrieved from the data packet metadata table and assigned to the next window to achieve window jumping.
[0067] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a traffic metadata extraction method based on damped incremental statistics and Selenium when executing the computer program.
[0068] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a traffic metadata extraction method based on damped incremental statistics and Selenium.
[0069] Beneficial effects of the present invention:
[0070] The present invention discloses a traffic metadata extraction method based on damped incremental statistics and Selenium, which uses Selenium crawler to generate DNS original traffic and uses Scapy traffic capture module to automatically collect the traffic, then uses deep packet inspection technology to extract the load characteristics of the original traffic, aggregates some characteristics into traffic characteristics through data aggregation technology, and finally forms session metadata through an improved damped incremental statistics algorithm.
[0071] The traffic metadata extraction method based on damped incremental statistics and Selenium described in the present invention better simulates the behavior of users in a real network environment during the traffic generation process, and the collected traffic is closer to the real traffic. In addition, in the metadata extraction process, the characteristic metadata at the packet level and the session level are more comprehensively considered, and the designed algorithm can efficiently complete the extraction and calculation of characteristic metadata. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] Figure 1 It is a flow chart of a traffic metadata extraction method based on damped incremental statistics and Selenium according to the present invention;
[0073] Figure 2 This is a diagram of the working mechanism of the Selenium crawler of the present invention;
[0074] Figure 3 The present invention is a flow chart of the network traffic capture program based on Scapy;
[0075] Figure 4 This is a diagram of data packet metadata extraction based on deep packet inspection of the present invention;
[0076] Figure 5 This is a graph of session metadata generation based on damped incremental statistics and data aggregation in the present invention. DETAILED DESCRIPTION
[0077] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.
[0078] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.
[0079] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 -Attached Figure 5 The detailed instructions are as follows: Specific implementation method one:
[0081] A traffic metadata extraction method based on damped incremental statistics and Selenium includes the following steps:
[0082] S1. Generate original traffic based on Selenium;
[0083] Furthermore, the specific implementation method of step S1 includes the following steps:
[0084] S1.1. Parameter setting: Set program-related parameters, including the number of search pages, waiting time after recommendation jump, waiting time after page turning, web browsing time, sleep time before clicking, and the number of keywords extracted from each word library;
[0085] S1.2.Drive selection: Select the browser drive, such as ChromeDrive;
[0086] S1.3. Search engine selection: Select a search engine as the basis for subsequent searches, such as Baidu, Sohu, and Bing;
[0087] S1.4. Keyword word library selection: Select the keyword word library as the search seed;
[0088] S1.5. Determine whether there are archive files: Use the interrupt-continuation mechanism to determine whether there are archive files left and whether the program has crashed, and then decide whether this run is a new round of crawling or a continuation of the previous round of crawling; the archive files are divided into a total keyword set and a completed keyword set; the keyword archive records all the keywords used in each round of crawling, and the completed keyword archive records the keywords that have been used;
[0089] S1.6. Random extraction of keywords: If there are no archived files left, this operation will be regarded as a new round of crawling. Randomly extract a number of data from the keyword thesaurus files in each field;
[0090] S1.7. Keyword archiving: storing the keywords extracted in this round of operation into the total keyword set;
[0091] S1.8. Keyword file reading: If there are archive files left, this run will be regarded as a continuation of the previous round of crawling. The keywords that have not been used are obtained by reading the total set of keywords and the complete set of keywords and taking the difference between them;
[0092] S1.9. Keyword traversal search: perform traversal search on the keywords used in this round. The keywords are regarded as seed keywords. After the keyword search, they are stored in the keyword completion set;
[0093] S1.10. Pre-avoidance of anti-crawler measures: Pre-avoid possible crawler detection measures by disabling the automatic control features of the Blink rendering engine, setting the User-Agent option, modifying the experimental options excludeSwitches, useAutomationExtension, and using JavaScript to overwrite the navigator.webdriver property;
[0094] S1.11. Determine whether the crawler is countered: Determine whether the search engine's anti-crawler measures are effective by detecting the structure of the web page obtained by the search;
[0095] S1.12. Introduce a recommended jump strategy. When the current keyword can no longer produce search results, it will jump through the relevant recommendations of the Baidu search engine to ensure that the number of browsed topics in each field is roughly the same;
[0096] S1.13. Search result traversal: After obtaining the search results, in order to simulate user behavior and circumvent anti-crawler measures, the Selenium crawler will randomly stay on the page for a period of time;
[0097] S1.14. Search page jump: select sequential page jump or random page jump according to the parameters set by the user;
[0098] Furthermore, sequential page jump or random page jump is selected. Sequential page jump is more in line with user habits. Users often only pay attention to web pages with higher rankings when using search engines, that is, sequential page browsing. Random page jump can expand the coverage of web pages collected for traffic data, and solve the problem of low ranking of web pages in search results with less visits, making traffic more comprehensive;
[0099] S1.15. Delete archive: After all the keywords used in this round of operation are traversed, the archive files recording the total set of keywords and the completed set of keywords are deleted to obtain the original traffic;
[0100] Furthermore, the main goal of the Selenium crawler is to collect as much normal network traffic as possible in an environment that simulates the user's daily network behavior. In order to increase the diversity of the collected network traffic, the Tsinghua University Open Chinese Vocabulary THUOCL was used in the selection of keywords. THUOCL related information is shown in Table 1.
[0101] Table 1 THUOCL vocabulary
[0102]
[0103] S2. Collect the original traffic generated in step S1 based on Scapy to obtain the original traffic file;
[0104] Furthermore, the specific implementation method of step S2 includes the following steps:
[0105] S2.1. Monitoring protocol selection: Select the type of network protocol to be collected, including DNS and HTTP;
[0106] S2.2.BPF filter condition setting: Set BPF filter conditions to quickly filter the underlying protocols such as transport layer, network layer, data link layer, physical layer, etc.
[0107] S2.3. Packet sniffing: Use Scapy to monitor the network traffic of the specified network adapter;
[0108] S2.4.BPF fast filtering: Fast filtering of network traffic based on BPF filtering conditions;
[0109] S2.5. Deep filtering: Filter network traffic at the application layer that meets the requirements of the underlying protocol;
[0110] S2.6. Traffic saving: save the traffic that meets the conditions by adding them to obtain the original traffic file;
[0111] S3. The original traffic file obtained in step S2 is first preprocessed, and then the data packet metadata is extracted based on deep packet inspection to obtain data packet-level metadata;
[0112] The typical characteristics of DNS tunnels can be divided into two categories: load characteristics and flow characteristics. Load characteristics extract characteristics from a more microscopic and specific perspective by analyzing the load information of a single flow packet or multiple flow packets. Flow characteristics extract characteristics from a more macroscopic perspective by analyzing the flow information of the overall flow over a period of time.
[0113] In view of the need for real-time performance and the requirements for low-throughput tunnel detection performance, load characteristics and traffic characteristics are comprehensively selected based on technologies such as deep packet inspection, data aggregation, and damped incremental statistics. The selected characteristics are shown in Table 2.
[0114] Table 2 Related characteristics
[0115]
[0116] In view of the characteristics of low-throughput DNS tunnels, the traffic features packet_num and dur_time are selected as analysis indicators. Under the premise that the length of the leaked data is fixed, in order to evade detection, low-throughput DNS tunnels usually take measures such as reducing the length of a single leaked data, increasing the total number of messages, and extending the total leakage time. The increase in the total number of messages will show abnormalities in the traffic feature packet_num, while the extension of the leakage time will make the traffic feature dur_time show abnormalities.
[0117] DNS tunnel messages can be divided into tunnel query messages used to transmit leaked data and tunnel response messages used to transmit control commands. The above two messages differ from normal DNS query and response messages in terms of function, and also in terms of structure and content. Therefore, in view of the differences in DNS tunnel traffic at the transport layer protocol load level, the load feature payload_size is selected.
[0118] Domain names are the core content of DNS messages, and domain name-related features play an important role in the detection of DNS tunnels. According to the principle of DNS tunnels, leaked data is mainly carried by the domain name field of DNS tunnel messages. In view of the characteristics that leaked data in DNS tunnels usually uses encoding and has a large amount of data, the load features domain_entropy, domain_lable, and domain_NL_rate are selected.
[0119] Character statistics were performed on 248,726 benign domain names from the Alexa domain name ranking and 105,083 tunnel domain names generated by the DNS tunnel tool. The statistical results show that lowercase characters account for about 97.9% of the total proportion in benign domain names, numbers account for about 1% of the total proportion, and underscores and hyphens account for about 1.07% of the total proportion. In contrast, the proportion of lowercase letters in tunnel domain names is about 28.97%, numbers are about 16%, uppercase letters are about 54%, and the proportion of underscores and hyphens is only about 0.0038%. In addition, special characters (excluding underscores and hyphens) account for about 1.15% of the total proportion. From these data, it can be seen that compared with benign domain names, the characteristics of tunnel domain names show some obvious changes. In tunnel domain names, the proportion of lowercase letters, underscores, and hyphens has decreased significantly, while the proportion of numbers has increased significantly. In addition, uppercase letters and special characters other than underscores and hyphens also appear in tunnel domain names. Therefore, the non-lowercase character ratio domain_NL_rate in the domain name is selected as one of the load features.
[0120] In the research phase, the extraction of session metadata based on deep packet inspection and damped incremental statistics is divided into two stages: data packet metadata extraction based on deep packet inspection, and session metadata generation based on damped incremental statistics and data aggregation. The purpose of step-by-step processing is to use the database to process large traffic files, thereby reducing the requirements for the hardware performance of the training equipment and improving the efficiency of data retrieval in the subsequent process. SQLite database is adopted in the research phase because of its advantages such as open source, lightweight, easy to deploy, and good cross-platform performance.
[0121] Furthermore, the specific implementation method of step S3 includes the following steps:
[0122] S3.1. Original traffic file reading: All original traffic files are stored in the same specified folder path. By specifying the folder path, the file names in the folder are extracted and saved in a list form to obtain a file name list;
[0123] S3.2. File traversal: Use the file name and path provided by the file name list to read the traffic data stored in the file one by one through the iterator;
[0124] S3.3. Perform the following processing on the traffic data of a single file obtained in step S3.2:
[0125] S3.3.1. Database initialization: Create the corresponding data packet metadata table in the specified database pcap_data.db by file name and path, as shown in Table 3:
[0126] Table 3 Data packet metadata table
[0127]
[0128]
[0129] S3.3.2. Packet filtering: Set the filtering conditions to DNS packets with transport layer UDP or TCP, network layer IPv4 or IPv6, and packet types A, AAAA, CNAME, MX, TXT, HTTPS, filter the data in the traffic file read in step S3.2, and obtain the filtered DNS packets;
[0130] S3.3.3. Aggregation key extraction: extract the address information and protocol of the filtered DNS data packet to form a five-tuple aggregation key, including source IP, source port, protocol, destination port, and destination IP;
[0131] The tuple order is normalized according to the port number size, and the address with smaller port number is placed at the front. This is convenient for subsequent aggregation calculations. The existence of the protocol attribute in the aggregation key is intended to leave room for the expansion of the detection range in the future.
[0132] S3.3.4. Packet-level metadata extraction: Extract and calculate packet-level metadata from the filtered DNS packets, including domain name entropy, timestamp, transport layer payload length, number of domain name labels, and ratio of non-lowercase characters in the domain name;
[0133] S3.4. Database update: Insert the aggregation key and package-level metadata of a single data packet into the table. Set it to submit an update to the database every time 10,000 data are inserted. Use this strategy as a heartbeat mechanism to observe whether the program has memory explosion.
[0134] S4. The packet-level metadata obtained in step S3 is first preprocessed, and then session metadata is generated based on metadata aggregation and damped incremental statistics;
[0135] Furthermore, the specific implementation method of step S4 includes the following steps:
[0136] S4.1. Set the extraction related parameters, including session timeout length Timeout_Num, time window sliding step Time_Step, time window length Time_Window, maximum number of empty windows Max_Null_Num, and attenuation factor Weak_Factor;
[0137] S4.2. Read the data packet-level metadata table obtained in step S3.4, read the names of all tables created by the user in the specified database, and store them in a list to obtain a table name list;
[0138] S4.3. Table traversal: The data packet metadata table name provided by the table name list obtained in step S4.2 is used to traverse the data packet metadata table;
[0139] S4.4. Processing a separate data packet metadata table includes the following steps:
[0140] S4.4.1. Database initialization: Create the corresponding session metadata table and session timetable in the specified database dns_tunnel.db using the metadata table name provided in the table name list. The session timetable records the start and end time of each session, using the five-tuple aggregate key as the primary key; the session metadata table records the session metadata after damped increment calculation;
[0141] Furthermore, a five-tuple aggregate key (source IP, source port, protocol, destination port, destination IP) is used as the primary key, and the record corresponding to each session is unique. The session metadata table records the session metadata after damped increment calculation. There are many advantages to separating the two tables: from the perspective of time cost, the separation of the two tables is conducive to quickly retrieving session time information in the timetable. From the perspective of space cost, since the start time of the same session is the same, the separation of the two tables is conducive to saving space. The session timetable is shown in Table 4, and the session metadata table is shown in the table:
[0142] Table 4 Session schedule
[0143]
[0144] Table 5 Session metadata table
[0145]
[0146] S4.4.2. Configure the time window: retrieve the timestamp of the data in the data packet metadata table, and use the minimum value as the starting time first_time and the maximum value as the ending time last_time; based on the time window length Time_Window, the time window value interval is from the starting window [0, Time_Window] to the ending window [last_time, last_time+Time_Window] through the window sliding / jumping;
[0147] When the window slides normally, the relevant calculation formulas of the time window are shown in formulas 1, 2, and 3:
[0148] window_right 0 =first_time+Time_Window (1)
[0149] window_right i =window_right i-1 +Time_Step (2)
[0150] window_left i =window_right i -Time_Window (3)
[0151] Among them, window_right 0 The right border of the initial window, window_right i is the right edge of the i-th time window, window_left i is the left edge of the i-th time window, window_right i-1 is the right boundary of the i-1th time window, Time_Window is the time window size, Time_Step is the time window sliding step, and first_time is the starting time;
[0152] S4.4.3. Extracting packet-level metadata: Search the data packet metadata table and extract the metadata in the time window interval configured in step S4.4.2;
[0153] S4.4.4. Metadata aggregation: The set of packets with the same aggregation key is regarded as a session, and the five-tuple aggregation key is used to aggregate the metadata of the packets under the same key;
[0154] During the calculation process, the packet timestamp is separated from other metadata, and an integer 1 representing the number of single packets is inserted into the metadata header. After the aggregation calculation is completed, the session metadata dictionary and session time dictionary of the current window are obtained, which correspond to the session metadata table and session time table created in step S4.4.1; metadata aggregation is as follows:
[0155] Metadata_Aggregation
[0156] Input: Packets
[0157] Output: Session metadata dictionary Sessions_Metadata, session time dictionary Sessions_Time
[0158] 1: Data aggregation stage:
[0159] 2: Create dictionary Packets_Metadata
[0160] 3: forpacket in Packets do
[0161] 4: Extract the aggregation key packet_key of the current data packet
[0162] 5: Extract the metadata packet_metadata of the current data packet
[0163] 6: Put the data packets with the same aggregation key into the collection under the same key of the dictionary Packest_Metadata
[0164] 7: end for
[0165] 8: Aggregation calculation stage:
[0166] 9: Create session metadata dictionary Sessions_Metadata
[0167] 10: Create session time dictionary Sessions_Time
[0168] 11: forpackets_metadata inPackets_Metadata do
[0169] 12: forpacket_metadata inpackets_metadata do
[0170] 13: Extract the current data packet aggregation key aggregation_key
[0171] 14: Extract the current data packet timestamp time
[0172] 15: Extract the metadata of the current data packet, delete the timestamp time, and add 1 to the first position to indicate the number
[0173] 16: If the current aggregation key aggregation_key does not exist in the session time dictionary Sessions_Time
[0174] 17: Create an entry in the session time data dictionary, with the key being the current aggregation key and the value being (time, time)
[0175] 18: Create an entry in the session metadata dictionary, with the key being the current aggregation key and the value being the processed metadata
[0176] 19: Otherwise
[0177] 20: Extract the historical metadata tuple session_data for this key in the session metadata dictionary
[0178] 21: Modify the session time dictionary and change the last element of the tuple representing the latest time under the key to time
[0179] 22: Modify the session metadata dictionary, add session_data and metadata bit by bit and modify the corresponding values
[0180] 23: end for
[0181] 24: end for.
[0182] S4.4.5. Damping increment calculation: damp the historical session metadata in the database and perform incremental calculation on the session metadata in the current window; introduce the concept of session timeout length Timeout_Num, that is, if the time interval between a new data packet and the previous data packet between the communicating parties is greater than the timeout length Timeout_Num, then this communication is considered to be a new session; the session metadata is updated by inserting instead of updating, and the session metadata table is changed, and the damping increment d λ The calculation formula of (t) is shown in Formula 4:
[0183] d λ (t) = 2 -λt (4)
[0184] Among them, λ is the attenuation factor, t is the interval time;
[0185] The calculation formula of the interval time t is shown in Formula 5:
[0186] t=cur_time-session_last_time(5)
[0187] Among them, session_last_time is the last update time of the session in the historical data, and cur_time is the current time window time;
[0188] The steps of damping increment calculation are as follows:
[0189] Algorithm damping decay and increment calculation Decay_and_Caculation
[0190] Input: Session metadata dictionary Sessions_Metadata, session time dictionary Sessions_Time, database Database, session timeout length Timeout_Num
[0191] Output: Database
[0192] 1: forkey in Sessions_Time.Keys do
[0193] 2: Extract the session start time session_start_time of the session recorded in the database session time table
[0194] 3: Extract the latest update time of the session recorded in the database session time table, session_update_time
[0195] 4: Extract the start time of the session in the session time dictionary, cur_start_time
[0196] 5: Extract the latest update time cur_end_time of the session in the session time dictionary
[0197] 6: If the key does not exist in the database
[0198] 7: Session duration dur_time is the time difference between the session in the dictionary cur_end_time-cur_start_time
[0199] 8: Insert data insert_data is the union of the session metadata metadata and the session duration dur_time
[0200] 9: Insert the union of the aggregate key key and the inserted data insert_data in the database session time table
[0201] 10: Otherwise
[0202] 11: The interval interval is the difference between cur_start_time and session_update_time
[0203] 12: If the time interval is greater than the session timeout, it is considered a new session
[0204] 13: Session duration dur_time is the time difference between the session in the dictionary cur_end_time and cur_start_time
[0205] 14: The data to be inserted insert_data is the union of the session metadata metadata and the duration dur_time
[0206] 15: Update the database schedule, update the start and latest time, the value is cur_start_time, cur_end_time
[0207] 16: Otherwise
[0208] 17: Session duration dur_time is the difference between cur_end_time and session_start_time
[0209] 18: The data to be calculated cal_data is the union of the session metadata metadata and the duration dur_time
[0210] 19: Extract the latest historical data history_data from the database metadata table
[0211] 20: Calculate the data to be inserted insert_data according to the following formula:
[0212] insert_data←cal_data+history_data·2 -λ′interval
[0213] 21: Update the database session time table and only update the latest value of the session, which is cur_end_time
[0214] 24: Insert into the database session metadata table, and update the value to be the union of the aggregation key key and the data to be inserted insert_data
[0215] 25: end for.
[0216] S4.4.5. Database update: Update the metadata calculated in step S4.4.4 in the database;
[0217] S4.4.6. Window sliding or jumping: The sliding step of the time window is Time_Step, that is, when there is still no traffic in the window after multiple window sliding, the timestamp closest to the current window and greater than the current window is retrieved from the data packet metadata table and assigned to the next window to achieve window jumping.
[0218] Embodiment 2:
[0219] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a traffic metadata extraction method based on damped incremental statistics and Selenium as described in Example 1 are implemented.
[0220] The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit. Furthermore, the processor is used to implement the steps of the above-mentioned recommendation method based on CREO software that can modify the recommendation data driven by the relationship when executing the computer program stored in the memory.
[0221] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0222] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0223] Embodiment 3:
[0224] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a traffic metadata extraction method based on damped incremental statistics and Selenium as described in Example 1.
[0225] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by a processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. A computer program is stored on the computer-readable storage medium. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned modeling method of modifiable relationship-driven modeling data based on CREO software can be implemented.
[0226] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.
[0227] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0228] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.
Claims
1. A traffic metadata extraction method based on damped incremental statistics and Selenium, characterized in that: The steps include: S1. Generate original traffic based on Selenium; S2. Collect the original traffic generated in step S1 based on Scapy to obtain the original traffic file; S3. The original traffic file obtained in step S2 is first preprocessed, and then the data packet metadata is extracted based on deep packet inspection to obtain data packet-level metadata; S4. The packet-level metadata obtained in step S3 is first preprocessed, and then session metadata is generated based on metadata aggregation and damped incremental statistics.
2. According to claim 1, a traffic metadata extraction method based on damped incremental statistics and Selenium is characterized in that: The specific implementation method of step S1 includes the following steps: S1.
1. Parameter setting: Set program-related parameters, including the number of search pages, waiting time after recommendation jump, waiting time after page turning, web browsing time, sleep time before clicking, and the number of keywords extracted from each word library; S1.2.Drive selection: Select the browser drive, such as ChromeDrive; S1.
3. Search engine selection: Select a search engine as the basis for subsequent searches, such as Baidu, Sohu, and Bing; S1.
4. Keyword word library selection: Select the keyword word library as the search seed; S1.
5. Determine whether there are archive files: Use the interrupt-continuation mechanism to determine whether there are archive files left and whether the program has crashed, and then decide whether this run is a new round of crawling or a continuation of the previous round of crawling; the archive files are divided into a total keyword set and a completed keyword set; the keyword archive records all the keywords used in each round of crawling, and the completed keyword archive records the keywords that have been used; S1.
6. Random extraction of keywords: If there are no archived files left, this operation will be regarded as a new round of crawling. Randomly extract a number of data from the keyword thesaurus files in each field; S1.
7. Keyword archiving: storing the keywords extracted in this round of operation into the total keyword set; S1.
8. Keyword file reading: If there are archive files left, this run will be regarded as a continuation of the previous round of crawling. The keywords that have not been used are obtained by reading the total set of keywords and the complete set of keywords and taking the difference between them; S1.
9. Keyword traversal search: perform traversal search on the keywords used in this round. The keywords are regarded as seed keywords. After the keyword search, they are stored in the keyword completion set; S1.
10. Pre-avoidance of anti-crawler measures: Pre-avoid possible crawler detection measures by disabling the automatic control features of the Blink rendering engine, setting the User-Agent option, modifying the experimental options excludeSwitches, useAutomationExtension, and using JavaScript to overwrite the navigator.webdriver property; S1.
11. Determine whether the crawler is countered: Determine whether the search engine's anti-crawler measures are effective by detecting the structure of the web page obtained by the search; S1.
12. Introduce a recommended jump strategy. When the current keyword can no longer produce search results, it will jump through the relevant recommendations of the Baidu search engine to ensure that the number of browsed topics in each field is roughly the same; S1.
13. Search result traversal: After obtaining the search results, in order to simulate user behavior and circumvent anti-crawler measures, the Selenium crawler will randomly stay on the page for a period of time; S1.
14. Search page jump: select sequential page jump or random page jump according to the parameters set by the user; S1.
15. Delete archive: After all the keywords used in this round of operation are traversed, the archive file recording the total set of keywords and the completed set of keywords is deleted to obtain the original traffic.
3. According to claim 2, a method for extracting traffic metadata based on damped incremental statistics and Selenium is characterized in that: The specific implementation method of step S2 includes the following steps: S2.
1. Monitoring protocol selection: Select the type of network protocol to be collected, including DNS and HTTP; S2.2.BPF filter condition setting: Set BPF filter conditions to quickly filter the underlying protocols such as transport layer, network layer, data link layer, physical layer, etc. S2.
3. Packet sniffing: Use Scapy to monitor the network traffic of the specified network adapter; S2.4.BPF fast filtering: Fast filtering of network traffic based on BPF filtering conditions; S2.
5. Deep filtering: Filter network traffic at the application layer that meets the requirements of the underlying protocol; S2.
6. Traffic saving: Save the traffic that meets the conditions by adding them to obtain the original traffic file.
4. According to claim 3, a method for extracting traffic metadata based on damped incremental statistics and Selenium is characterized in that: The specific implementation method of step S3 includes the following steps: S3.
1. Original traffic file reading: All original traffic files are stored in the same specified folder path. By specifying the folder path, the file names in the folder are extracted and saved in a list form to obtain a file name list; S3.
2. File traversal: Using the file name and path provided by the file name list, read the traffic data stored in the file one by one through the iterator; S3.
3. Perform the following processing on the traffic data of a single file obtained in step S3.2: S3.3.
1. Database initialization: Create the corresponding data packet metadata table in the specified database pcap_data.db by file name and path; S3.3.
2. Packet filtering: Set the filtering conditions to DNS packets with transport layer UDP or TCP, network layer IPv4 or IPv6, and packet types A, AAAA, CNAME, MX, TXT, HTTPS, filter the data in the traffic file read in step S3.2, and obtain the filtered DNS packets; S3.3.
3. Aggregation key extraction: extract the address information and protocol of the filtered DNS data packet to form a five-tuple aggregation key, including source IP, source port, protocol, destination port, and destination IP; The tuple order is normalized according to the port number size, and the address with smaller port number is placed at the front. This is convenient for subsequent aggregation calculations. The existence of the protocol attribute in the aggregation key is intended to leave room for the expansion of the detection range in the future. S3.3.
4. Packet-level metadata extraction: Extract and calculate packet-level metadata from the filtered DNS packets, including domain name entropy, timestamp, transport layer payload length, number of domain name labels, and ratio of non-lowercase characters in the domain name; S3.
4. Database update: Insert the aggregation key and package-level metadata of a single data packet into the table. Set it to submit an update to the database every time 10,000 data are inserted. Use this strategy as a heartbeat mechanism to observe whether the program has memory explosion.
5. According to claim 4, a method for extracting traffic metadata based on damped incremental statistics and Se1enium is characterized in that: The specific implementation method of step S4 includes the following steps: S4.
1. Set the extraction related parameters, including session timeout length Timeout_Num, time window sliding step Time_Step, time window length Time_Window, maximum number of empty windows Max_Null_Num, and attenuation factor Weak_Factor; S4.
2. Read the package-level metadata table obtained in step S3.4, read the names of all tables created by the user in the specified database, and store them in a list form to obtain a table name list; S4.
3. Table traversal: The data packet metadata table name provided by the table name list obtained in step S4.2 is used to traverse the data packet metadata table; S4.
4. Processing a separate data packet metadata table includes the following steps: S4.4.
1. Database initialization: Create the corresponding session metadata table and session timetable in the specified database dns_tunnel.db using the metadata table name provided in the table name list. The session timetable records the start and end time of each session, and uses the five-tuple aggregate key as the primary key. The session metadata table records the session metadata after the damping increment calculation; S4.4.
2. Configure the time window: retrieve the timestamp of the data in the data packet metadata table, and use the minimum value as the starting time first_time and the maximum value as the ending time last_time; based on the time window length Time_Window, the time window value interval is from the starting window [0, Time_Window] to the ending window [last_time, last_time+Time_Window] through the window sliding / jumping; When the window slides normally, the relevant calculation formulas of the time window are shown in formulas 1, 2, and 3: window_right0=first_time+Time_Window (1) window_right i =window_right i-1 +Time_Step (2) window_left i =window_right i -Time_Window (3) Among them, window_right0 is the right border of the initial window, window_right i is the right edge of the i-th time window, window_left i is the left edge of the i-th time window, window_right i-1 is the right boundary of the i-1th time window, Time_Window is the time window size, Time_Step is the time window sliding step, and first_time is the starting time; S4.4.
3. Extracting packet-level metadata: Search the data packet metadata table and extract the metadata in the time window interval configured in step S4.4.2; S4.4.
4. Metadata aggregation: The set of packets with the same aggregation key is regarded as a session, and the five-tuple aggregation key is used to aggregate the metadata of the packets under the same key; During the calculation process, the packet timestamp is separated from other metadata, and an integer 1 representing the number of individual packets is inserted into the metadata header. After the aggregation calculation is completed, the session metadata dictionary and session time dictionary of the current window are obtained, which correspond to the session metadata table and session time table created in step S4.4.1; S4.4.
5. Damping increment calculation: damp the historical session metadata in the database and perform incremental calculation on the session metadata in the current window; introduce the concept of session timeout length Timeout_Num, that is, if the time interval between a new data packet and the previous data packet between the communicating parties is greater than the timeout length Timeout_Num, then this communication is considered to be a new session; the session metadata is updated by inserting instead of updating, and the session metadata table is changed, and the damping increment d λ The calculation formula of (t) is shown in Formula 4: d λ (t)=2 -λt (4) Among them, λ is the attenuation factor, t is the interval time; The calculation formula of the interval time t is shown in Formula 5: t=cur_time-session_last_time (5) Among them, session_last_time is the last update time of the session in the historical data, and cur_time is the current time window time; S4.4.
5. Database update: Update the metadata calculated in step S4.4.4 in the database; S4.4.
6. Window sliding or jumping: The sliding step of the time window is Time_Step, that is, when there is still no traffic in the window after multiple window sliding, the timestamp closest to the current window and greater than the current window is retrieved from the data packet metadata table and assigned to the next window to achieve window jumping.
6. An electronic device, characterized in that: It comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of a traffic metadata extraction method based on damped incremental statistics and Selenium as described in any one of claims 1 to 7 when executing the computer program.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting traffic metadata based on damped incremental statistics and Selenium as described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Method for extracting, analyzing and searching network flow and content
CN103281213A
Network traffic statistics implement method supporting multi-dimensional aggregation classification
CN106452868A
A fast backtracking analysis method for network packets
CN109067711A
DNS hidden tunnel detection method and system
CN111953673A
Network traffic retrieval method and device
CN113590910A
