Plaintext WEB scanning detection method, device, electronic device and storage medium

Abstract

The present invention provides a method, device, electronic device and storage medium for plaintext WEB scanning detection, which relates to the technical field of network security. The method comprises: extracting corresponding TCP handshake feature data and HTTP feature data from each scanned TCP data stream in the plaintext WEB scanning traffic, and then combining the two feature data corresponding to each scanned TCP data stream to obtain a combined feature table; then calculating the information summary based on the combined feature table, and obtaining the corresponding first information summary value table after filtering; matching the second information summary value corresponding to the combined feature of the TCP data stream to be detected with the first information summary value table, and if the match is successful, the TCP data stream to be detected is a scanning traffic. The method solves the technical problems of insufficient timely response and low accuracy for plaintext WEB scanning detection, and achieves the technical effect of improving scanning detection efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a plaintext WEB scanning detection method, device, electronic equipment and storage medium. Background Art

[0002] Network attacks refer to the behavior of attacking systems and resources by exploiting the vulnerabilities and security flaws in network information systems. They are usually composed of several subtasks with specific goals, such as reconnaissance, weapon control, delivery, utilization, installation, command, control, and implementation. In reality, programs that perform various reconnaissance tasks are collectively referred to as scanners. Depending on the scanned objects, the functions of scanners can be further divided into system scanning, WEB scanning, database scanning, etc. Among them, a large number of requests sent by WEB scanning will cause a surge in server traffic, which may cause service interruption, affect normal website operation and user access; at the same time, a large number of scan requests may also generate a large number of error records in the server log, which may overwhelm the real problem log, making it difficult for administrators to quickly locate and solve the problem; in addition, attackers may use security vulnerabilities discovered during the scanning process to attack the server, thereby stealing sensitive information, destroying WEB applications and even operating systems, and seriously threatening the security of the server; finally, attackers may tamper with specific web pages by manipulating the database to embed web horse links to carry out horse attacks.

[0003] Based on this, an effective real-time and accurate detection method is needed to perform plaintext WEB scanning for detection, so as to reduce the risk of WEB applications and operating systems being attacked by network attacks. In other words, the detection method for plaintext WEB scanning in the prior art has the technical problems of insufficient response time and low accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide a plaintext WEB scanning detection method, device, electronic device and storage medium to alleviate the technical problems existing in the prior art of untimely response and low accuracy in detecting plaintext WEB scanning.

[0005] In the first aspect, an embodiment of the present invention provides a plaintext WEB scanning detection method, the method comprising: extracting TCP handshake feature data and HTTP feature data corresponding to each scanning TCP data stream in the plaintext WEB scanning traffic; the plaintext WEB scanning traffic is collected and obtained by various WEB scanners; the plaintext WEB scanning traffic includes a plurality of scanning TCP data streams; performing feature combination on the TCP handshake feature data and the HTTP feature data corresponding to each of the scanning TCP data streams to obtain a combined feature table; performing information summary calculation based on the combined feature table to obtain a first information summary value table corresponding to each of the scanning TCP data streams after filtering; matching the second information summary value corresponding to the combined feature of the TCP data stream to be detected with the first information summary value table corresponding to each of the scanning TCP data streams; if the match is successful, the TCP data stream to be detected is determined to be scanning traffic, otherwise, the TCP data stream to be detected is determined to be regular traffic.

[0006] In some optional implementations, TCP handshake feature data and HTTP feature data corresponding to each scanned TCP data stream in the plaintext WEB scanning traffic are respectively extracted from the above scanned TCP data streams, including: performing feature extraction on each of the above scanned TCP data streams according to the data packet type to obtain the TCP handshake feature and HTTP feature corresponding to each of the above scanned TCP data streams; each TCP handshake feature corresponding to the above scanned TCP data stream corresponds to a feature number, and the above feature number includes a first category number, a second category number and a third category number; performing byte sampling on the above TCP handshake features and the above HTTP features, and preprocessing the results of the byte sampling to obtain the TCP handshake feature data and HTTP feature data.

[0007] In some optional implementations, the above-mentioned preprocessing includes: recording the missing features after feature extraction as character 0; converting the results of byte-sampling of the features corresponding to the above-mentioned first category number into decimal numbers; discretizing the results of byte-sampling of the features corresponding to the above-mentioned second category number according to pre-established rules; segmenting the results of byte-sampling of the features corresponding to the above-mentioned third category number into units of single bytes, and converting each byte after segmentation into decimal numbers; and reassigning the sum of the decimal numbers converted from the features corresponding to the above-mentioned three category numbers to the features corresponding to the above-mentioned three category numbers in the form of characters.

[0008] In some optional implementations, the TCP handshake feature data and the HTTP feature data corresponding to each of the above-mentioned scanned TCP data streams are respectively feature combined to obtain a combined feature table, including: grouping the TCP handshake feature data and the HTTP feature data corresponding to each of the above-mentioned scanned TCP data streams to obtain a first number of feature groups; determining all combination methods of the first number of feature groups under a set length to generate a combined feature table; the set length is any positive integer in the range of 1 to the first number.

[0009] In some optional implementations, information summary calculation is performed based on the above-mentioned combined feature table, and a first information summary value table corresponding to each of the above-mentioned scanned TCP data streams is obtained after filtering, including: according to the above-mentioned combined feature table, the values ​​of the corresponding features in each combination method are sequentially concatenated into a string; the MD5 value of each of the above-mentioned strings is calculated to generate a basic information summary table; the MD5 values ​​of the strings corresponding to the TCP handshake features and HTTP features in the same above-mentioned scanned TCP data stream are in the same row in the above-mentioned basic information summary table; the above-mentioned basic information summary table is filtered based on a preset parameter table to generate a first information summary value table corresponding to each of the above-mentioned scanned TCP data streams.

[0010] In some optional implementations, the above-mentioned preset parameter table includes: a first variable parameter, a filtering parameter, a feature combination weight parameter, and a matching threshold parameter; the above-mentioned basic information summary table is filtered based on the preset parameter table to generate a first information summary value table corresponding to each of the above-mentioned scanned TCP data streams, including: based on the above-mentioned filtering parameters, the above-mentioned basic information summary table is respectively filtered by columns and rows to generate a first information summary value table corresponding to each of the above-mentioned scanned TCP data streams.

[0011] In some optional implementations, the above method also includes: based on the above first variable parameter, extracting the TCP-SYN handshake feature data and HTTP feature data corresponding to the TCP data stream to be detected; based on the above feature combination weight parameter and the number of columns of the above first information summary value table, determining the combined features of the TCP-SYN handshake feature data and HTTP feature data corresponding to the above TCP data stream to be detected; and calculating the second information summary value corresponding to the above combined features.

[0012] In some optional implementations, the second information summary value corresponding to the combined feature of the TCP data stream to be detected is matched with the first information summary value table corresponding to each of the above-mentioned scanned TCP data streams, including: comparing the second information summary value corresponding to the above-mentioned TCP data stream to be detected with the value of the corresponding combined feature in the above-mentioned first information summary value table, and determining that the combined features corresponding to the same value are matching items; determining the row index number of each matching item in the above-mentioned first information summary value table; determining the target row index number and the frequency of occurrence of the above-mentioned target row index number based on several of the above-mentioned row index numbers; and judging whether the above-mentioned TCP data stream to be detected is scanning traffic based on the frequency of occurrence of the above-mentioned target row index number and the above-mentioned matching threshold parameter.

[0013] In the second aspect, an embodiment of the present invention provides a plaintext WEB scanning detection device, which includes: a feature data extraction module, which is used to extract TCP handshake feature data and HTTP feature data corresponding to each scanning TCP data stream in the plaintext WEB scanning traffic; the plaintext WEB scanning traffic is collected and obtained by various WEB scanners; the plaintext WEB scanning traffic includes several scanning TCP data streams; a feature combination module, which is used to perform feature combination on the TCP handshake feature data and the HTTP feature data corresponding to each of the above scanning TCP data streams, and obtain a combined feature table; an information summary calculation module, which is used to perform information summary calculation based on the above combined feature table, and obtain a first information summary value table corresponding to each of the above scanning TCP data streams after filtering; a matching module, which is used to match the second information summary value corresponding to the combined feature of the TCP data stream to be detected with the first information summary value table corresponding to each of the above scanning TCP data streams; if the match is successful, the TCP data stream to be detected is determined to be scanning traffic, otherwise, the TCP data stream to be detected is determined to be regular traffic.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the steps of any one of the methods described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to execute any method described in the first aspect.

[0016] The present invention provides a method, device, electronic device and storage medium for plaintext WEB scanning detection. The method comprises: firstly extracting TCP handshake feature data and HTTP feature data corresponding to each scanning TCP data stream from each scanning TCP data stream in the plaintext WEB scanning traffic, then combining the TCP handshake feature data and HTTP feature data corresponding to each scanning TCP data stream to obtain a combined feature table; then calculating information summary based on the combined feature table, filtering and obtaining a corresponding first information summary value table; matching the second information summary value corresponding to the combined feature of the TCP data stream to be detected with the first information summary value table corresponding to each scanning TCP data stream, if the match is successful, the TCP data stream to be detected is determined to be scanning traffic, otherwise, the TCP data stream to be detected is determined to be regular traffic. The method solves the technical problems of insufficient timely response and low accuracy for plaintext WEB scanning detection, and achieves the technical effect of improving scanning detection efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of a process flow of a plaintext WEB scanning detection method provided by an embodiment of the present invention;

[0019] Figure 2 A flowchart of collecting WEB scanning traffic and extracting and preprocessing features from its TCP data stream in an embodiment of the present invention;

[0020] Figure 3 This is a flow chart of calculating the MD5 value of a feature combination in an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. Some embodiments of the present invention are described in detail below in conjunction with the drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0025] In reality, programs that perform various reconnaissance tasks are collectively referred to as scanners. Depending on the scanned object, the functions of scanners can be further divided into system scanning, WEB scanning, database scanning, etc. Among them, a large number of requests sent by WEB scanning will cause a surge in server traffic, which may cause service interruption, affecting normal website operation and user access; at the same time, a large number of scan requests may also generate a large number of error records in the server log, which may overwhelm the real problem log, making it difficult for administrators to quickly locate and solve the problem; in addition, attackers may use security vulnerabilities discovered during the scanning process to attack the server, thereby stealing sensitive information, destroying WEB applications and even operating systems, and seriously threatening the security of the server; finally, attackers may tamper with specific web pages by manipulating the database to embed web Trojan links to launch Trojan attacks.

[0026] Therefore, an effective real-time and accurate detection method is needed to perform plaintext WEB scanning for detection, so as to reduce the risk of WEB applications and operating systems being attacked by network attacks. In other words, the methods for detecting plaintext WEB scanning in the prior art have technical problems such as insufficient response time and low accuracy. Based on this, the embodiments of the present invention provide a plaintext WEB scanning detection method, device, electronic device and storage medium to solve the above technical problems.

[0027] To facilitate understanding of this embodiment, a plaintext WEB scanning detection method disclosed in an embodiment of the present invention is first described in detail. Figure 1 The flowchart of a plain text WEB scanning detection method shown in FIG. 1 is a flowchart of a plain text WEB scanning detection method, which can be executed by an electronic device and mainly includes the following steps S110 to S140:

[0028] S110: extracting TCP handshake feature data and HTTP feature data corresponding to each scanned TCP data stream from each scanned TCP data stream in the plaintext WEB scan traffic;

[0029] The plaintext WEB scanning traffic is collected by various WEB scanners; the plaintext WEB scanning traffic includes several scanning TCP data streams. That is, the plaintext WEB scanning traffic sent by various scanners is collected.

[0030] In one embodiment, S110 extracts TCP handshake feature data and HTTP feature data corresponding to each scanned TCP data stream from each scanned TCP data stream in the plaintext WEB scan traffic, including:

[0031] (S111) extracting features of each scanned TCP data stream according to the data packet type to obtain a TCP handshake feature and an HTTP feature corresponding to each scanned TCP data stream; each TCP handshake feature corresponding to the scanned TCP data stream corresponds to a feature number, and the feature number includes a first-class number, a second-class number, and a third-class number;

[0032] (S112) Byte sampling is performed on the TCP handshake feature and the HTTP feature respectively, and the byte sampling results are preprocessed to obtain TCP handshake feature data and HTTP feature data.

[0033] As a specific example, all TCP data streams are extracted from the scan traffic collected above, 8 features (TCP handshake feature data) and 9 features (HTTP feature data) are extracted from the TCP-SYN of each TCP data stream and each HTTP data packet, respectively, and the feature values ​​are preprocessed. The specific situation of the extracted features is shown in Table 1.

[0034] Table 1 - 2 sets of features extracted from TCP_SYN and HTTP packets of TCP data flow;

[0035] .

[0036] As a specific example, the first category number corresponds to features 01, 02, 04, 05, 06, and 07 in Table 1; the second category number corresponds to feature 03 in Table 1; and the third category number corresponds to feature 08 in Table 1.

[0037] It should be noted that each scanned TCP data stream may include two types of data packets, TCP-SYN and HTTP, and each type of data packet corresponds to TCP features, that is, TCP features are common to both types of data packets, i.e., HTTP features 01-08 refer to TCP handshake features of data packets such as HTTP, while the real HTTP features are HTTP_ 09- HTTP _14. Therefore, only the TCP handshake features of the two types of data packets are numbered and classified, and different preprocessing is performed based on the classification; HTTP features numbered 09-14 will not be preprocessed.

[0038] In this embodiment, the above preprocessing includes:

[0039] (S1121) recording the missing features after feature extraction as character 0;

[0040] (S1122) converting the result of byte-sampling of the feature corresponding to the category number into a decimal number;

[0041] That is, the byte values ​​of features 01, 02, 04, 05, 06, and 07 in Table 1 are converted into corresponding decimal character formats.

[0042] (S1123) discretizing the result of taking byte values ​​of the features corresponding to the second-category numbers according to a pre-defined rule;

[0043] That is, feature 03 in Table 1 (denoted as ) is converted into the corresponding decimal number and discretized into the corresponding string according to the following conditions:

[0044]

[0045] (S1124) dividing the result of byte-sampling the features corresponding to the three types of numbers into single bytes, and converting each byte after the division into a decimal number;

[0046] (S1125) The sum of the decimal numbers converted from the features corresponding to the three types of numbers is reassigned to the features corresponding to the three types of numbers in character form.

[0047] That is, the value of feature No. 08 in Table 1 is segmented into single bytes, each byte after segmentation is converted into a decimal number, and the sum of the decimal numbers is assigned to feature No. 08 in the form of characters.

[0048] S120: performing feature combination on the TCP handshake feature data and HTTP feature data corresponding to each scanned TCP data stream to obtain a combined feature table;

[0049] In one embodiment, S120 performs feature combination on the TCP handshake feature data and HTTP feature data corresponding to each scanned TCP data stream to obtain a combined feature table, including:

[0050] (S121) grouping the TCP handshake feature data and HTTP feature data corresponding to each scanned TCP data stream to obtain a first number of feature groups;

[0051] (S122) Determine all combinations of the first number of feature groups under a set length and generate a combination feature table; the set length is any positive integer in the range of 1 to the first number.

[0052] As a specific example, the preprocessed features are divided into 8 feature groups as shown in Table 2, and the features in groups 01 and 02 in Table 2 are sorted within the groups, and the sorting can be changed.

[0053] Table 2 - Feature group numbers, representative letters, and rankings;

[0054] .

[0055] Find all the combinations of the above 8 feature groups at various lengths (see Table 3). The length means the number of feature groups that appear in the combination. The order of the feature groups in each combination can be changed.

[0056] Table 3- All combinations of 8 feature groups;

[0057] .

[0058] S130: Calculate the information summary based on the combined feature table, and obtain a first information summary value table corresponding to each scanned TCP data stream after filtering;

[0059] In one embodiment, S130 performs information summary calculation based on the combined feature table, and obtains a first information summary value table corresponding to each scanned TCP data stream after filtering, including:

[0060] (S131) ​​according to the combination feature table, sequentially concatenating the values ​​of the corresponding features in each combination into a character string;

[0061] (S132) calculating the MD5 value of each string and generating a basic information summary table; the MD5 values ​​of the strings corresponding to the TCP handshake feature and the HTTP feature in the same scanned TCP data stream are in the same row of the basic information summary table;

[0062] (S133) Filtering the basic information summary table based on the preset parameter table to generate a first information summary value table corresponding to each scanned TCP data stream.

[0063] Among them, the preset parameter table includes: first variable parameters, filtering parameters, feature combination weight parameters, and matching threshold parameters.

[0064] As a specific example, the preset parameter table can set 6 parameters with reference to Table 4, wherein the first variable parameter is variable M (equal to the value of parameter num_pkt_http); the filtering parameters include: fingerprint selection parameter sw_http_only, detection mode parameter mod, threshold parameter num_fp_empt for simplified fingerprint table; feature combination weight parameter is perc_fp; and matching threshold parameter is perc_prd.

[0065] Table 4- 6 parameter types, value ranges, and default values;

[0066] .

[0067] Then the above (S133) filters the basic information summary table based on the preset parameter table to generate a first information summary value table corresponding to each scanned TCP data stream, including:

[0068] The basic information summary table is respectively subjected to column filtering and row filtering based on the filtering parameters to generate a first information summary value table corresponding to each scanned TCP data stream.

[0069] As a specific example, in combination with Table 3, the values ​​of the corresponding features in each combination are concatenated in sequence in a given order using specific symbols (the default is an English hyphen) to form a string, and the MD5 value of each string is calculated; wherein, the given order may refer to Table 2 if necessary, for example, MD5(HAE) = MD5(HTTP_02- HTTP_03-HTTP_08- HTTP_09-HTTP_11).

[0070] Taking each HTTP data packet and the TCP-SYN traffic packet belonging to the same TCP flow as a unit, all MD5 values ​​in Table 3 are calculated according to the above method, and the MD5 value calculated by each unit feature is saved as a row in Table 5, that is, each column in Table 5 corresponds to the MD5 value of various feature combinations in Table 3, and the duplicate rows in Table 5 are removed (a total of 255 columns, select 15 columns).

[0071] Table 5 - Filter out some MD5 from Table 5 based on the filtering conditions in both rows and columns; .

[0072] In Table 5, 0 in rows 2 to 7 represents MD5 cfcd208495d565ef66e7dff9f98764da; row 1 indicates that the MD5 to be calculated for each TCP data stream is consistent; row 2 indicates TCP-SYN of TCP data stream 1 and MD5 of HTTP1; row 3 indicates TCP-SYN of TCP data stream 1 and MD5 of HTTP 2; row 4 indicates TCP-SYN of TCP data stream 2 and MD5 of HTTP1; row 5 indicates TCP-SYN of TCP data stream 3 and MD5 of HTTP1; row 6 indicates TCP-SYN of TCP data stream 3 and MD5 of HTTP 2; row 7 indicates TCP-SYN of TCP data stream 3 and MD5 of HTTP 3. Row 8 in Table 5 indicates whether each column is retained or discarded; the last column indicates whether each row is retained or discarded. The length of the feature combination of the four columns retained is determined by mod=4. Since sw_http_only=1, the feature combination containing S and H should not be included, that is, num_fp=15 instead of 70 (see Table 3); the two discarded rows are determined by num_fp_empt. Since the number of MD5(0) in ACELTU is ≥ num_fp_empt(2), the fingerprints of the 3rd and 4th rows are discarded.

[0073] As a specific example, some MD5 values ​​in Table 5 are saved in Table 6, and the filtered MD5 values ​​are determined by the union of the following row and column filtering conditions:

[0074] The row filtering conditions include: if the fingerprint selection parameter sw_http_only in Table 4 is set to 0, then the values ​​of MD5(S), MD5(H), MD5(A), MD5(C), MD5(E), MD5(L), MD5(T), MD5(U) in each row in Table 5 are checked one by one; if the fingerprint selection parameter sw_http_only in Table 4 is set to 1, then the values ​​of MD5(A), MD5(C), MD5(E), MD5(L), MD5(T), MD5(U) in each row in Table 5 are checked one by one; if the number of MD5 values ​​in a row being checked that is equal to cfcd208495d565ef66e7dff9f98764da is not less than the value of the threshold parameter num_fp_empt of the simplified fingerprint table in Table 4, then the row should not be stored in the table 6.

[0075] The column filtering conditions include: checking the feature combination corresponding to each column MD5 value in Table 5 one by one, if the length of a feature combination is not equal to the value of the detection mode parameter mod in Table 4, the corresponding column should not be stored in the table 6;

[0076] If the fingerprint selection parameter sw_http_only in Table 4 is set to 1, the columns corresponding to the feature combination including feature groups 01 (S) and 02 (H) (ie, S and H) in Table 2 should not be stored in Table 6.

[0077] As a specific example, the above method further includes: saving some MD5 values ​​in Table 5 into Table 6, that is, scanning the first information summary value table corresponding to the TCP data stream, and recording the number of columns in Table 6 as num_fp.

[0078] In one embodiment, the method further comprises the following steps:

[0079] (S1301) extracting TCP-SYN handshake feature data and HTTP feature data corresponding to the TCP data flow to be detected based on the first variable parameter;

[0080] (S1302) determining a combination feature of TCP-SYN handshake feature data and HTTP feature data corresponding to the TCP data flow to be detected based on a feature combination weight parameter and the number of columns of the first information summary value table;

[0081] (S1303) Calculate a second information summary value corresponding to the combined feature.

[0082] That is, the TCP-SYN and the first M HTTP data packets in the TCP data stream to be detected are extracted, and the MD5 values ​​of some feature combinations appearing in Table 6, that is, the second information digest values ​​corresponding to the combined features, are calculated.

[0083] As a specific example, the MD5 values ​​of some feature combinations shown in Table 6 are calculated. The feature combinations whose MD5 values ​​are to be calculated are determined by the following two steps:

[0084] (1) The above , the perc_fp is defined in Table 4 (i.e., feature combination weight parameter), symbol It means round up;

[0085] (2) Randomly select the MD5 values ​​of the above num_fp columns from Table 6. The feature combination corresponding to the selected column is the feature combination of the MD5 value to be calculated.

[0086] Table 6 - Some MD5 values ​​extracted from Table 5;

[0087] .

[0088] S140: Match the second information summary value corresponding to the combined feature of the TCP data stream to be detected with the first information summary value table corresponding to each scanned TCP data stream. If the match is successful, the TCP data stream to be detected is determined to be scanned traffic; otherwise, the TCP data stream to be detected is determined to be regular traffic.

[0089] In one embodiment, S140 matches the second information summary value corresponding to the combined feature of the TCP data stream to be detected with the first information summary value table corresponding to each scanned TCP data stream, including:

[0090] (S141) comparing the second information summary value corresponding to the TCP data flow to be detected with the value of the corresponding combined feature in the first information summary value table, and determining that the combined features corresponding to the same values ​​are matching items;

[0091] (S142) determining the row index number of each matching item in the first information summary value table;

[0092] (S143) determining a target row index number and an occurrence frequency of the target row index number according to the plurality of row index numbers;

[0093] (S144) Based on the occurrence frequency of the target row index number and the matching threshold parameter, it is determined whether the TCP data flow to be detected is scanning traffic.

[0094] That is, the calculated MD5 value is matched with the corresponding MD5 value in Table 6. If the match is successful, the TCP data flow to be detected is determined to be scanning traffic, otherwise, it is determined to be regular traffic.

[0095] As a specific example, the calculated MD5 value is matched with the corresponding MD5 value in Table 6. The matching step is implemented in the following three steps:

[0096] (1) Match the calculated MD5 value with all MD5 values ​​of the corresponding feature combination in Table 6, and record the row index number of each matching item in Table 6;

[0097] (2) The number of occurrences of the highest frequency row index number is recorded as fp_freq_num, and the ratio of the number of occurrences of the highest frequency row index number to the number of columns randomly selected from Table 6 is recorded as fp_freq_perc;

[0098] in, .

[0099] (3) If the above fp_freq_perc>perc_prd (matching threshold parameter in Table 4), it is determined to be a match; otherwise, it is determined to be a mismatch.

[0100] The embodiment of the present invention provides a method for detecting plaintext WEB scanning. First, the TCP handshake feature data and HTTP feature data corresponding to each scanning TCP data stream are extracted from each scanning TCP data stream in the plaintext WEB scanning traffic. Then, the TCP handshake feature data and HTTP feature data corresponding to each scanning TCP data stream are combined to obtain a combined feature table. Then, information summary calculation is performed based on the combined feature table, and the corresponding first information summary value table is obtained after filtering. The second information summary value corresponding to the combined feature of the TCP data stream to be detected is matched with the first information summary value table corresponding to each scanning TCP data stream. If the match is successful, the TCP data stream to be detected is determined to be scanning traffic. Otherwise, the TCP data stream to be detected is determined to be regular traffic. The method solves the technical problems of insufficient response time and low accuracy for plaintext WEB scanning detection, and achieves the technical effect of improving scanning detection efficiency and accuracy.

[0101] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments. It should be understood that the embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0102] Figure 2 This is a flow chart of collecting WEB scanning traffic and extracting and preprocessing features from its TCP data stream in an embodiment of the present invention. The method includes:

[0103] 1. Collect plain text web scanning traffic from various scanners (scanner 1, scanner 2, scanner 3... scanner N);

[0104] 2. Extract all TCP data streams (TCP data stream 1, TCP data stream 2, ... TCP data stream N) from the scanned traffic, extract 8 and 9 features from the TCP-SYN of each TCP data stream and each HTTP data packet respectively (see Table 1), and preprocess the feature values; among them, each TCP data stream corresponds to a TCP-SYN packet and multiple HTTP packets (HTTP1, HTTP 2, ... HTTP N).

[0105] Specifically, the feature values ​​are preprocessed, including:

[0106] (1) Any missing feature is recorded as character 0;

[0107] (2) Convert the byte values ​​of features 01, 02, 04, 05, 06, and 07 in Table 1 into the corresponding decimal character format:

[0108] (3) After converting the byte value of feature 03 (denoted as x) in Table 1 into the corresponding decimal number, discretize it into the corresponding string according to the following conditions:

[0109]

[0110] (4) Split the value of feature 08 in Table 1 into single bytes, convert each byte after the splitting into decimal numbers, and assign the sum of the decimal numbers to feature 08 in character form;

[0111] At this point, the feature value that can be used to directly calculate the MD5 value of the feature combination is obtained.

[0112] Figure 3 The flowchart of calculating the MD5 value of the feature combination in an embodiment of the present invention (taking MD5(H_A_E) as an example), the method also includes:

[0113] 1. Divide the preprocessed features into 8 groups (see Table 2), and sort the features in groups 01 and 02 in Table 2 within the group. The sorting can be changed;

[0114] 2. Find all combinations of the eight feature groups at different lengths (see Table 3). The length refers to the number of feature groups in the combination. The order of feature groups in each combination can be changed. The values ​​of the corresponding features in each combination are concatenated in a given order (refer to Table 2 if necessary) using specific symbols (default is the English hyphen) to form a string, and calculate the MD5 value of each string.

[0115] 3. Take each HTTP packet and the TCP-SYN packet belonging to the same TCP flow as a unit, calculate all the MD5 values ​​in Table 3, and save the MD5 value calculated by each unit feature as a row in Table 5, that is, each column in Table 5 corresponds to the MD5 value of various feature combinations, and remove the duplicate rows in Table 5.

[0116] At this point, we have obtained a complete set of MD5 values ​​that may be used to determine the scanning of TCP data streams.

[0117] In an embodiment of the present invention, a schematic diagram of filtering out some MD5 values ​​from Table 5 and storing them in Table 6 according to the filtering conditions in both row and column aspects (Table 5 in the example has 6 rows of MD5 values, and the parameters are respectively set to mod=4, sw_http_only=1, num_fp_empt=2, and only the MD5 values ​​marked in yellow are stored in Table 6), the method further includes:

[0118] 1. Set 6 parameters according to Table 4;

[0119] 2. Save some MD5 values ​​in Table 5 to Table 6, and record the number of columns in Table 6 as num_fp;

[0120] Specifically, some MD5 values ​​in Table 5 are saved in Table 6, wherein the filtered MD5 values ​​are determined by the union of the following two aspects of filtering conditions:

[0121] (1) Row filtering: If the fingerprint selection parameter sw_http_only in Table 4 is set to 0, then the values ​​of MD5(S), MD5(H), MD5(A), MD5(C), MD5(E), MD5(L), MD5(T), and MD5(U) in each row of Table 5 are checked one by one; if the fingerprint selection parameter sw_http_only in Table 4 is set to 1, then the values ​​of MD5(A), MD5(C), MD5(E), MD5(L), MD5(T), and MD5(U) in each row of Table 5 are checked one by one; if the number of MD5 values ​​in a row being checked that is equal to cfcd208495d565ef66e7dff9f98764da is not less than the value of the threshold parameter num_fp_empt of the simplified fingerprint table in Table 4, then the row should not be stored in Table 6;

[0122] (2) Column filtering: Check the feature combination corresponding to each column of the MD5 value in Table 5 one by one: if the length of a feature combination is not equal to the value of the detection mode parameter mod in Table 4, the corresponding column should not be stored in Table 6; if the fingerprint selection parameter sw_http_only in Table 4 is set to 1, the column corresponding to the feature combination containing feature groups 01 (S) and 02 (H) (i.e., S and H) in Table 2 should not be stored in Table 6.

[0123] At this point, the MD5 value subset actually used to determine the scanned TCP data stream is obtained.

[0124] As a specific example, the MD5 value calculated according to the TCP data flow to be detected is matched with the corresponding MD5 value in Table 6 (Table 6 in the example has 4 rows of MD5 values, and the parameters are set to perc_fp = 0.5 and perc_prd = 0.5 respectively. The gray-marked values ​​in Table 6 are randomly selected MD5 values ​​to be compared). The method also includes: extracting TCP-SYN and the first M HTTP data packets in the TCP data flow to be detected (M=2 in Table 6, so only the first 2 HTTP data packets are processed), extracting the features of the data packets (8 features of the TCP-SYN data packet, 9 features of the data packet HTTP 1, and 9 features of the data packet HTTP 2) respectively and performing preprocessing, and then calculating the MD5 value of some feature combinations appearing in Table 6, matching the calculated MD5 value with the corresponding MD5 value in Table 6, and if the match is successful, determining that the TCP data flow to be detected is scanning traffic, otherwise, determining that it is regular traffic;

[0125] The variable M is equal to the value of the parameter num_pkt_http in Table 4.

[0126] The above calculations show the MD5 values ​​of some feature combinations in Table 6. In this embodiment, the feature combination to be calculated for the MD5 value is selected by the following two steps:

[0127] (1) Order , perc_fp is defined in Table 4, symbol means round up; for example, ;

[0128] (2) Randomly select the MD5 values ​​of num_fp columns from Table 6. The feature combination corresponding to the selected column is the feature combination of the MD5 value to be calculated.

[0129] In this embodiment, the calculated MD5 value is matched with the corresponding MD5 value in Table 6, specifically including:

[0130] (1) Match the calculated MD5 value with all MD5 values ​​of the corresponding feature combination in Table 6, and record the row index number of each matching item in Table 6 (see Table 7);

[0131] (2) The number of occurrences of the most frequent row index number is recorded as fp_freq_num, and the ratio of the number of occurrences of the most frequent row index number to the number of columns randomly selected from Table 6 is recorded as fp_freq_perc;

[0132] in, .

[0133] (3) If fp_freq_perc>perc_prd (see Table 4), it is determined to be a match; otherwise, it is determined to be a mismatch.

[0134] For example, as shown in Table 7 below, the highest frequency index 3 is hit 5 times, that is, fp_freq_num=5, so the TCP data flow to be detected is determined to be scanning traffic according to the following inequality:

[0135]

[0136] Table 7 - The calculated MD5 value matches all MD5 values ​​in the corresponding feature combination in Table 6; .

[0137] At this point, the determination of whether the TCP data stream to be detected is scanning traffic is completed based on the matching result of the MD5 value.

[0138] The above-mentioned plaintext WEB scanning detection method provided by the embodiment of the present invention has been deployed and successfully applied in corresponding products, and it can achieve extremely low (to zero) false alarm rate in various network environments. In particular, the method is highly efficient and constant and predictable, and will not become a bottleneck for the WEB application to provide services to the outside world in terms of performance. Practice has also proved that once the scanning action is successfully identified, most scanners will stop working immediately and will not be able to issue a scanning report (the reports issued by very few exceptional scanners contain only a very small amount of general information and do not contain vulnerability information). In summary, the plaintext WEB scanning detection method provided by the embodiment of the present invention truly reduces the risk of WEB applications and operating systems being attacked by networks due to the detection of their own vulnerabilities.

[0139] In addition, an embodiment of the present invention further provides a plaintext WEB scanning detection device, which includes: a feature data extraction module, which is used to extract TCP handshake feature data and HTTP feature data corresponding to each scanning TCP data stream from each scanning TCP data stream in the plaintext WEB scanning traffic; the plaintext WEB scanning traffic is collected and obtained by various WEB scanners; the plaintext WEB scanning traffic includes a plurality of scanning TCP data streams;

[0140] A feature combination module is used to perform feature combination on the TCP handshake feature data and HTTP feature data corresponding to each scanned TCP data stream to obtain a combined feature table;

[0141] An information summary calculation module, used for performing information summary calculation based on the combined feature table, and obtaining a first information summary value table corresponding to each scanned TCP data stream after filtering;

[0142] The matching module is used to match the second information summary value corresponding to the combined characteristics of the TCP data flow to be detected with the first information summary value table corresponding to each scanned TCP data flow. If the match is successful, the TCP data flow to be detected is determined to be scanned traffic; otherwise, the TCP data flow to be detected is determined to be regular traffic.

[0143] The plaintext WEB scanning detection device provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present application are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here. The plaintext WEB scanning detection device provided in the embodiment of the present application has the same technical features as the plaintext WEB scanning detection method provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0144] An embodiment of the present application further provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0145] Figure 4 A structural schematic diagram of an electronic device provided in an embodiment of the present application, the electronic device 400 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43, wherein the processor 40, the communication interface 43 and the memory 41 are connected via the bus 42; the processor 40 is used to execute an executable module stored in the memory 41, such as a computer program.

[0146] The memory 41 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0147] The bus 42 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0148] Among them, the memory 41 is used to store programs, and the processor 40 executes the program after receiving the execution instruction. The method executed by the process definition device disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0149] The processor 40 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 40. The above processor 40 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to execute, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 41, and the processor 40 reads the information in the memory 41 and completes the steps of the above method in combination with its hardware.

[0150] Corresponding to the above method, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above method.

[0151] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0152] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0153] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0154] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., various media that can store program codes.

[0155] It should be noted that similar numbers and letters represent similar items in the accompanying drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A plaintext WEB scanning detection method, characterized in that: include: Extracting TCP handshake feature data and HTTP feature data corresponding to each scanned TCP data stream from each scanned TCP data stream in the plaintext WEB scanned traffic; The plaintext WEB scanning traffic is collected and obtained through various WEB scanners; the plaintext WEB scanning traffic includes several scanning TCP data streams; Combining the TCP handshake feature data and the HTTP feature data corresponding to each of the scanned TCP data streams to obtain a combined feature table; Perform information summary calculation based on the combined feature table, and obtain a first information summary value table corresponding to each of the scanned TCP data flows after filtering; Matching the second information summary value corresponding to the combined feature of the TCP data stream to be detected with the first information summary value table corresponding to each of the scanned TCP data streams, if the match is successful, determining that the TCP data stream to be detected is scanned traffic, otherwise, determining that the TCP data stream to be detected is regular traffic; The TCP handshake feature data and HTTP feature data corresponding to each scanned TCP data stream in the plaintext WEB scan traffic are extracted respectively, including: Extract features from each of the scanned TCP data streams according to the data packet type to obtain TCP handshake features and HTTP features corresponding to each of the scanned TCP data streams; each of the TCP handshake features corresponding to the scanned TCP data streams corresponds to a feature number, and the feature number includes a first-class number, a second-class number, and a third-class number; Taking byte values ​​of the TCP handshake feature and the HTTP feature respectively, and preprocessing the results of the byte taking values ​​to obtain TCP handshake feature data and HTTP feature data; The pre-processing comprises: The missing features after feature extraction are recorded as character 0; Convert the result of taking the byte value of the feature corresponding to the category number into a decimal number; The results of byte-sampling the features corresponding to the second type of numbers are discretized according to a pre-defined rule; The result of taking byte values ​​of the features corresponding to the three types of numbers is segmented into single bytes, and each byte after segmentation is converted into a decimal number; The sum of the decimal numbers converted from the features corresponding to the three types of numbers is reassigned to the features corresponding to the three types of numbers in character form.

2. The plaintext WEB scanning detection method according to claim 1 is characterized in that: The TCP handshake feature data and the HTTP feature data corresponding to each of the scanned TCP data streams are respectively combined to obtain a combined feature table, including: Grouping the TCP handshake feature data and the HTTP feature data corresponding to each of the scanned TCP data streams to obtain a first number of feature groups; Determine all combinations of the first number of feature groups under a set length and generate a combination feature table; the set length is any positive integer in the range of 1 to the first number.

3. The plaintext WEB scanning detection method according to claim 2 is characterized in that: An information summary calculation is performed based on the combined feature table, and after filtering, a first information summary value table corresponding to each of the scanned TCP data flows is obtained, including: According to the combination feature table, the values ​​of the corresponding features in each combination are sequentially connected in series to form a character string; Calculate the MD5 value of each string to generate a basic information summary table; the MD5 values ​​of the strings corresponding to the TCP handshake features and HTTP features in the same scanned TCP data stream are in the same row of the basic information summary table; The basic information summary table is filtered based on a preset parameter table to generate a first information summary value table corresponding to each of the scanned TCP data flows.

4. The plaintext WEB scanning detection method according to claim 3 is characterized in that: The preset parameter table includes: first variable parameter, filtering parameter, feature combination weight parameter, matching threshold parameter; The basic information summary table is filtered based on a preset parameter table to generate a first information summary value table corresponding to each of the scanned TCP data streams, including: Based on the filtering parameters, column filtering and row filtering are performed on the basic information summary table respectively to generate a first information summary value table corresponding to each of the scanned TCP data flows.

5. The plaintext WEB scanning detection method according to claim 4 is characterized in that: The method further comprises: Based on the first variable parameter, extract TCP-SYN handshake feature data and HTTP feature data corresponding to the TCP data stream to be detected; Determine, based on the feature combination weight parameter and the number of columns of the first information summary value table, a combined feature of the TCP-SYN handshake feature data and the HTTP feature data corresponding to the TCP data stream to be detected; Calculate a second information summary value corresponding to the combined feature.

6. The plaintext WEB scanning detection method according to claim 4, characterized in that: Matching the second information summary value corresponding to the combined feature of the TCP data stream to be detected with the first information summary value table corresponding to each of the scanned TCP data streams includes: Compare the second information summary value corresponding to the TCP data flow to be detected with the value of the corresponding combination feature in the first information summary value table, and determine that the combination features corresponding to the same values ​​are matching items; Determine the row index number of each matching item in the first information summary value table; Determine a target row index number and an occurrence frequency of the target row index number according to the plurality of row index numbers; Based on the occurrence frequency of the target row index number and the matching threshold parameter, it is determined whether the TCP data flow to be detected is scanning traffic.

7. A plain text WEB scanning detection device, characterized in that: include: A feature data extraction module is used to extract TCP handshake feature data and HTTP feature data corresponding to each scanned TCP data stream from each scanned TCP data stream in the plaintext WEB scanning traffic; the plaintext WEB scanning traffic is collected and obtained by various WEB scanners; the plaintext WEB scanning traffic includes several scanned TCP data streams; A feature combination module, used for performing feature combination on the TCP handshake feature data and the HTTP feature data corresponding to each of the scanned TCP data streams to obtain a combined feature table; An information summary calculation module, used for performing information summary calculation based on the combined feature table, and obtaining a first information summary value table corresponding to each of the scanned TCP data flows after filtering; A matching module, used for matching the second information summary value corresponding to the combined feature of the TCP data flow to be detected with the first information summary value table corresponding to each of the scanned TCP data flows, and if the match is successful, determining that the TCP data flow to be detected is scanned traffic, otherwise, determining that the TCP data flow to be detected is regular traffic; Among them, the TCP handshake feature data and HTTP feature data corresponding to each scanned TCP data stream in the plaintext WEB scanning traffic are respectively extracted, including: extracting features of each scanned TCP data stream according to the data packet type to obtain the TCP handshake features and HTTP features corresponding to each scanned TCP data stream; each TCP handshake feature corresponding to the scanned TCP data stream corresponds to a feature number, and the feature number includes a first-class number, a second-class number and a third-class number; performing byte value extraction on the TCP handshake feature and the HTTP feature, and preprocessing the results of the byte value extraction to obtain the TCP handshake feature data and HTTP feature data; The preprocessing includes: recording the missing features after feature extraction as character 0; converting the results of byte-sampling of the features corresponding to the first category number into decimal numbers; discretizing the results of byte-sampling of the features corresponding to the second category number according to pre-established rules; segmenting the results of byte-sampling of the features corresponding to the third category number into units of single bytes, and converting each byte after segmentation into decimal numbers; and reassigning the sum of the decimal numbers converted from the features corresponding to the three category numbers to the features corresponding to the three category numbers in character form.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method according to any one of claims 1 to 6.

Images