Scanner identification method, device, electronic device and storage medium

Abstract

An embodiment of the present invention provides a scanner identification method, device, electronic device and storage medium related to the field of network security technology. The method includes: obtaining multiple original access log data, and filtering out attack log data from the multiple original access log data. Based on text similarity, it is judged whether the attack log data contains scanning behavior. If so, access feature information is constructed based on the attack log data. The access feature information is identified based on a scanner identification model, and the access source address corresponding to the access feature information identified as a scanner is shielded. The scanner identification method provided by the present invention can effectively enhance the scanner identification capability and improve the accuracy of scanner identification.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a scanner identification method, device, electronic device and storage medium. Background Art

[0002] With the widespread availability of easy-to-use network scanners, network attacks using them have become easier. For example, hackers can use scanners to scan corporate and government websites and discover vulnerabilities. Therefore, it is crucial to identify malicious scanners during network access to protect website security.

[0003] The inventors have found that most existing scanner recognition methods are based on rule matching based on feature keywords to identify the scanner, which has weak recognition capabilities and low accuracy. Summary of the Invention

[0004] The objects of the present invention include, for example, providing a scanner recognition method, device, electronic device and storage medium, which can at least partially solve the above-mentioned technical problems.

[0005] The embodiments of the present invention can be implemented as follows:

[0006] In a first aspect, an embodiment of the present invention provides a scanner recognition method, the method comprising:

[0007] Acquire multiple original access log data, and filter out attack log data from the multiple original access log data;

[0008] Determining whether there is scanning behavior in the attack log data based on text similarity;

[0009] If so, constructing access feature information based on the attack log data;

[0010] The access feature information is identified based on a scanner identification model, and the access source address corresponding to the access feature information identified as a scanner is masked.

[0011] Optionally, the attack log data includes a first scanning data packet, and determining whether the attack log data contains scanning behavior based on text similarity includes:

[0012] Obtaining the first scanning data packet;

[0013] Based on text similarity, the first scan data packet is matched with the second scan data packet in a preset scan database, and the first scan data packet with successful matching is marked as a similar scan data packet;

[0014] Determining whether the ratio of the similar scan data packets to the first scan data packets is greater than a preset threshold;

[0015] If so, it is determined that the attack log data contains the scanning behavior.

[0016] Optionally, the method further includes:

[0017] The preset scanning database is updated based on the similar scanning data packets corresponding to the access feature information identified as the scanner.

[0018] Optionally, the updating of the preset scanning database based on the similar scanning data packet corresponding to the access feature information identified as the scanner includes:

[0019] Acquire a similar scan data packet corresponding to the access feature information identified as the scanner;

[0020] Calculating a similarity coefficient between the similar scan data packet and the second scan data packet;

[0021] Marking the similar scan data packets whose similarity coefficient is lower than a preset similarity coefficient threshold as potential second scan data packets;

[0022] Performing text clustering on the potential second scan data packet to obtain a cluster;

[0023] Determining whether the log data in the cluster meets a preset update condition;

[0024] If so, the similar scan data packets corresponding to the clusters that meet the preset update condition are added to the preset scan database to complete the update of the preset scan database.

[0025] Optionally, constructing access feature information based on the attack log data includes:

[0026] Obtaining an access log of attack log data determined to contain the scanning behavior;

[0027] Access feature information is constructed based on the access log.

[0028] Optionally, the access characteristic information includes one or more of the number of website visits, the number of website visits, the proportion of error status codes, the number of similar scan data packets, and the access duration.

[0029] Optionally, the method further comprises the step of constructing the scanner recognition model, comprising:

[0030] Get sample scanner and normal access log data;

[0031] Based on the fingerprint of the sample scanner, filtering out the sample access source address of the sample scanner;

[0032] Marking the access source address in the normal access log data as a normal access source address;

[0033] Based on a binary classification model, the sample access source address and the normal access source address are used to perform model training to obtain the scanner recognition model.

[0034] In a second aspect, an embodiment of the present invention provides a scanner recognition device, the scanner recognition device comprising:

[0035] an original access log data screening unit, configured to obtain a plurality of original access log data and screen out attack log data from the plurality of original access log data;

[0036] A scanning behavior judgment unit, configured to judge whether the attack log data contains scanning behavior based on text similarity;

[0037] an access feature information constructing unit, configured to construct access feature information based on the attack log data when the attack log data contains the scanning behavior;

[0038] The scanner identification unit is used to identify the access feature information based on the scanner identification model, and mask the access source address corresponding to the access feature information identified as the scanner.

[0039] In a third aspect, an embodiment of the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the program.

[0040] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a computer program, and when the computer program is executed, the server where the computer-readable storage medium is located is controlled to implement the steps of any of the above methods.

[0041] The beneficial effects of the embodiments of the present invention include, for example:

[0042] By filtering raw access log data, we obtain attack log data. We then determine whether this filtered attack log data contains scanning activity. For attack log data that does contain scanning activity, we construct access signature information. This access signature information is then fed into a scanner recognition model to ultimately determine if the attack is a scanner. By combining multiple screening and judgment methods and employing a trained model for identification, we achieve more accurate identification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 An architecture diagram of an electronic device provided by an embodiment of the present invention;

[0045] Figure 2 A flowchart of a scanner recognition method provided by an embodiment of the present invention;

[0046] Figure 3 A flowchart of the steps for training a scanner recognition model provided by an embodiment of the present invention;

[0047] Figure 4 This is an architectural diagram of a scanner recognition device provided by an embodiment of the present invention.

[0048] Icon: 100 - electronic device; 110 - memory; 120 - processor; 130 - communication module; 300 - scanner identification device; 301 - original access log data screening unit; 302 - scanning behavior judgment unit; 303 - access feature information construction unit; 304 - scanner identification unit. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0050] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0051] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0052] In addition, the terms "first", "second", etc., if used, are merely used to distinguish and describe, and should not be understood as indicating or implying relative importance.

[0053] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0054] In the prior art, there are generally the following ways to identify a scanner:

[0055] 1. Mark non-business ports. When an IP (Internet Protocol Address) accesses this type of port, it is marked as a suspicious IP. When the number of suspicious IPs accessing non-business ports exceeds the set threshold, the IP is considered a scanner.

[0056] The above method uses the method of marking the website port to identify the scanner, which has a good identification effect, but it requires sufficient familiarity with the website and port marking work in advance. When a new website appears or there are many websites, it requires more manpower costs.

[0057] 2. Abstract the scanner's behavioral characteristics into a conditional transfer process, and then identify the scanner based on a finite state machine.

[0058] The above scheme is based on an abstract transfer matrix for scanner recognition, which requires a good understanding of known scanner behaviors in advance and can better identify scanners with known information, but its recognition ability for new scanners is weak.

[0059] 3. Extract access keywords from the historical URL information of the user's visit to the website, calculate the anomaly score of the access keyword, and then perform scanner recognition based on the anomaly score.

[0060] The above method constructs a scanner for identifying website keywords, but as the number of website visits increases, the number of website keywords will also increase to a certain extent, and it will be difficult to store and calculate the efficiency of keywords.

[0061] Based on the above situation, the embodiments of this specification provide a scanner recognition method, device, electronic device and storage medium, which can effectively alleviate the above technical problems.

[0062] Please refer to Figure 1 , is a block diagram of an electronic device 100 provided herein. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, processor 120, and communication module 130 are electrically connected to each other, directly or indirectly, to enable data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.

[0063] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0064] The processor 120 is used to read / write data or programs stored in the memory and execute corresponding functions.

[0065] The communication module 130 is used to establish a communication connection between the server and other communication terminals through the network, and to send and receive data through the network.

[0066] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0067] Correspondingly, the embodiment of this specification provides a scanner recognition method, which can be applied to the electronic device 100, and the method includes the following steps: Figure 2 The following steps are shown:

[0068] Step S110: Acquire multiple original access log data, and filter out attack log data from the multiple original access log data.

[0069] Step S120: determining whether the attack log data contains any scanning behavior based on the text similarity.

[0070] Step S130: If yes, construct access feature information based on the attack log data.

[0071] Step S140: Identify the access feature information based on the scanner identification model, and mask the access source address corresponding to the access feature information identified as the scanner.

[0072] First, step S110 is performed to obtain a plurality of original access log data, and then filter out attack log data from the plurality of original access log data.

[0073] The original access log data may be the initial access log of a website visit. After obtaining the original access log data, the original access log data may be filtered through a firewall, etc., and abnormal access behaviors in the original access log data may be filtered out as attack log data.

[0074] Firewalls are devices that identify and filter user access traffic based on rules and other methods. They allow normal access behavior to pass through while blocking abnormal access behavior. Therefore, log data containing abnormal access behavior in the original access log data can be treated as attack log data.

[0075] Execute step S120: determine whether the attack log data contains scanning behavior based on the text similarity.

[0076] After filtering out the attack log data, the attack log data can be parsed to obtain various text information in the attack log data. Then, the text information in the attack log data is compared with the preset control text information to determine the text similarity between the text information in the attack log data and the control text information. The attack log data corresponding to the text information with a text similarity higher than a certain threshold is determined to contain scanning behavior.

[0077] Optionally, the attack log data includes a first scanning data packet, and determining whether the attack log data contains scanning behavior based on text similarity includes:

[0078] The first scanning data packet is obtained.

[0079] Based on text similarity, the first scan data packet is matched with a second scan data packet in a preset scan database, and the first scan data packet that successfully matches is marked as a similar scan data packet.

[0080] It is determined whether the ratio of the similar scan data packets to the first scan data packets is greater than a preset threshold.

[0081] If so, it is determined that the attack log data contains the scanning behavior.

[0082] Scanning data packets are data packets sent by the scanner to the website when scanning for vulnerabilities in the website (such as adding " / cache / backup / " after the URL to detect the vulnerability of TxExam sensitive information leakage).

[0083] The first scanning data packet may be a scanning data packet in the attack log data, and the second scanning data packet may be a scanning data packet in a preset scanning database. The preset scanning database may be a database storing identified scanning data packets.

[0084] After obtaining the first scan packet from the attack log data, a text similarity match can be performed against the second packet in the pre-set scan database. For example, the text similarity of the accessed URL, cookie, User_Agent, and other data can be matched. A match is considered successful if the similarity exceeds a certain threshold. The first scan packet that successfully matches is marked as a similar scan packet. A determination is then made as to whether the ratio of similar scan packets to all first scan packets exceeds a pre-set threshold. If so, scanning activity is determined to have occurred in the attack log data.

[0085] Execute step S130, and if so, construct access feature information based on the attack log data.

[0086] The attack log data determined to contain scanning behavior in step S120 is used to construct a behavior feature vector (ie, access feature information) to provide input data for model recognition in step S140.

[0087] Optionally, constructing access feature information based on the attack log data includes:

[0088] Acquire access logs of attack log data that are determined to contain the scanning behavior, and construct access feature information based on the access logs.

[0089] The access log of the attack log data can be data such as IP, user agent, visited website domain name, visited URL link, visited URL status code, etc. Based on this data, multiple access feature information can be constructed, such as access feature information of the number of website visits based on the visited website domain name.

[0090] As an optional embodiment, the access feature information includes one or more of the number of website visits, the number of website visits, the proportion of error status codes, the number of similar scan data packets, and the access duration.

[0091] Among them, the number of website visits can be the total number of times the access source visits the website on that day; the number of website visits can be the number of domain names accessed by the access source after deduplication; the error status code ratio can be the number of error status codes (status codes 400 and above) / total visit PV; the number of similar scan data packets can be the amount of data in the log data that is similar to the scan data feature library; the visit duration can be the duration between the end time of the website visit on that day and the start time of the website visit.

[0092] Step S140 is executed to identify the access feature information based on the scanner identification model, and mask the access source address corresponding to the access feature information identified as the scanner.

[0093] The access feature information constructed in step S130 is input into the trained scanner recognition model for scanner recognition, and the access source address corresponding to the access feature information of the scanner as the recognition result is shielded to achieve the effect of website protection.

[0094] Optionally, the method further comprises the step of constructing the scanner recognition model, including: Figure 3 The following substeps are shown:

[0095] Sub-step S210: Obtain a sample scanner and normal access log data.

[0096] Sub-step S220: Based on the fingerprint of the sample scanner, filter out the sample access source address of the sample scanner.

[0097] Sub-step S230: Mark the access source address in the normal access log data as a normal access source address.

[0098] Sub-step S240: Based on a binary classification model, the sample access source address and the normal access source address are used to perform model training to obtain the scanner recognition model.

[0099] The sample scanner can be a scanner already stored in the system, and the normal access log data can be the access data of normal website users. Since some open source scanners have corresponding feature information, such as the sqlmap scanner, its access URL or User_Agent contains "sqlmap". Therefore, the relevant identification features of the sample scanner can also be used as the sample scanner fingerprint of the sample scanner. At the same time, the access source address in the normal access log data is marked as the normal access source address. The sample access source address and the normal access source address are then input into a binary classification model for model training, and the model obtained after the training is completed is used as the scanner recognition model.

[0100] Optionally, the method further includes: updating the preset scanning database based on similar scanning data packets corresponding to the access feature information identified as the scanner.

[0101] After the scanner recognition model recognizes the scanner, the recognition result can be used as new scanner data to update the preset scan database, thereby enriching the preset scan database and further improving the recognition capability of the new scanner.

[0102] As an optional embodiment, updating the preset scanning database based on the similar scanning data packet corresponding to the access feature information identified as the scanner includes:

[0103] Acquire a similar scan data packet corresponding to the access feature information identified as the scanner;

[0104] Calculating a similarity coefficient between the similar scan data packet and the second scan data packet;

[0105] Marking the similar scan data packets whose similarity coefficient is lower than a preset similarity coefficient threshold as potential second scan data packets;

[0106] Performing text clustering on the potential second scan data packet to obtain clusters;

[0107] Determining whether the log data in the cluster meets a preset update condition;

[0108] If so, the similar scan data packets corresponding to the clusters that meet the preset update condition are added to the preset scan database to complete the update of the preset scan database.

[0109] First, similar scan packets corresponding to access signature information identified as a scanner by the scanner recognition model are obtained. A similarity coefficient, such as angle cosine similarity, is calculated between the similar scan packets and the second scan packets. Similar scan packets with similarity coefficients below a preset similarity coefficient threshold are then marked as potential second scan packets. Potential second scan packets are thus first scan packets that may be added to the preset scan database.

[0110] Then, text clustering (e.g., DBSACAN) is performed on the potential second scan data packets to obtain clusters. It is then determined whether the log data in the clusters meet a preset update condition. If so, similar scan data packets corresponding to the clusters that meet the preset update condition are added to the preset scan database, completing the update of the preset scan database. The preset update condition can be manually set for different scan data packets.

[0111] Based on the same inventive concept, Figure 4 As shown, the embodiment of the present invention provides a scanner recognition device 300, including:

[0112] The original access log data screening unit 301 is configured to obtain a plurality of original access log data and screen out attack log data from the plurality of original access log data.

[0113] The scanning behavior determination unit 302 is configured to determine whether the attack log data contains scanning behavior based on text similarity.

[0114] The access feature information constructing unit 303 is configured to construct access feature information based on the attack log data when the attack log data contains the scanning behavior.

[0115] The scanner identification unit 304 is configured to identify the access feature information based on a scanner identification model, and mask the access source address corresponding to the access feature information identified as a scanner.

[0116] Regarding the above-mentioned scanner recognition device 300, the specific functions of each unit therein have been described in detail in the embodiment of the scanner recognition method provided in this specification, and will not be elaborated here.

[0117] Based on the same inventive concept, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any method of the previous text scanner recognition method when executed by a processor.

[0118] The present invention has at least the following beneficial effects:

[0119] By filtering raw access log data, we obtain attack log data. We then determine whether this filtered attack log data contains scanning activity. For attack log data that does contain scanning activity, we construct access signature information. This access signature information is then fed into a scanner recognition model to ultimately determine if the attack is a scanner. By combining multiple screening and judgment methods and employing a trained model for identification, we achieve more accurate identification results.

[0120] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0121] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0122] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0123] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A scanner recognition method, characterized in that: The method comprises: Acquire multiple original access log data, and filter out attack log data from the multiple original access log data, wherein the attack log data includes a first scanning data packet; Determining whether the attack log data contains scanning behavior based on text similarity includes: obtaining the first scanning data packet; matching the first scanning data packet with a second scanning data packet in a preset scanning database based on text similarity, and marking the first scanning data packet that successfully matches as a similar scanning data packet; determining whether the proportion of the similar scanning data packets to the first scanning data packet is greater than a preset threshold; if so, determining that the attack log data contains the scanning behavior; If so, constructing access feature information based on the attack log data; Identifying the access feature information based on a scanner identification model, and shielding the access source address corresponding to the access feature information identified as a scanner; Based on the similar scanning data packets corresponding to the access feature information identified as a scanner, the preset scanning database is updated, including: obtaining the similar scanning data packets corresponding to the access feature information identified as a scanner; calculating the similarity coefficient between the similar scanning data packets and the second scanning data packets; marking the similar scanning data packets whose similarity coefficient is lower than a preset similarity coefficient threshold as potential second scanning data packets; performing text clustering on the potential second scanning data packets to obtain cluster clusters; judging whether the log data in the cluster clusters meets the preset update conditions; if so, adding the similar scanning data packets corresponding to the cluster clusters that meet the preset update conditions to the preset scanning database to complete the update of the preset scanning database.

2. The scanner recognition method according to claim 1, wherein: The constructing access feature information based on the attack log data includes: Obtaining an access log of attack log data determined to contain the scanning behavior; Access feature information is constructed based on the access log.

3. The scanner recognition method according to claim 2, wherein: The access characteristic information includes one or more of the number of website visits, the number of website visits, the proportion of error status codes, the number of similar scan data packets, and the access duration.

4. The scanner recognition method according to claim 1, wherein: The method further comprises the step of constructing the scanner recognition model, comprising: Get sample scanner and normal access log data; Based on the fingerprint of the sample scanner, filtering out the sample access source address of the sample scanner; Marking the access source address in the normal access log data as a normal access source address; Based on a binary classification model, the sample access source address and the normal access source address are used to perform model training to obtain the scanner recognition model.

5. A scanner recognition device, characterized in that: The scanner recognition device comprises: an original access log data screening unit, configured to obtain a plurality of original access log data and screen out attack log data from the plurality of original access log data, wherein the attack log data includes a first scanning data packet; a scanning behavior judgment unit, configured to judge whether the attack log data contains scanning behavior based on text similarity, comprising: obtaining the first scanning data packet; matching the first scanning data packet with a second scanning data packet in a preset scanning database based on text similarity, and marking the first scanning data packet that successfully matches as a similar scanning data packet; judging whether a proportion of the similar scanning data packets to the first scanning data packet is greater than a preset threshold; and if so, judging that the attack log data contains the scanning behavior; an access feature information constructing unit, configured to construct access feature information based on the attack log data when the attack log data contains the scanning behavior; A scanner identification unit is used to identify the access feature information based on a scanner identification model, and mask the access source address corresponding to the access feature information identified as a scanner; based on the similar scanning data packet corresponding to the access feature information identified as a scanner, the preset scanning database is updated, including: obtaining the similar scanning data packet corresponding to the access feature information identified as a scanner; calculating the similarity coefficient between the similar scanning data packet and the second scanning data packet; marking the similar scanning data packet whose similarity coefficient is lower than a preset similarity coefficient threshold as a potential second scanning data packet; performing text clustering on the potential second scanning data packet to obtain a cluster cluster; judging whether the log data in the cluster cluster meets the preset update condition; if so, adding the similar scanning data packet corresponding to the cluster cluster that meets the preset update condition to the preset scanning database to complete the update of the preset scanning database.

6. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 4 when executing the program.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a computer program, and when the computer program is executed, the server where the computer-readable storage medium is located is controlled to implement the steps of the method according to any one of claims 1 to 4.

Images