Vulnerability scanning and attack detection methods, model training methods and devices

Abstract

This application provides a vulnerability scanning attack detection method, model training method, and apparatus. The method includes: acquiring access logs and extracting key field information from the access logs; generating feature vectors based on the key field information; inputting the feature vectors into a vulnerability scanning attack detection model to obtain the analysis results output by the vulnerability scanning attack detection model; wherein, the vulnerability scanning attack detection model is obtained by pre-generating corresponding training feature vectors based on training logs, and training the model using the training feature vectors and labels used to characterize whether the training source IP address corresponding to the training logs exhibits vulnerability scanning attack behavior; the analysis results are used to characterize whether the behavior corresponding to the access logs is a vulnerability scanning attack behavior. This application improves the accuracy of vulnerability scanning attack behavior identification by analyzing the feature vectors of access logs using a machine learning model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, in particular to a vulnerability scanning attack detection method and device and a model training method. BACKGROUND

[0002] During the process of writing a website program, a staff member may cause a vulnerability due to incomplete consideration. In order to attack the website, an illegal person will perform vulnerability scanning on the program of the website to find the vulnerability existing in the website.

[0003] In order to improve the security of the website, a rule matching method is currently used to find whether a vulnerability scanning attack behavior exists, that is, a detection rule is set in advance, and the detection rule is used to match the generated log, if the matching is successful, it is indicated that the behavior corresponding to the log is a vulnerability scanning attack behavior. Since the behavior of the illegal person in vulnerability scanning is variable, the rule matching method has the problem of low detection accuracy. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a vulnerability scanning attack detection method and device and a model training method to improve the accuracy of identifying a vulnerability scanning attack behavior.

[0005] In a first aspect, the embodiments of the present application provide a vulnerability scanning attack detection method, comprising:

[0006] obtaining an access log and extracting key field information from the access log;

[0007] generating a feature vector according to the key field information;

[0008] inputting the feature vector into a vulnerability scanning attack discovery model to obtain an analysis result output by the vulnerability scanning attack discovery model; wherein the vulnerability scanning attack discovery model is obtained by pre-generating a corresponding training feature vector according to a training log and performing model training using the training feature vector and a label used to represent whether a training source IP address corresponding to the training log exists a vulnerability scanning attack behavior; and the analysis result is used to represent whether a behavior corresponding to the access log is a vulnerability scanning attack behavior.

[0009] The embodiments of the present application extract key field information from an access log, generate a feature vector according to the key field information, input the feature vector into a machine learning model, and make the machine learning model analyze the feature vector of the access log, thereby improving the accuracy of identifying a vulnerability scanning attack behavior.

[0010] In any embodiment, the key field information includes an access time and a source IP address; and generating a feature vector according to the key field information comprises:

[0011] According to the access time and the source IP address, the key field information is stored into a corresponding set of time windows;

[0012] Feature extraction is performed on the key field information in the set to obtain a feature vector.

[0013] Embodiments of the present application generate a feature vector corresponding to the key field information in a time window by using the time window, so that the feature vector corresponding to the time window can be analyzed by using a vulnerability scanning attack discovery model to discover vulnerability scanning attack behaviors in a timely manner.

[0014] In any embodiment, the feature vector is input into the vulnerability scanning attack discovery model, including:

[0015] The feature vector corresponding to the time window is input into the vulnerability scanning attack discovery model, and if the analysis result output by the vulnerability scanning attack discovery model indicates that the behavior corresponding to the time window is not a vulnerability scanning attack behavior, a new feature vector is obtained according to the feature vector corresponding to the time window and the feature vector of a next time window corresponding to the source IP address;

[0016] The new feature vector is input into the vulnerability scanning attack discovery model.

[0017] In embodiments of the present application, the time window corresponding to the feature vector input into the vulnerability scanning attack discovery model becomes larger, so that vulnerability scanning attack behaviors can be discovered in a timely manner, and low-frequency and time-divided vulnerability scanning attack behaviors can be identified.

[0018] In any embodiment, the features in the feature vector include at least one of the following: the number of access times of the source IP address, the number of access URLs, the number of access ports, the number of different payload data, the cumulative access duration, the average length of the payload data, the average length of the accessed URL, the number of successful state code returns, the number of failed state code returns, the number of same data and same URLs, the number of same data and same domain names, and the number of sensitive words in the payload data.

[0019] In embodiments of the present application, the features are determined through a large number of research experiments, so that the specific content contained in the feature vector is determined, and the accuracy of access log identification is improved.

[0020] In any embodiment, the key field information further includes at least one of the following: a returned state code, an access duration, the length of an access header, the number of bytes of an access packet, a request method, a target port, an accessed URL, an accessed domain name, a destination IP address, and payload data sent during access.

[0021] Since the key field information is the data basis of the feature vector, it is crucial to determine the specific information contained in the key field information. The embodiments of the present application provide data basis for improving the accuracy of vulnerability scanning attack behavior by extracting the key field information from the access log.

[0022] In any embodiment, after obtaining the analysis result output by the vulnerability scanning attack discovery model, the method further comprises:

[0023] If the analysis result represents that the behavior corresponding to the access log is a vulnerability scanning attack behavior, the source IP address corresponding to the access log is banned.

[0024] In the embodiments of the present application, when it is determined that the behavior corresponding to the access log is a vulnerability scanning attack behavior, the corresponding source IP address is banned, thereby improving the security of the website.

[0025] In any embodiment, the method further comprises:

[0026] Obtaining a training sample, the training sample comprising training logs corresponding to a plurality of training source IP addresses and labels corresponding to the training source IP addresses; wherein the labels are used to represent whether the training source IP addresses have vulnerability scanning attack behaviors;

[0027] Extracting keywords from the training logs to obtain training keywords;

[0028] Generating training feature vectors of the training source IP addresses according to the training keywords corresponding to each training source IP address;

[0029] Training the to-be-trained model using the training feature vectors and the corresponding labels to obtain a vulnerability scanning attack discovery model.

[0030] The embodiments of the present application obtain a vulnerability scanning attack discovery model through model training, and improve the accuracy of access log recognition using the vulnerability scanning attack discovery model.

[0031] In a second aspect, the embodiments of the present application provide a vulnerability scanning attack discovery model training method, comprising:

[0032] Obtaining a training sample, the training sample comprising training logs corresponding to a plurality of training source IP addresses and labels corresponding to the training source IP addresses; wherein the labels are used to represent whether the training source IP addresses have vulnerability scanning attack behaviors;

[0033] Extracting keywords from the training logs to obtain training keywords;

[0034] Generating training feature vectors of the training source IP addresses according to the training keywords corresponding to each training source IP address;

[0035] The model to be trained is trained by using the training feature vector and the corresponding label, and a vulnerability scanning attack discovery model is obtained.

[0036] The training keyword is extracted from the training log, and the training feature vector is generated according to the training keyword. The model to be trained is trained by using the generated training feature vector and the corresponding training source IP address, and the obtained vulnerability scanning attack discovery model can improve the accuracy of detecting whether the source IP address has a vulnerability scanning attack behavior.

[0037] In any embodiment, the features in the training feature vector include at least one of the following: the number of accesses corresponding to the training source IP address in a preset time period, the number of accessed URLs, the number of accessed ports, the number of different payload data, the cumulative access time, the average length of the payload data, the average length of the accessed URLs, the number of successful state code returns, the number of failed state code returns, the number of same data and same URLs, the number of same data and same domain names, and the number of sensitive words in the payload data.

[0038] In the embodiment of the application, the features used by the training model are determined through a large number of research experiments, thereby determining the specific content contained in the training feature vector, and improving the performance of the vulnerability scanning attack discovery model.

[0039] In a third aspect, an embodiment of the present application provides a vulnerability scanning attack detection device, comprising:

[0040] A log acquisition module is configured to acquire an access log and extract key field information from the access log.

[0041] A feature extraction module is configured to generate a feature vector according to the key field information.

[0042] A detection module is configured to input the feature vector into a vulnerability scanning attack discovery model, and obtain an analysis result output by the vulnerability scanning attack discovery model. The analysis result is used to represent whether the behavior corresponding to the access log is a vulnerability scanning attack behavior.

[0043] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory and a bus, wherein,

[0044] The processor and the memory complete mutual communication through the bus.

[0045] The memory stores program instructions executable by the processor, and the processor calling the program instructions can execute the method of the first aspect or the second aspect.

[0046] In a fifth aspect, an embodiment of the present application provides a non-transitory computer readable storage medium, comprising:

[0047] The non-transitory computer readable storage medium stores computer instructions that cause the computer to perform the method of the first aspect or the second aspect.

[0048] Other features and advantages of the present application will be set forth in the following description, and in part will be apparent from the description, or can be learned by practice of the application. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0050] Figure 1 A vulnerability scanning attack detection method flow diagram is provided for the embodiments of the present application;

[0051] Figure 2 A model training method flow diagram is provided for the embodiments of the present application;

[0052] Figure 3 An interaction diagram of the model and the server is provided for the embodiments of the present application;

[0053] Figure 4 A vulnerability scanning attack detection device structure diagram is provided for the embodiments of the present application;

[0054] Figure 5 An electronic device physical structure diagram is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0055] The embodiments of the technical solutions of the present application will be described in detail below with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terms used herein are only for the purpose of describing specific embodiments of the present application, and are not intended to limit the present application; the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above description of drawings are intended to cover non-exclusive inclusion.

[0057] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise explicitly specified and limited.

[0058] Reference herein to "embodiments" means that the particular features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily a separate or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0059] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0060] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two), and similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0061] In the description of the embodiments of the present application, the technical terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the embodiments of the present application.

[0062] In the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the technical terms "mounting", "connection", "connection", "fixing" and the like should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0063] Since the code writer may cause a vulnerability in the website program due to carelessness when writing the website program, in order to patch the vulnerability, the website program maintainer may perform vulnerability scanning on the website program to patch the vulnerability after the vulnerability is found. Meanwhile, the illegal person may also perform vulnerability scanning to attack the vulnerability after the vulnerability is found. The application refers to this behavior as vulnerability scanning attack. However, vulnerability scanning may not be able to find the vulnerability, and therefore, the illegal person may perform vulnerability scanning by sending multiple access requests to the server. With the deeper understanding of the vulnerability through the result returned by the illegal person through vulnerability scanning, the illegal person may find more vulnerabilities in the website program.

[0064] Currently, a rule matching method is usually used to find vulnerability scanning from illegal persons, for example, the access path commonly used in vulnerability scanning and the common attack load may be matched, and once the matching is successful, it is determined that it belongs to illegal behavior. This way needs the staff to set the matching rule in advance according to experience. However, the illegal person may change the behavior habit of vulnerability scanning in order to escape the matching rule, for example, the vulnerability scanning is performed in a low frequency and time-sharing manner. Therefore, the accuracy of the rule matching method for detecting vulnerability scanning attack is low.

[0065] In order to improve the accuracy of vulnerability scanning attack detection, the embodiment of the application provides a vulnerability scanning attack detection method and device based on machine learning, an electronic device and a storage medium. Before introducing the specific embodiments of the application, the related concepts involved in the application are explained and described:

[0066] nginx: nginx (engine x) is a high-performance HTTP and reverse proxy web server, and also provides IMAP / POP3 / SMTP service. Nginx is a lightweight web server / reverse proxy server and email (IMAP / POP3) proxy server, which has the characteristics of small memory occupation and strong concurrency capability. In fact, the concurrency capability of nginx is better than that of the same type of web server. The logs generated on nginx include access logs and error logs; among them, the user region source, jump source, terminal used, and certain URL access amount and other related information can be obtained through the access logs; through the error logs, the performance bottleneck of a certain service or server of the system can be obtained.

[0067] Web vulnerability: web vulnerability usually refers to the vulnerability on the website program, which may be caused by the carelessness of the code writer when writing the code. Common web vulnerabilities include sql injection, xss vulnerability, upload vulnerability, etc.

[0068] Vulnerability scanning attacks: Vulnerability scanning refers to a security detection (penetration attack) activity that uses vulnerability databases and scanning techniques to detect the security vulnerabilities of a specified remote or local computer system and discover exploitable vulnerabilities. Vulnerability scanners include different types such as network vulnerability scanners, host vulnerability scanners, and database vulnerability scanners. Similarly, hackers can also use vulnerability scanning to discover vulnerabilities in servers, preparing for subsequent attacks.

[0069] Payload data: This is the data that contains the information. Typically, during data transmission, to make data transmission more reliable, the raw data is transmitted in batches, with auxiliary information added to the beginning and end of each batch, such as the size of the batch and checksums. This is like adding "outer garments" to the batched raw data; these "outer garments" serve as identifiers, making the raw data less likely to be lost. A batch of data with its "outer garments" forms the basic transmission unit in the transmission channel, called a data frame or data packet (in some places, such as network transmission, data frames and data packets are not the same concept). The raw data containing the information in these data frames is the payload data.

[0070] It is understood that the model training method and vulnerability scanning attack detection method provided in this application embodiment can be applied to electronic devices, which may include terminals and servers; wherein the terminal may specifically be a smartphone, tablet computer, computer, personal digital assistant (PDA), etc.; the server may specifically be an application server or a web server. In addition, the model training method and the vulnerability scanning attack detection method may be executed by the same electronic device or by different electronic devices.

[0071] Figure 1 This is a schematic diagram of a vulnerability scanning and attack detection method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:

[0072] Step 101: Obtain the access log and extract key field information from the access log;

[0073] Step 102: Generate a feature vector based on the key field information;

[0074] Step 103: Input the feature vector into the vulnerability scanning and attack discovery model to obtain the analysis results output by the vulnerability scanning and attack discovery model; wherein, the vulnerability scanning and attack discovery model is obtained by pre-generating corresponding training feature vectors based on training logs, and using the training feature vectors and labels used to characterize whether the training source IP address corresponding to the training logs has vulnerability scanning and attack behavior for model training; the analysis results are used to characterize whether the behavior corresponding to the access logs is vulnerability scanning and attack behavior.

[0075] In step 101, the access log refers to the log generated when a user accesses the target object, i.e., the nginx access log. The user may be a compliant employee or customer, or a malicious actor. The target object refers to the object on which this application performs vulnerability scanning and attack detection, such as a website.

[0076] When a user visits a website, the server corresponding to that website generates an access log. If the electronic device is the web server, it can directly use the access log for subsequent processing. If the electronic device is a terminal, it communicates with the web server to obtain the access log and uses it for subsequent processing.

[0077] Understandably, access logs include multiple fields of information, such as: client (user) IP address (also known as source IP address), access time, access port, response time, request time, user address location code, requested URL address, request method, request status, requested page size, source page (i.e., which page the user was redirected from), user browser language, and other user browser information (e.g., browser version, browser type, etc.).

[0078] Since access logs contain many fields, some of which are required by this application and others are not, the electronic device can extract key fields from the access logs after obtaining them to obtain critical information. Understandably, the key fields to be extracted from the access logs can be pre-defined.

[0079] In step 102, after obtaining the key field information, the electronic device performs feature engineering on the key field information, that is, analyzes the key field information to obtain a feature vector. Specifically, the electronic device also stores historical access logs of the source IP address corresponding to the access log within a historical time period. Therefore, a feature vector can be generated based on the historical access logs and the current access log. It can be understood that the feature vector can be composed of access characteristics of the source IP address corresponding to the access log accessing the target object. For example, the feature vector can be composed of the number of times the URL is accessed and the number of URLs accessed. If the number of times the URL is accessed is 5 and the number of URLs accessed is 2, then the feature vector can be (5,2). Of course, the feature vector can also include values ​​corresponding to other access characteristics, which are not specifically limited in this embodiment.

[0080] In step 103, the vulnerability scanning attack discovery model is obtained by training the model to be trained in advance using training logs. The vulnerability scanning attack discovery model can be a convolutional neural network model, decision tree model, support vector machine model, random forest model, etc. This application embodiment does not limit the specific form of the vulnerability scanning attack discovery model.

[0081] After generating feature vectors, electronic devices input these feature vectors into a pre-trained vulnerability scanning and attack detection model. The model then outputs the analysis results corresponding to these feature vectors. These results can indicate whether the access behavior of the source IP address corresponding to the access log is a vulnerability scanning and attack, or not.

[0082] This application embodiment extracts key field information from access logs, generates feature vectors based on the key field information, and inputs the feature vectors into a machine learning model, enabling the machine learning model to analyze the feature vectors of the access logs, thereby improving the accuracy of identifying vulnerability scanning attack behaviors.

[0083] Based on the above embodiments, the key field information includes access time and source IP address; a feature vector is generated based on the key field information, including:

[0084] Based on the access time and source IP address, key field information is stored in the corresponding time window set;

[0085] Feature vectors are obtained by extracting features from key fields in the set.

[0086] In the specific implementation process, key field information of the same source IP address within a certain time period is stored in the set corresponding to the time window. A time window refers to a period of time; for example, a 5-minute or 10-minute window can be used. The size of the time window can also be set according to the actual situation. This application embodiment does not specifically limit the size of the time window. Taking October 10, 2022, 00:01 as an example, with a time window of 10 minutes and a source IP address of 111.111.111.11, the first time window is: October 10, 2022, 00:01-00:10; the second time window is: October 10, 2022, 00:11-00:20; the third time window is: October 10, 2022, 00:21-00:30; ... If the access time is 00:15 on October 10, 2022, then the corresponding key field information will be written into the set corresponding to the second time window. It should be noted that each appearing source IP address can obtain a corresponding set of time windows using the above-described partitioning method. After generating the corresponding feature vector based on the key field information in the set corresponding to a certain time window, the key field information in that set will be deleted.

[0087] In another embodiment, the start time of the time window for each source IP address can also be different; that is, the time window can be divided starting from the first access time of each source IP address to the target object. The advantage of doing this is that if the access behavior of a certain source IP address starts from a certain time in the middle, such as 12 noon, then there is no need to set a corresponding time window for that source IP address between 0:00 and 12:00, thus reducing the amount of computation.

[0088] A collection corresponding to a time window may be empty, may contain keyword field information corresponding to one access log, or may contain key field information corresponding to multiple access logs.

[0089] It should be noted that key field information may also include at least one of the following:

[0090] The returned status code, access duration, access header length, access packet size (in bytes), request method, target port, accessed URL, accessed domain name, destination IP address, and payload data sent during access.

[0091] When the current time reaches the end time of each time window, the key field information in the set corresponding to that time window is obtained, and feature extraction is performed on the key field information to obtain the feature vector corresponding to that time window.

[0092] It should be noted that the features in the feature vector may include at least one of the following:

[0093] The time window includes the number of accesses to the source IP address, the number of accessed URLs, the number of accessed ports, the number of different payload data, the cumulative access duration, the average length of the payload data, the average length of the accessed URLs, the number of successful status codes, the number of failed status codes, the number of identical data and URLs, the number of identical data and domain names, and the number of sensitive words in the payload data.

[0094] The following metrics are used to define access counts and status codes: Access Count: The number of times the source IP address sends access requests to the target object within the specified time window. Access URL Count: The number of different URLs accessed by the source IP address within the specified time window. Access Port Count: The number of ports accessed by the source IP address within the specified time window. Different Payload Data Count: The number of different payloads within the specified time window. Cumulative Access Duration: The cumulative access duration of the source IP address within the specified time window. Average Payload Data Length: The quotient of the total length of all payload data within the specified time window and the number of payloads. Average URL Length Accessed: The quotient of the total length of the accessed URLs in bytes within the specified time window and the number of different URLs. Success Status Code Count: The number of successfully returned status codes to the source IP address within the specified time window. Failure Status Code Count: The number of status codes that did not successfully return status codes to the source IP address within the specified time window. Different status codes can be used to represent success or failure. For example, status code 200 indicates success, and status code 404 indicates failure. The number of successful and failed status codes is determined by counting the number of 200 and 404 status codes. It should be noted that other numbers can also be used to represent success and failure; these can be pre-configured, and this embodiment does not impose specific limitations on them. The number of identical data and identical URLs refers to the number of access requests with the same payload and accessing the same URL within the time window. The number of identical data and identical domain names refers to the number of access requests with the same payload and accessing the same domain name within the time window. The number of sensitive words in the payload data refers to the total number of sensitive words present in all payload data within the time window. Here, sensitive words are pre-defined words.

[0095] In another embodiment, key field information of multiple related source IP addresses within a certain time period can be stored in a set of corresponding time windows. The correlation between multiple source IP addresses can be determined by analyzing the accessed URLs, target IP addresses, and payload data contained in the feature vectors within historical time periods.

[0096] Understandably, by analyzing feature vectors over historical time periods, attacker profiles can be generated, and it is also possible to analyze which business systems are more vulnerable to attacks, thereby strengthening the protection of vulnerable business systems.

[0097] This application embodiment utilizes a time window to generate feature vectors corresponding to key field information within the time window. This allows the vulnerability scanning attack discovery model to analyze the feature vectors corresponding to the time window and promptly detect vulnerability scanning attack behaviors.

[0098] Based on the above embodiments, the feature vector is input into the vulnerability scanning and attack discovery model, including:

[0099] Input the feature vector corresponding to the time window into the vulnerability scanning and attack discovery model. If the analysis result output by the vulnerability scanning and attack discovery model indicates that the behavior corresponding to the time window is not a vulnerability scanning and attack behavior, then obtain a new feature vector based on the feature vector corresponding to the time window and the feature vector of the next time window corresponding to the source IP address.

[0100] Input the new feature vectors into the vulnerability scanning and attack discovery model.

[0101] In the specific implementation process, taking the time window provided in the above embodiment as an example, after 10:00 on October 10, 2022, the feature vector corresponding to the first time window can be input into the vulnerability scanning and attack discovery model. If the analysis result output by the vulnerability scanning and attack discovery model indicates that the access behavior of the source IP address corresponding to the first time window does not belong to vulnerability scanning and attack behavior, then after 20:00 on October 10, 2022, the feature vector corresponding to the second time window can be added to the feature vector corresponding to the first time window to obtain a new feature vector. This new feature vector is then input into the vulnerability scanning and attack discovery model. If the analysis result output by the vulnerability scanning and attack discovery model indicates that the access behavior of the source IP address corresponding to the first time window does not belong to vulnerability scanning and attack behavior, then the feature vector corresponding to the third time window can be input into the vulnerability scanning and attack discovery model along with the feature vectors corresponding to the first two time windows for analysis. This process continues until the vulnerability scanning and attack discovery model analyzes the feature vector of the Nth time window and the sum of the feature vectors of the first N-1 time windows. If the analysis result indicates that the access behavior of the source IP address belongs to vulnerability scanning and attack behavior, then the source IP address will be further processed. Alternatively, the vulnerability scanning attack discovery model can analyze the sum of feature vectors from the first N time windows until the access behavior corresponding to the source IP address is still not a vulnerability scanning attack. It is understood that N is a pre-set value, such as N=1440, or it can be the number of time windows corresponding to one day; this application embodiment does not specifically limit this.

[0102] The advantage of this approach is that, taking a one-hour time window as an example, if the vulnerability scanning and attack detection model detects malicious actors engaging in vulnerability scanning and attack activities within the first two hours of the day, the source IP address used by these malicious actors can be blocked as early as the second hour of the day. Therefore, a smaller time window allows for more timely detection of vulnerability scanning and attacks, but also places a greater load on electronic devices; conversely, a larger time window results in less timely detection of vulnerability scanning and attacks, but a relatively smaller load on electronic devices. In practical applications, the size of the time window can be set according to the specific circumstances.

[0103] In this embodiment, the number of time windows corresponding to the feature vectors input into the vulnerability scanning attack discovery model increases from few to many, which can promptly detect vulnerability scanning attack behavior and identify low-frequency, time-divided vulnerability scanning attack behavior.

[0104] Based on the above embodiments, after obtaining the analysis results output by the vulnerability scanning attack discovery model, the method further includes:

[0105] If the analysis results indicate that the behavior corresponding to the access log is a vulnerability scanning attack, then the source IP address corresponding to the access log will be blocked.

[0106] In practice, to ensure the security of the target object, once it is determined that the behavior corresponding to the access log is a vulnerability scanning attack, the source IP address corresponding to that access log is blocked. This means prohibiting that source IP address from accessing the target object.

[0107] In another embodiment, if the analysis results output by the vulnerability scanning attack detection model indicate that the behavior corresponding to the access log belongs to vulnerability scanning attack behavior, the electronic device sends an alarm notification to the terminal corresponding to the operation and maintenance personnel. The operation and maintenance personnel then re-evaluate the behavior. If the evaluation result confirms that it belongs to vulnerability scanning attack behavior, a confirmation message is sent to the electronic device, and the electronic device blocks the source IP address. The advantage of this approach is that it prevents legitimate source IP addresses from being blocked due to false detections by the vulnerability scanning attack detection model.

[0108] Based on the above embodiments, Figure 2 This is a schematic diagram of a model training method provided in an embodiment of this application, such as... Figure 2 As shown, the method includes:

[0109] Step 201: Obtain training logs corresponding to multiple training source IP addresses. It can be understood that the training logs are access logs collected by the electronic device from the training source IP addresses within a historical time period.

[0110] Step 202: Extract keywords from the training logs to obtain training keywords. After obtaining the training logs, the electronic device parses the training logs and extracts the training keywords. It is understood that the specific fields corresponding to the training keywords are the same as the specific fields corresponding to the key field information used in the vulnerability scanning attack detection in the above embodiments, and will not be repeated here. After obtaining the training keywords, based on the experience of security experts, the training keywords of each training log are tagged. The tags are used to characterize whether the training source IP address exhibits vulnerability scanning attack behavior. For example, "1" can be used to indicate the presence of vulnerability scanning attack behavior as a negative sample; "0" can be used to indicate the absence of vulnerability scanning attack behavior as a positive sample. Other identifiers can also be used to label the training logs; this embodiment does not specifically limit this.

[0111] Step 203: Generate training feature vectors for the corresponding training IP addresses based on the training keywords corresponding to each training source IP address. A preset time period can be determined according to the actual situation; for example, the preset time period can be set to one day. This involves judging the behavior of the training source IPs each day to determine if it is vulnerability scanning behavior, and aggregating the training source IPs and dates to obtain training feature vectors. The training feature vectors include at least one of the following within a day: the number of accesses corresponding to the training source IP address, the number of accessed URLs, the number of accessed ports, the number of different payload data items, the cumulative access duration, the average length of the payload data, the average length of the accessed URLs, the number of successful status code returns, the number of failed status code returns, the number of identical data and URLs, the number of identical data and domain names, and the number of sensitive words present in the payload data. The training feature vectors and corresponding labels generate training samples, and multiple training samples constitute a training set. Therefore, the training set includes training samples corresponding to multiple training source IP addresses, and each training source IP address corresponds to multiple training samples. It is understandable that during model training, the features contained in the training feature vector input into the model to be trained are the same as the features contained in the feature vector input into the trained vulnerability scanning and attack discovery model during actual detection.

[0112] Step 204: Train the model to be trained using the training feature vector and corresponding labels to obtain the vulnerability scanning attack discovery model. The labels are used to characterize whether the training source IP address exhibits vulnerability scanning attack behavior. The training set is split; for example, 80% of the training samples can be used for training, and 20% for validation. The training data is then trained using a 10-fold cross-validation method. After training, the model is validated using the validation set. If the accuracy of the trained model is above 95%, it is adopted as the final vulnerability scanning attack discovery model. It should be noted that the above training set splitting ratio and the accuracy at which training stops can be set according to actual conditions, and this embodiment does not impose specific limitations on them.

[0113] After training, the model files corresponding to the vulnerability scan attack discovery models are saved in the database and are called by the system when it performs detection.

[0114] The embodiments of this application obtain a vulnerability scanning and attack discovery model through model training, and improve the accuracy of access log identification by using this vulnerability scanning and attack discovery model.

[0115] In another embodiment, the deployment method of the vulnerability scanning attack discovery model is as follows:

[0116] (1) Real-time vectorization of nginx logs

[0117] The vulnerability scanning and attack discovery model is a real-time computation model that can be deployed in a server cluster. After Nginx logs are parsed and sent to the cluster, they are first grouped according to access time and source IP address. Logs with the same source IP address and access time within a defined time window are grouped into the corresponding set within that time window. This time window can be set to five or ten minutes. Within this time window, Nginx logs are transformed into feature vectors through feature engineering. These feature vectors are then sent to the vulnerability scanning and attack discovery model to obtain analysis results. If the classification result indicates that the behavior is an attack, an alert is issued. Otherwise, the feature vector is written to a cache. When the next window contains a group that has already been calculated, the two calculated feature vectors are summed and sent back to the vulnerability scanning and attack discovery model. The result is then returned. This process continues until the analysis result indicates that the source IP address has engaged in a vulnerability scanning attack, or until the number of time windows corresponding to the feature vectors input into the vulnerability scanning and attack discovery model reaches a preset number.

[0118] (2) Deployment method of vulnerability scanning and attack discovery model on server

[0119] The vulnerability scanning and attack discovery model's program and the server cluster's log processing program are both deployed on the same server and communicate with each other using inter-process communication. Figure 3 This is a schematic diagram illustrating the interaction between the model and the server provided in the embodiments of this application, such as... Figure 3 As shown, after the log processing program generates feature vectors, it calls the vulnerability scanning and attack discovery model process, which then sends the analysis results back to the server via inter-process communication.

[0120] (3) Alarm generation when an anomaly is detected

[0121] When a vulnerability scan detects an anomaly in the model, the results are sent to the alert front-end page. This page displays detailed information about the vulnerability scan, providing operational staff with information for assessment. If it is confirmed to be a scan attack, the server can work with the firewall to block it. If it is normal behavior, operational staff can annotate the behavior. The annotated results will be saved for periodic model updates to improve accuracy.

[0122] Figure 4 This is a schematic diagram of a vulnerability scanning and attack detection device provided in an embodiment of this application. The device can be a module, program segment, or code on an electronic device. It should be understood that this device is similar to the one described above. Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The various steps involved in the method embodiment, and the specific functions of the device, can be found in the description above. To avoid repetition, detailed descriptions are omitted here. The device includes: a log acquisition module 401, a feature extraction module 402, and a detection module 403, wherein:

[0123] The log acquisition module 401 is used to acquire access logs and extract key field information from the access logs;

[0124] The feature extraction module 402 is used to generate feature vectors based on key field information;

[0125] The detection module 403 is used to input the feature vector into the vulnerability scanning attack discovery model and obtain the analysis results output by the vulnerability scanning attack discovery model; wherein, the vulnerability scanning attack discovery model is obtained by pre-generating corresponding training feature vectors based on training logs, and using the training feature vectors and labels used to characterize whether the training source IP address corresponding to the training log has vulnerability scanning attack behavior for model training; the analysis results are used to characterize whether the behavior corresponding to the access log is vulnerability scanning attack behavior.

[0126] Based on the above embodiments, the key field information includes access time and source IP address; the feature extraction module 402 is specifically used for:

[0127] Based on the access time and the source IP address, the key field information is stored in the corresponding time window set;

[0128] Feature vectors are obtained by extracting features from key fields in the set.

[0129] Based on the above embodiments, the detection module 403 is specifically used for:

[0130] Input the feature vector corresponding to the time window into the vulnerability scanning and attack discovery model. If the analysis result output by the vulnerability scanning and attack discovery model indicates that the behavior corresponding to the time window is not a vulnerability scanning and attack behavior, then obtain a new feature vector based on the feature vector corresponding to the time window and the feature vector of the next time window corresponding to the source IP address.

[0131] Input the new feature vectors into the vulnerability scanning and attack discovery model.

[0132] Based on the above embodiments, the features in the feature vector include at least one of the following: the number of times the source IP address is accessed, the number of accessed URLs, the number of accessed ports, the number of different payload data, the cumulative access duration, the average length of the payload data, the average length of the accessed URLs, the number of successful status codes, the number of failed status codes, the number of identical data and identical URLs, the number of identical data and identical domain names, and the number of sensitive words in the payload data.

[0133] Based on the above embodiments, the key field information also includes at least one of the following: the returned status code, access duration, access header length, access data packet size, request method, target port, accessed URL, accessed domain name, destination IP address, and payload data sent during access.

[0134] Based on the above embodiments, the device further includes a processing module for:

[0135] If the analysis results indicate that the behavior corresponding to the access log is a vulnerability scanning attack, then the source IP address corresponding to the access log will be blocked.

[0136] Based on the above embodiments, the device further includes a model training module, used for:

[0137] Obtain training samples, which include training logs corresponding to multiple training source IP addresses and tags corresponding to the training source IP addresses; wherein, the tags are used to characterize whether the training source IP addresses exhibit vulnerability scanning attack behavior;

[0138] Extract keywords from the training logs to obtain training keywords;

[0139] A training feature vector is generated for each training source IP address based on the training keywords corresponding to each training source IP address. The training feature vector includes at least one of the following: the number of accesses, the number of accessed URLs, the number of access ports, the number of different payload data, the cumulative access duration, the average length of the payload data, the average length of the accessed URL, the number of successful status codes, the number of failed status codes, the number of identical data and the same URL, the number of identical data and the same domain name, and the number of sensitive words in the payload data.

[0140] The training feature vectors and corresponding labels are used to train the model to be trained, thereby obtaining a vulnerability scanning and attack discovery model.

[0141] In another embodiment, this application provides a vulnerability scanning and attack discovery model training device, including: a sample acquisition module, a keyword extraction module, a training feature vector generation module, and a training module, wherein:

[0142] The sample acquisition module is used to acquire training samples, which include training logs corresponding to multiple training source IP addresses and tags corresponding to the training source IP addresses; wherein, the tags are used to characterize whether the training source IP addresses have vulnerability scanning attack behavior;

[0143] The keyword extraction module is used to extract keywords from the training logs to obtain training keywords;

[0144] The training feature vector generation module is used to generate a training feature vector corresponding to each training source IP address based on the training keywords corresponding to each training source IP address.

[0145] The training module is used to train the model to be trained using the training feature vector and the corresponding labels to obtain the vulnerability scanning and attack discovery model.

[0146] Based on the above embodiments, the features in the training feature vector include at least one of the following: the number of accesses corresponding to the training source IP address within a preset time period, the number of accessed URLs, the number of access ports, the number of different payload data, the cumulative access duration, the average length of the payload data, the average length of the accessed URLs, the number of successful status codes, the number of failed status codes, the number of identical data and identical URLs, the number of identical data and identical domain names, and the number of sensitive words in the payload data.

[0147] Figure 5 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 5As shown, the electronic device includes: a processor 501, a memory 502, and a bus 503; wherein,

[0148] The processor 501 and the memory 502 communicate with each other through the bus 503;

[0149] The processor 501 is used to call program instructions in the memory 502 to execute the methods provided in the above-described method embodiments, including, for example,: acquiring access logs and extracting key field information from the access logs; generating feature vectors based on the key field information; inputting the feature vectors into a vulnerability scanning attack discovery model to obtain the analysis results output by the vulnerability scanning attack discovery model; wherein, the vulnerability scanning attack discovery model is obtained by pre-generating corresponding training feature vectors based on training logs, and training the model using the training feature vectors and labels used to characterize whether the training source IP address corresponding to the training logs exhibits vulnerability scanning attack behavior; the analysis results are used to characterize whether the behavior corresponding to the access logs is a vulnerability scanning attack behavior.

[0150] Processor 501 can be an integrated circuit chip with signal processing capabilities. The processor 501 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0151] The memory 502 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0152] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the methods provided in the above-described method embodiments, such as: acquiring access logs and extracting key field information from the access logs; generating feature vectors based on the key field information; inputting the feature vectors into a vulnerability scanning attack discovery model to obtain the analysis results output by the vulnerability scanning attack discovery model; wherein, the vulnerability scanning attack discovery model is obtained by pre-generating corresponding training feature vectors based on training logs, and training the model using the training feature vectors and labels used to characterize whether the training source IP address corresponding to the training logs exhibits vulnerability scanning attack behavior; the analysis results are used to characterize whether the behavior corresponding to the access logs is vulnerability scanning attack behavior.

[0153] This embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the methods provided in the above-described method embodiments. These instructions include, for example,: acquiring access logs and extracting key field information from the access logs; generating feature vectors based on the key field information; inputting the feature vectors into a vulnerability scanning attack discovery model to obtain the analysis results output by the vulnerability scanning attack discovery model; wherein the vulnerability scanning attack discovery model is obtained by pre-generating corresponding training feature vectors based on training logs, and training the model using the training feature vectors and labels used to characterize whether the training source IP address corresponding to the training logs exhibits vulnerability scanning attack behavior; the analysis results are used to characterize whether the behavior corresponding to the access logs is a vulnerability scanning attack behavior.

[0154] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0155] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0157] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0158] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method of vulnerability scan attack detection, the method comprising: The method comprises: obtaining an access log and extracting key field information from the access log; generating a feature vector according to the key field information; inputting the feature vector into a vulnerability scanning attack discovery model to obtain an analysis result output by the vulnerability scanning attack discovery model; wherein the vulnerability scanning attack discovery model is pre-generated according to a training feature vector generated from a training log, and is obtained by model training using the training feature vector and a label representing whether a training source IP address corresponding to the training log has a vulnerability scanning attack behavior; and the analysis result is used to represent whether the behavior corresponding to the access log is a vulnerability scanning attack behavior; the key field information comprises access time and source IP address; and the generation of the feature vector according to the key field information comprises: storing the key field information into a corresponding set of time windows according to the access time and the source IP address; and determining whether a plurality of source IP addresses have an association relationship by including the accessed URL, target IP address and payload data in the feature vector in a historical time period, and storing the key field information of the plurality of source IP addresses having the association relationship into the corresponding set of time windows; and the start time of the time window is the access time when the source IP address accesses the target object for the first time; performing feature extraction on the key field information in the set to obtain the feature vector.

2. The method of claim 1, wherein, the inputting of the feature vector into the vulnerability scanning attack discovery model comprises: inputting the feature vector corresponding to the time window into the vulnerability scanning attack discovery model; if the analysis result output by the vulnerability scanning attack discovery model represents that the behavior corresponding to the time window is not a vulnerability scanning attack behavior, then obtaining a new feature vector according to the feature vector corresponding to the time window and the feature vector of the next time window corresponding to the source IP address; inputting the new feature vector into the vulnerability scanning attack discovery model.

3. The method of claim 1, wherein, The features in the feature vector include at least one of the following: the number of accesses of the source IP address, the number of accessed URLs, the number of accessed ports, the number of different payload data, the cumulative access duration, the average length of the payload data, the average length of the accessed URL, the number of successful status code returns, the number of failed status code returns, the number of same data and same URLs, the number of same data and same domain names, and the number of sensitive words in the payload data.

4. The method of claim 1, wherein, The key field information further comprises at least one of the following: returned status code, access duration, length of access header, number of bytes of access data packet, request method, target port, accessed URL, accessed domain name, destination IP address, and payload data sent during access.

5. The method of claim 1, wherein, After obtaining the analysis result output by the vulnerability scanning attack discovery model, the method further comprises: if the analysis result represents that the behavior corresponding to the access log is a vulnerability scanning attack behavior, then banning the source IP address corresponding to the access log.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: Obtaining training samples, the training samples including training logs corresponding to a plurality of training source IP addresses and labels corresponding to the training source IP addresses; wherein the labels are used to represent whether the training source IP addresses exist vulnerability scanning attack behaviors; Performing keyword extraction on the training logs to obtain training keywords; Generating training feature vectors of the training source IP addresses according to the training keywords corresponding to each of the training source IP addresses; Training a to-be-trained model using the training feature vectors and the corresponding labels to obtain the vulnerability scanning attack discovery model.

7. A vulnerability scan attack detection apparatus characterized by comprising: Comprise: A log obtaining module, configured to obtain an access log and extract key field information from the access log; A feature extraction module, configured to generate a feature vector according to the key field information; A detection module, configured to input the feature vector into a vulnerability scanning attack discovery model to obtain an analysis result output by the vulnerability scanning attack discovery model; wherein the vulnerability scanning attack discovery model is obtained by pre-generating corresponding training feature vectors according to training logs, and training a model using the training feature vectors and labels used to represent whether training source IP addresses corresponding to the training logs exist vulnerability scanning attack behaviors; the analysis result is used to represent whether a behavior corresponding to the access log is a vulnerability scanning attack behavior; The key field information includes access time and source IP address; the feature extraction module is specifically configured to: According to the access time and the source IP address, store the key field information into a corresponding set of time windows; and determine whether a plurality of source IP addresses have a correlation relationship by including accessed URLs, target IP addresses and payload data in the feature vectors in a historical time period, and store key field information of a plurality of source IP addresses having a correlation relationship into a corresponding set of time windows; a start time of the time window is an access time corresponding to a first time that the source IP address accesses a target object; Perform feature extraction on the key field information in the set to obtain the feature vector.

8. An electronic device, comprising: Comprise: A processor, a memory and a bus, wherein, The processor and the memory complete mutual communication through the bus; The memory stores program instructions executable by the processor, and the processor calling the program instructions can execute the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, and when the computer instructions are run by a computer, the computer executes the method of any one of claims 1-6.

Images