A method and system for port scan detection using server logs

Abstract

The application discloses a method and system for port scanning detection by using server logs, comprising a feature extraction step and a port scanning detection step; first, access logs on a server cluster are collected, then feature extraction is carried out to construct aggregated records describing access IP behaviors, and finally, whether port scanning occurs is judged by using rules or a clustering method according to the number of positive samples, and meanwhile, the detection result is fed back to an administrator and a positive sample database, so that the accuracy of the method is further improved. The application fully considers the actual needs of operation and maintenance safety audit managers, and has strong implementability and applicability.

Description

TECHNICAL FIELD

[0001] The present application relates to log analysis, log mining and port scanning detection technology, and particularly relates to a technology of extracting information from port access logs to determine port scanning. BACKGROUND

[0002] At present, with the development of computer and network technology, computer systems and the Internet are increasingly applied to different industries. Many enterprises use computer systems to store and manage important information, and use the Internet to operate their own business, such as some e-commerce platforms building server clusters to manage customer information and process orders. At the same time, attacks on computer systems are becoming more and more frequent and the attack means are becoming diversified, so it is necessary to find some better measures to prevent computer and network systems from being attacked.

[0003] For attackers, initiating a port scan to the target network to discover the services provided by the server and its vulnerabilities is the first step of intrusion. Therefore, it is crucial to detect port scanning behavior in advance to prevent servers from being intruded. For the current defense means of servers, the administrator of the server configures firewall access rules, deploys intrusion detection systems (IDS) or intrusion prevention systems (IPS) to discover IP with scanning behavior, and takes measures such as warning, blocking or banning. However, at present, completely relying on the defense system for detection is faced with the problems of inaccurate detection and false positives, and in the actual application process, the administrator often needs to analyze the access logs of the server and combine the business development situation to adjust the defense system, such as adding new filtering rules, relaxing the warning of a certain IP, etc. However, the access logs of the server often have massive data records, and the administrator usually needs to manage more than one server, which further increases the difficulty of manual analysis. The access logs of the server generally have the following characteristics: a large number of records without analysis value, including invalid fields, redundant records, and false alarms; a small number of suspicious records, such as new access IPs or IPs that have appeared before but suddenly have obvious abnormal access behavior. SUMMARY

[0004] In view of the current server defense problems, the defense of the server depends on the cooperation of the defense system and the administrator, the administrator faces the difficulty of analyzing the massive data log, and the existing method based on the number of statistical packets to judge the port scanning has the problem of low detection accuracy. A technology capable of extracting access records from the original access log of the server and judging whether the access record is a port scan, the technology has the characteristics of constructing a simplified record from the original access record, presenting the record with abnormal access behavior to the administrator, and constructing the simplified record for the administrator to further diagnose, which is convenient for the administrator to understand and easy to further trace the related records before and after it occurs. Specifically, the IP pairs such as access IP and server IP are summarized, the access records under each IP pair are aggregated and divided according to a certain aggregation granularity, the aggregation record of the access behavior history of the IP pair is obtained, the existing positive sample number sampling rule method or clustering method is used to judge whether the record is a scan, the aggregation record judged as normal is added to the positive sample database for similarity evaluation with the new record; the record judged as abnormal is handed over to the administrator for analysis, and the administrator can analyze the access behavior history before the record from the positive sample library to analyze and judge. The method provided by the application can greatly reduce the burden of the administrator to analyze the log, and with the expansion of the positive sample database, the detection accuracy of the method for port scanning can be further improved.

[0005] Technical scheme: A method for detecting port scanning by using server log, comprising feature extraction and port scanning detection steps.

[0006] The feature extraction step is specifically:

[0007] Step 100, collecting and summarizing the port access log on the server cluster.

[0008] Step 101, reading the summarized port access log, extracting access IP, server IP, access time, access IP, access port and other field information from each access record in the log, and obtaining a simplified description of the access record.

[0009] Step 102, counting all repeated simplified descriptions to calculate the frequency information of their occurrence, and compressing all repeated simplified descriptions into one simplified description and its frequency.

[0010] Step 103, taking access IP and server IP as the key, and one access IP and one server IP as an IP pair, summarizing all records containing the IP pair.

[0011] Step 104, arrange the related records of each IP pair in ascending order according to the time of access occurrence, set the minimum aggregation granularity, and regard the records with an access occurrence time interval not exceeding the minimum aggregation granularity as part of one continuous access, and combine all the records of one continuous access to obtain the aggregated record of the access behavior history of the IP pair.

[0012] In step 104, the related records of each IP pair are arranged in ascending order according to the time of access occurrence.

[0013] The port scanning detection step is specifically:

[0014] Step 200, determine whether the number of records of each IP pair in the positive sample database is greater than a predetermined number, and the predetermined number is the minimum sample number that can apply the clustering method. If yes, go to step 201, otherwise go to step 202.

[0015] Positive sample: the aggregated record obtained through feature processing, which is a non-exceptional sample after port scanning detection judgment and administrator confirmation. The sample is constructed with access IP and server IP as keys, so the record of a certain IP pair in the positive sample database describes the normal access behavior of the access IP pair to the server IP. When there is a new aggregated record of the IP pair to be detected, the similarity calculation with the existing record is performed, which actually evaluates whether the current access behavior deviates from the normal access behavior of the IP pair in the past.

[0016] Positive sample database: the aggregated record of the positive sample stored with the IP pair as the key; when the positive sample of a certain IP pair is sufficient, it can define the normal access behavior of the IP pair, and any aggregated record deviating from the normal access behavior will be labeled as an exception through port scanning detection. The positive sample database is used to describe the normal access behavior, and when the number of a certain IP pair in the positive sample database is not enough, the clustering method cannot achieve good results, and the positive sample needs to be accumulated by the rule method first. Therefore, a predetermined number is considered to be set, and the clustering method is used when the number of positive samples is greater than the predetermined number. The predetermined number can be determined by evaluating the detection accuracy of the clustering method on the abnormal samples in the data set composed of the positive samples and the abnormal samples determined by the rule method, and when the detection accuracy meets the requirements of the enterprise, the number of positive samples in the data set can be used as the predetermined number. Step 201, calculate the similarity matrix of the port access frequency of the current aggregated record and the records in the positive sample database, detect outliers by the clustering method, and label the aggregated record corresponding to the outlier as port scanning.

[0017] Step 202, judging the current record is port scanning according to the rule written by business logic. Business refers to the transaction handled by the user using the server, such as the server only in the intranet and the server can be accessed by the external IP, and the rules written for the two cases are different. For the former, only need to confirm whether the access IP is intranet IP, for the latter, need to define more complex rules.

[0018] Step 203, aggregate port scanning detection results, submit to administrator confirmation, remove the aggregate record confirmed by administrator as port scanning, add the remaining aggregate record to the positive sample database.

[0019] The record with access time interval not more than the minimum aggregation granularity is regarded as part of one continuous access, and the aggregate record of the access behavior history of the IP pair is obtained by merging all records of one continuous access. Specifically, the time interval of adjacent records is calculated, the records are divided into several groups, the access time of adjacent records in the same group does not exceed the minimum aggregation granularity, and the adjacent records between adjacent groups are greater than the minimum aggregation granularity. The records in the same group are aggregated, and the aggregation rule of each field of the records is: the access IP and the server IP are unchanged; the minimum occurrence time and the maximum occurrence time are the start time and the end time of the aggregate record; the union of IP domains is the IP domain of the aggregate record; the union of access ports is the access port of the aggregate record, and the frequency of the accessed port is recorded; and the sum of the frequency of the records participating in the merging is the frequency of the aggregate record.

[0020] The similarity of the port scanning frequency is specifically: for any two aggregate records R1 and R2, the access port of R1 is {(p1, f1), (p2, f2), …, (p m ,f m )}, the frequency of the aggregate record is Freq, and the port access frequency X1 of R1 is (f1 / Freq, f2 / Freq, …, f m / Freq). The similarity of the port access frequency of the two aggregate records R1 and R2 is defined as:

[0021]

[0022] In the formula, I is an indicator function, w i is the weight of the port, p represents the port, and f represents the number of times the port is accessed in this aggregate record, such as (80, 20) representing that the "80" port is accessed 20 times, and X1(i) representing the i-th component of the port access frequency vector X1.

[0023] A computer device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the method for port scan detection using server logs as described above when executing the computer program.

[0024] A computer readable storage medium stores a computer program for implementing the method for port scan detection using server logs as described above.

[0025] A system for port scan detection using server logs comprises a feature extraction module and a port scan detection module.

[0026] The feature extraction module collects and aggregates port access logs on a server cluster, reads the aggregated port access logs, extracts part of field information from each access record to obtain a concise description of the access record, the part of field information includes access IP and server IP field information, counts all repeated concise descriptions to calculate the frequency information of their occurrence, compresses all repeated concise descriptions into one concise description and its occurrence frequency, aggregates all records containing the IP pair of access IP and server IP, sorts the related records under each IP pair according to the time of access occurrence, sets the minimum aggregation granularity, and considers the records with an access time interval not exceeding the minimum aggregation granularity as part of one continuous access, merges all records of one continuous access to obtain the aggregated records of the access behavior history of the IP pair.

[0027] The port scan detection module judges whether the number of records corresponding to each IP pair in the positive sample database is greater than a predetermined number, if yes, calculates the similarity matrix of the port access frequency of the current aggregated record and the records in the positive sample database, detects outliers using clustering method, and marks the aggregated record corresponding to the outlier as port scan, otherwise, judges the current record as port scan according to the rules written according to the business logic, aggregates the port scan detection results, submits them to the administrator for confirmation, removes the aggregated records confirmed by the administrator as port scan, and adds the remaining aggregated records to the positive sample database.

[0028] Advantages: Compared with the prior art, the method fully considers the actual needs of operation and maintenance safety audit managers, has strong implementability and strong applicability, can greatly reduce the burden of administrator analysis logs, and the detection accuracy of the method for port scan can be further improved as the positive sample database expands. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The feature extraction workflow diagram of the embodiment of the present application;

[0030] Figure 2 Anomaly detection workflow diagram for embodiments of the present application. DETAILED DESCRIPTION

[0031] The present application will be further illustrated by the following embodiments. It should be understood that these embodiments are only used to explain the present application and not used to limit the scope of the present application. After reading the present application, those skilled in the art can make various modifications to the present application, and all modifications fall within the scope of the appended claims.

[0032] A method for port scanning detection using server logs, including feature extraction and port scanning detection steps.

[0033] The workflow of feature extraction is shown in Figure 1 The records in the access log of the server at least need to contain the access IP, server IP, time or timestamp of the scanning occurrence, the home of the access IP, and the port of the server accessed. In actual use, some additional information can also be obtained for constructing a rule method, and the additional information is information determined according to the user's business, such as a whitelist IP, which can not be detected. The following are specific steps for implementing feature extraction: an administrator obtains log information from a server cluster managed by himself (step 10), and the frequency of obtaining these log information depends on the detection requirements, and a database can be used to complete the subsequent processing when the data amount is large; regular expressions are used to extract necessary fields according to the record format of the log, and the extracted fields are spliced into records (step 11), as shown in Table 1; the hash value of the record is calculated, the frequency of repeated records is counted, and the repeated records are represented as a single record and its frequency (step 12), such as the 1st and 3rd records in Table 1 are repeated, so only record 1 is retained and its frequency of occurrence is recorded as 2; the relevant records are summarized with the access IP and server IP as indexes (step 13), such as record 1 and record 4 in Table 1 are summarized together; the relevant records under each pair of IP are arranged in ascending order according to the time or timestamp of the scanning occurrence (step 14); a minimum aggregation granularity is set, and multiple accesses with a time interval not exceeding the minimum aggregation granularity are regarded as a continuous access, and the records of a continuous access are aggregated into a record, and the relevant records under each pair of IP are arranged in ascending order according to the timestamp of the scanning occurrence (step 15), such as the aggregation granularity is 5 min, and records 1 and 4 in Table 1 are regarded as a continuous access and aggregated into a record, and the setting of the aggregation granularity depends on the situation of the server being accessed in the business, and if the frequency of the server being accessed is relatively high, the aggregation granularity can be set to be smaller.

[0034] Table 1 records extracted from logs

[0035] Access IP Server IP Access time IP locality Access port 1 10.10.1.12 11.14.12.11 20220114-09:02:12 Local area network 80,443,8080,… 2 88.88.1.14 11.14.12.13 20220114-09:08:07 Holland 80,143,144,443,… 3 10.10.1.12 11.14.12.11 20220114-09:02:12 Local area network 80,443,8080,… 4 10.10.1.12 11.14.12.11 20220114-09:04:10 Local area network 80,443,8080,…

[0036] The workflow of port scanning detection is as follows: Figure 2 As shown. When accessing a server, typically only ports providing services to a portion of the server are accessed. Therefore, a set of high-risk ports can be defined based on the provided services, and only the access frequency of ports in this set needs to be calculated. DBSCAN can be used as the clustering method because it doesn't require pre-specifying the number of clusters. If there is prior knowledge of the normal port access behavior patterns of accessing IPs, k-medoids can also be chosen, and the judgment can be made based on whether the distance from the sample to the cluster center is greater than a certain threshold. The construction of rules mainly depends on business logic: for example, judging by IP location, if there is no cross-border business but there are overseas IP accesses, then this record is judged as a port scan, attracting the administrator's attention. Judging by scan frequency, if a record lasts for a very short time but is scanned multiple times during that period, it can be judged as an explosive scan, and thus a port scan. Judging by the access frequency of high-risk ports, for example, some ports are not frequently used within the business scope but are risky; if such ports are accessed multiple times in a record, then it can be judged as a port scan. The following are the specific steps for implementing port scanning detection: For a batch of records to be detected, extract their access IP and server IP, and query the number of samples containing that IP pair in the positive sample database (step 16); determine whether this number is less than the minimum sample size N. lim Large (Step 17), N lim The value can be determined based on business needs; for example, it can be estimated as the average number of positive samples received per week. If the number is greater than the preset N... lim Then, extract all positive samples containing the current IP pair from the positive sample database (step 18a); next, calculate the port access frequency of the sample to be detected and the positive samples, and the similarity of the port access frequency vectors of the sample to be detected and the positive samples (step 19). The port weight can be set according to the frequency of port access in the business. For example, if the business needs to frequently access port 22 and less frequently access port 143, then the weight of port 143 can be set to be larger. If the IP of a certain aggregate record frequently accesses an infrequent but risky port, which is significantly deviating from the business behavior, then its similarity with the port access frequency in the positive sample database will be very low, and it is easier to be judged as an outlier. Here, the similarity can also be replaced by other calculation methods, such as weighted cosine distance, etc. Use DBSCAN to detect outlier samples, and these samples are marked as port scans (step 20). DBSCAN can select appropriate parameters based on the previous rule judgment and the results of the administrator's review. For example, it can appropriately relax false alarms and minimize false negatives. If the number is less than or equal to N limIf the sample is not a port scan, the sample is added to the positive sample database (step 22b). In order to improve the calculation efficiency, the maximum number of samples that can be stored in the positive sample database for each IP pair can be set. When the storage number is about to overflow, the data that has been in the positive sample database the longest is deleted.

[0037] A system for detecting port scanning using server logs, comprising a feature extraction module and a port scanning detection module.

[0038] The feature extraction module: collects and aggregates port access logs on a server cluster; reads the aggregated port access logs, extracts part of the field information in each access record to obtain a simplified description of the access record; the part of the field information includes access IP and server IP field information; counts all repeated simplified descriptions to calculate the frequency information of their occurrence, and compresses all repeated simplified descriptions into one simplified description and its occurrence frequency; aggregates all records containing the IP pair with access IP and server IP as the key, one access IP and one server IP as an IP pair; sorts the related records under each IP pair in ascending order according to the time of access occurrence, sets the minimum aggregation granularity, and considers the records with an access occurrence time interval not exceeding the minimum aggregation granularity as part of one continuous access, and merges all records of one continuous access to obtain the aggregated record of the access behavior history of the IP pair;

[0039] The port scanning detection module: judges whether the number of records corresponding to each IP pair in the positive sample database is greater than a predetermined number; if so, calculates the similarity matrix of the port access frequency of the current aggregated record and the records in the positive sample database, detects outliers using clustering methods, and marks the aggregated record corresponding to the outlier as port scanning; otherwise, according to the rules written according to the business logic, the current record is judged to be port scanning; aggregate the port scanning detection results and hand them over to the administrator for confirmation, remove the aggregated records confirmed by the administrator to be port scanning, and add the remaining aggregated records to the positive sample database.

[0040] Obviously, those skilled in the art should understand that each step of the method for detecting port scanning by using server log of the above-mentioned embodiments of the present application or each module of the system for detecting port scanning by using server log can be realized by general computing devices, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, and in some cases, the steps shown or described can be executed in different order, or they can be respectively made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to realize. Thus, the embodiments of the present application are not limited to any particular combination of hardware and software.

Claims

1. A method for port scan detection using server logs, the method comprising: The method comprises feature extraction and port scan detection steps. The feature extraction step specifically comprises: Step 100, collecting and aggregating port access logs on a server cluster; Step 101, reading the aggregated port access logs, extracting part of the field information in each access record to obtain a concise description of the access record; the part of the field information includes access IP and server IP field information; Step 102, counting all repeated concise descriptions to calculate the frequency information of their occurrence, and compressing all repeated concise descriptions into one concise description and its occurrence frequency; Step 103, taking access IP and server IP as keys, one access IP and one server IP as an IP pair, aggregating all records containing the IP pair; Step 104, sorting the related records under each IP pair according to the time of access occurrence, setting a minimum aggregation granularity, regarding the records with an access occurrence time interval not exceeding the minimum aggregation granularity as part of one continuous access, merging all records of one continuous access to obtain the aggregated records of the access behavior history of the IP pair; The port scan detection step specifically comprises: Step 200, judging whether the number of records corresponding to each IP pair in the positive sample database is greater than a predetermined number; if yes, going to step 201, otherwise going to step 202; Step 201, calculating the similarity matrix of the port access frequency of the current aggregated record and the records in the positive sample database, detecting outliers by clustering method, and marking the aggregated record corresponding to the outlier as port scan; Step 202, judging whether the current record is port scan according to the rules written according to business logic; Step 203, aggregating the port scan detection results, submitting them to the administrator for confirmation, removing the aggregated records confirmed by the administrator as port scan, and adding the remaining aggregated records to the positive sample database; The similarity of the port scanning frequencies is specifically: the access port of any two aggregated records and , is , and the frequency is Freq, The port access frequency of ; the similarity of the port access frequencies of the two aggregated records and is defined as: In the formula is the weight of the port.

2. The method for port scan detection using server logs according to claim 1, wherein, The step of regarding the records with an access occurrence time interval not exceeding the minimum aggregation granularity as part of one continuous access, merging all records of one continuous access to obtain the aggregated records of the access behavior history of the IP pair, specifically comprises: calculating the time interval of adjacent records, dividing the records into several groups, the access time of adjacent records in the same group does not exceed the minimum aggregation granularity, and the adjacent records between adjacent groups are greater than the minimum aggregation granularity; aggregating the records in the same group, and the aggregation rules of the fields of the merged records are: the access IP and server IP remain unchanged; The minimum occurrence time and the maximum occurrence time are the start time and the end time of the aggregated record; the union of IP attributes is the IP attribute of the aggregated record; the union of access ports is the access port of the aggregated record, and the frequency of the accessed port is recorded; the sum of the frequencies of the records participating in the merging is the frequency of the aggregated record.

3. The method for port scan detection using server logs as claimed in claim 1, wherein, In step 104, the related records under each IP pair are sorted in ascending order according to the time of access occurrence.

4. The method for port scan detection using server logs as claimed in claim 1, wherein, In step 101, the aggregated port access logs are read, and part of the field information in each access record is extracted to obtain a concise description of the access record, including access IP, server IP, time of access occurrence, attribute of access IP, and access port field information.

5. The method for port scan detection using server logs as claimed in claim 1, wherein, In the step 200, it is judged whether the number of records corresponding to each IP pair recorded in the positive sample database is greater than a predetermined number, which is the minimum sample number capable of applying the clustering method.

6. A computer device, characterized by: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method for port scan detection using server logs according to any one of claims 1-5 when executing the computer program.

7. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that: The computer program / instructions implement the steps of the method for port scan detection using server logs according to any one of claims 1-5 when executed by the processor.

8. A system for port scan detection using server logs implemented by the method of claim 1, wherein, The computer device comprises a feature extraction module and a port scan detection module. The feature extraction module collects and aggregates port access logs on a server cluster, reads the aggregated port access logs, extracts part of field information from each access record to obtain a simplified description of the access record, the part of field information comprising access IP and server IP field information, counts all repeated simplified descriptions to calculate the frequency information of the repeated simplified descriptions, compresses all repeated simplified descriptions into one simplified description and the frequency information of the repeated simplified descriptions, aggregates all records containing the IP pair with the access IP and the server IP as the key and one access IP and one server IP as one IP pair, sorts the related records under each IP pair according to the time of access occurrence, sets a minimum aggregation granularity, regards records with an access occurrence time interval not exceeding the minimum aggregation granularity as part of one continuous access, combines all records of one continuous access to obtain the aggregated records of the access behavior history of the IP pair, and outputs the aggregated records of the access behavior history of the IP pair. The port scan detection module judges whether the number of records corresponding to each IP pair recorded in the positive sample database is greater than a predetermined number. If yes, a similarity matrix of port access frequencies of the current aggregated records and the records in the positive sample database is calculated, outliers are detected using a clustering method, and the aggregated records corresponding to the outliers are marked as port scans; otherwise, it is judged whether the current records are port scans according to rules written according to business logic. The port scan detection results are aggregated and submitted to an administrator for confirmation, the aggregated records confirmed by the administrator as port scans are removed, and the remaining aggregated records are added to the positive sample database.

Images