A method and system for network service discovery and analysis

Abstract

The present invention relates to a method and system for network service discovery and analysis, wherein the method comprises the following steps: generating a data set to be detected from Netflow data, and comparing the data to be detected with the detected data set; performing simulated detection on each record in the data set to be detected, so as to determine whether the network service is enabled; automatically analyzing the intercepted network service content, including clustering analysis of a large number of website screenshots obtained, and automatically dividing them into multiple types, using a deep learning model for modeling training, and thus using a new model to realize automatic analysis of website content, so as to determine the purpose of the website content and whether it is suspicious; combining the IP address and port of the relevant network service discovered in the data detection process, and providing data analysis and management on the premise of determining whether it is a normal service.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a method and system for discovering and analyzing network services. Background Art

[0002] Information systems play a key role in all walks of life. For example, in the medical field, they are used for patient medical record management and medical equipment monitoring, while in the financial field, they can be used to process various transactions, conduct risk assessments, and detect anomalies. With the rapid development of computer networks, information systems have gradually penetrated into the application scope of various fields, which can not only improve work efficiency and accuracy, but also have advantages in managing large amounts of data and conducting analysis. However, while bringing various conveniences, they also bring serious security risks, leading to the leakage of personal information, the theft of key business data, and even the hacking and control of hardware equipment, encryption of core files and extortion.

[0003] In order to eliminate the above-mentioned hidden dangers in advance and prevent them before they happen, the concept of vulnerability scanning was created. Vulnerability scanning can be regarded as an automated security testing method, which is mainly used to actively detect potential vulnerabilities in various information systems, hardware resources, networks, and applications through tools to improve security and reliability. Vulnerability scanning can be used to proactively discover security risks in a timely manner, and prevent hacker attacks and illegal intrusions through timely repairs, thereby ensuring the overall security of the information system. However, vulnerability scanning requires a range of sniffing detection for the scanned object, either scanning a port value range of a specified IP address one by one, or scanning an IP address segment under the specified port value. This will consume certain resources and a lot of time. If the port value range or IP address segment range is set unreasonably, either the security vulnerabilities that may be discovered will be missed (for example, the port or IP address occupied by the vulnerability is not within the scanning range), or a lot of computing resources will be consumed. The two are mutually exclusive and cannot be perfectly balanced. In fact, except for the extreme case of physical contact with information systems and related hardware devices, hackers usually remotely attack and invade information systems through the network, so it is a good way to detect hacker behavior in a timely manner through network traffic. Hackers who invade information systems often leave backdoors or traces for subsequent use. A large part of them will secretly open Web services. Therefore, it is a good prevention method to actively detect whether Web services are involved through the obtained network traffic, and then determine whether the service is a normal service application. This avoids the disadvantage of blindly scanning a large range of IP addresses and a large range of port numbers, which consumes a lot of resources, and also provides a good means for timely detection of abnormalities. Summary of the invention

[0004] The purpose of the present invention is to solve the above problems existing in the existing safety testing technology.

[0005] In order to achieve the above object, on the one hand, the present invention provides a method for network service discovery and analysis, comprising the following steps:

[0006] Generate the data set to be detected from Netflow data;

[0007] Perform simulated detection on each record in the data set to be detected to determine whether the network service is enabled;

[0008] Automatically analyze the intercepted network service content, including clustering analysis of the numerous website screenshots obtained, and automatically classifying them into multiple types, using deep learning models for modeling training, so as to use the new model to realize automatic analysis of website content, so as to determine the purpose of the website content and whether it is suspicious;

[0009] The IP addresses and ports of related network services discovered during the data detection process are combined to provide more detailed data analysis and management based on whether it is a normal service.

[0010] In another aspect, the present invention provides a system for discovering and analyzing network services, comprising:

[0011] The first processing module is used to generate a data set to be detected from the Netflow data, and compare the data to be detected with the detected data set;

[0012] The second processing module is used to simulate detection of each record in the data set to be detected, so as to determine whether the network service is enabled;

[0013] The third processing module automatically analyzes the intercepted network service content, including clustering analysis of the numerous website screenshots obtained, and automatically classifying them into multiple types, using deep learning models for modeling training, so as to use the new model to realize automatic analysis of website content, so as to determine the purpose of the website content and whether it is suspicious;

[0014] The fourth processing module is used to combine the IP addresses and ports of related network services discovered during the data detection process to provide more sophisticated data analysis and management based on the premise of determining whether it is a normal service.

[0015] The present invention provides a method and system for automatic website discovery and analysis, which periodically extracts IP address and port number combinations from Netflow data, detects newly appeared IP addresses and port numbers after removing the detected combination data, determines whether network services are provided based on the feedback results, and then determines whether the service is normal or abnormal through methods such as flow analysis. Therefore, abnormal network services can be discovered in a timely manner, thereby improving the overall security of related information systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a method flow for network service discovery and analysis provided by an embodiment of the present invention;

[0017] Figure 2 for Figure 1 A schematic diagram of the address-based data comparison method in the method shown;

[0018] Figure 3 A schematic diagram of the system structure of network service discovery and analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0021] Figure 1 A schematic diagram of a method flow for network service discovery and analysis provided by an embodiment of the present invention. Figure 1 As shown in FIG. 1 , the method mainly includes five parts. First, the data to be detected is generated and compared and screened; second, the use of network services is simulated as a user to determine whether the network service is enabled; after the network service is discovered, data analysis and traffic statistics are used to analyze whether the service is suspicious; finally, the discovered network service and analysis results are presented to the user for manual verification. Specifically:

[0022] The first part is the generation and comparison of the data to be detected. It includes two basic functions:

[0023] 1) Generation of data to be detected: Generate the data set to be detected from Netflow data. A Netflow data includes the source IP address, source port and destination IP address, destination port, so this Netflow data can be split into two records: source IP address + source port and destination IP address + destination port.

[0024] 2) Comparison of data to be detected: The data to be detected is compared with the detected data set (i.e., the data set that has been detected within a certain period of time recently) through the candidate data comparison program. If the detected data set contains a certain IP address + port record, the record will not be re-detected.

[0025] From the above two basic functions, it can be seen that the first part is mainly used to extract IP address and port combinations from Netflow data, and exclude data records that have been detected recently to reduce the amount of data to be detected, and finally generate data records to be detected.

[0026] Specifically, the Netflow data extraction and screening method can quickly extract the data to be detected from the Netflow data and perform comparative screening (eliminating the IP address and port combination that has been detected before). The steps include:

[0027] 1) Netflow data extraction: Each Netflow data is split into two records, namely, source IP address + source port, and destination IP address + destination port, thus forming a data set to be detected.

[0028] 2) Data screening: Netflow data volume is huge, and each Netflow data is split into two pieces of data to be detected, and each record needs to detect different network services separately, such as detecting whether web services are provided in two different protocol modes, http and https, detecting whether ftp services are provided in ftp protocol mode; trying to log in in SSH mode to determine whether SSH services are provided, etc., which causes the actual number of detections to become several times the combination of IP addresses and ports, bringing huge pressure to subsequent data detection work. Because each detection must wait for the other party to give feedback, this period may take several seconds or even longer, and the time spent on comparing the similarity between the two pieces of data is likely to be less than the detection time. Therefore, it is very necessary to eliminate the data that has been detected within a certain period of time before, and only need to detect the newly appeared data records. To this end, it is necessary to develop a candidate data comparison program to quickly compare the detected data set (that is, the data set formed by the combination of IP addresses and ports that have been detected recently) and the newly appeared data to be detected with the detected data set. If the record exists in the detected data set, the data record will be removed from the data set to be detected, thereby obtaining a new data record set that has not been detected before (called the data set to be detected).

[0029] The purpose of forming the above data set and comparing it with the data to be detected is to determine whether the data to be detected needs to be detected. If the comparison time exceeds the actual detection time, then the above design is completely unnecessary and even counterproductive. This puts forward strict requirements on how to improve the efficiency of data comparison. The embodiment of the present invention proposes an address-based data comparison method, which can significantly shorten the time consumed by data comparison, thereby improving the overall operation efficiency.

[0030] In detail, the embodiment of the present invention is based on the idea of ​​exchanging space for time and proposes an address-based data comparison method as follows:

[0031] Because the data to be compared is a combination of IP addresses and ports, if the detected data set is traversed sequentially, the time complexity of each comparison is O(n). If a four-dimensional array is constructed with the IP address as the array dimension, each dimension represents 8 bits in the IP address. For example, if the IP address is 192.168.1.12 and the port number is 110, the corresponding storage location is data

[192]

[168] [1]

[12] , and the value of this location is 110. If the IP address has multiple port numbers that need to be detected, for example, the port numbers are 110, 220, and 80, then the value of this location is 110, 220, 80. When comparing, you only need to compare the value of this location, thereby using the characteristics of the IP address to significantly reduce the amount of data comparison calculations, and the time complexity of each comparison is reduced to O(1).

[0032] In addition, the embodiment of the present invention proposes a network service detection process based on an IP address and a port combination, which is applicable to a variety of network service discoveries, such as web services, FTP services, SSH services, DNS services, etc. Taking the web service detection process as an example, it includes the following steps:

[0033] 1) Extract a data record from the dataset to be detected.

[0034] 2) Then use the IP address and port detection program to simulate the operation of ordinary users directly accessing the URL address composed of the IP address and port (in two different protocol modes, http and https).

[0035] 3) Determine whether the URL provides normal Web services based on the information fed back by the other party.

[0036] 4) If the web service is normal, the IP and port combination plus the protocol type (i.e. http or https) is stored in the network service data set. The web page is automatically screenshotted and saved in the web page screenshot data set. Then jump to the first step to obtain a new piece of data for detection.

[0037] 5) If the Web service is not provided, jump to the first step to obtain a new piece of data for detection.

[0038] From the above steps, it can be seen that no matter whether the URL provides Web services or not, a new piece of data will be extracted for detection, and the process will be repeated until all data records in the data set to be detected are detected. Then it will sleep and wait for new data to be detected, and then repeat the above process.

[0039] The process of detecting other network services is similar to this, because each type of network service detection extracts records from the data set to be detected, and if the service is detected, it is written into the network service data set. It can be seen that network service detection and context interaction are both carried out through the data set, which has nothing to do with the specific implementation of how to detect internally. Therefore, this low-coupling method is convenient for dynamically increasing the types of network services and integrating detection methods.

[0040] The second part is data detection, or network service discovery. That is, simulate detection of each record in the detection data set as a user. For example: simulate the operation of accessing a web page as an ordinary user, check the feedback from the other party to determine whether Web services are provided; try to connect as an FTP client, check the feedback from the other party to determine whether FTP services are provided; try to log in as SSH, and determine whether SSH services are provided at the IP and port combination based on the feedback; try to perform domain name resolution with the nslookup command, and determine whether DNS services are provided at the IP address and port based on the feedback.

[0041] The third part is service content analysis. This part mainly takes Web services as an example, that is, clustering analysis is performed on the numerous website screenshots obtained, and they are automatically divided into multiple types. The deep learning model is used for modeling training, so that the new model can be used to automatically analyze the website content, so as to determine the purpose of the website content and whether it is compliant and legal, thereby greatly reducing the manual workload.

[0042] Specifically, the website content captured by automatic detection will result in an increasing number of website screenshots as the program continues to run, which will lead to an increasing amount of time spent on manual review. Therefore, it is very necessary to conduct automatic analysis on website screenshot data, so that the automatically classified data can be quickly browsed, and the unclassified isolated point images can be focused on, because they may be new classes or anomalies, thereby significantly improving work efficiency.

[0043] The steps of the embodiment of the present invention are as follows: firstly, extracting image feature vectors by means of a deep learning image model (such as Resnet50, etc.) to form an image feature vector set;

[0044] Secondly, cluster analysis is carried out on the image feature vector set to calculate the similarity between each image and divide them into several major types, thereby automatically generating multiple classified sample data sets.

[0045] Then after manual review, especially finding outliers, we determine whether it is abnormal. For the classification of normal webpage screenshots, we extract their feature vectors to form a webpage screenshot classification feature vector set, and compare the feature vectors of newly generated website screenshots with them to determine whether they belong to the known webpage screenshot classification. If new outliers are found (i.e. new images that do not belong to the known webpage screenshot classification), we manually determine and consider forming a new classification.

[0046] It should be pointed out that when comparing web page text, many web pages display dynamic information such as IP addresses and timestamps, and such information will not be well compared by text comparison. However, the method of extracting image features and then clustering can avoid the difference in the specific content of the text to a certain extent.

[0047] The fourth part is Netflow traffic analysis. Through the above data detection process, the IP address and port combination that provides relevant network services can be found. On the premise of determining whether it is a normal service, more detailed data analysis and management can be provided for these IP addresses, so as to find abnormal situations more timely and take countermeasures as soon as possible.

[0048] Specifically, the embodiment of the present invention combines multiple traffic statistics methods by combined voting to find abnormal traffic as much as possible. The steps include:

[0049] First, through the time series analysis of the total traffic of the IP address, it is found that the traffic volume suddenly increases or decreases suspiciously;

[0050] Secondly, through the time series analysis of the traffic providing network services, suspicious situations of the network service traffic are discovered;

[0051] Once again, the source of the visit is determined by the visiting IP address, such as whether it is a foreign IP or a designated blacklist, so that suspicious situations can be discovered as soon as possible.

[0052] Through the above-mentioned multi-layer judgment, suspicious situations can be discovered in a timely manner and summarized and presented to users for judgment and analysis.

[0053] The fifth part is user authentication, which is to present the newly discovered IP address and port combination records of network services, as well as the analysis of their traffic and content, to the user, who will judge whether the combination record is confirmed to be voluntarily opened, otherwise it is highly suspected that the service is passively opened due to an attack. For services that are confirmed to be voluntarily opened, the record will be stored in the authenticated IP address and port combination; otherwise, the user is advised to take protective measures such as disabling to correct it.

[0054] Figure 2 for Figure 1 Schematic diagram of the address-based data comparison method in the method shown. Figure 2 As shown, the method steps include:

[0055] When the system starts, a four-dimensional matrix array dataset is first constructed. The value of each dimension ranges from 0 to 255, corresponding to an 8-bit value in the IP address.

[0056] For example, if the IP address is 192.168.1.12 and the port number is 110, the corresponding storage location is dataset

[192]

[168] [1]

[12] (for ease of explanation, this location value is called data).

[0057] First check the data. If the port number to be detected (110 here) is not found, modify the data value, append (110, timestamp) to the existing data value, and indicate that the record needs to be detected this time, where the timestamp is the current timestamp.

[0058] On the contrary, if the port number to be detected is found (i.e. 110 here), the time interval between the timestamp and the current time is determined. If it is less than the specified threshold, it means that the port has been detected recently and does not need to be detected this time; otherwise, it means that the port has not been detected for more than the specified time interval and needs to be detected this time. Regardless of whether the port needs to be re-detected, the timestamp corresponding to the port needs to be modified to the current timestamp.

[0059] Accordingly, an embodiment of the present invention provides a system for network service discovery and analysis. Figure 3 As shown, the system includes:

[0060] The first processing module 31 is used to generate a data set to be detected from the Netflow data, and compare the data to be detected with the detected data set;

[0061] The second processing module 32 is used to simulate detection of each record in the data set to be detected, so as to determine whether the network service is enabled;

[0062] The third processing module 33 automatically analyzes the intercepted network service content, including clustering analysis of the numerous website screenshots obtained, and automatically classifying them into multiple types, using a deep learning model for modeling training, so as to use the new model to realize automatic analysis of website content, so as to determine the purpose of the website content and whether it is suspicious;

[0063] The fourth processing module 34 is used to combine the IP address and port of the relevant network service discovered during the data detection process, and provide more sophisticated data analysis and management on the premise of determining whether it is a normal service.

[0064] The embodiment of the present invention provides a method and system for automatic website discovery and analysis, which periodically extracts IP address and port number combinations from Netflow data, detects newly appeared IP addresses and port numbers after removing the detected combination data, determines whether network services are provided based on the feedback results, and then determines whether the service is normal or abnormal through methods such as flow analysis. Therefore, abnormal network services can be discovered in a timely manner, thereby improving the overall security of related information systems.

[0065] It is obvious that many variations of the invention described herein are possible without departing from the true spirit and scope of the invention. Therefore, all modifications obvious to those skilled in the art should be included within the scope of the claims. The scope of the invention is limited only by the claims.

Claims

1. A method for network service discovery and analysis, characterized in that: The following steps are involved: Generate the data set to be detected from Netflow data; Perform simulated detection on each record in the data set to be detected to determine whether the network service is enabled; Automatically analyze the intercepted network service content, including clustering analysis of the numerous website screenshots obtained, and automatically classifying them into multiple types, using deep learning models for modeling training, so as to use the new model to realize automatic analysis of website content, so as to determine the purpose of the website content and whether it is suspicious; Discover the IP address and port combination records of the opened network services through the data detection process; provide data analysis and management based on the premise of determining whether the opened network services are suspicious; The step of generating a data set to be detected from Netflow data includes: Generate a data set to be detected from Netflow data, where one Netflow data is split into two records: source IP address and source port, and destination IP address and destination port; Compare the data to be detected with the detected data set. If the detected data set contains a certain IP address and port record, the record will not be re-detected; The step of comparing the data to be detected with the detected data set includes: Adopting the address-based data comparison method, a four-dimensional array is constructed with the IP address as the array dimension, and each dimension represents 8 bits in the IP address; The address-based data comparison method steps include: Construct a four-dimensional matrix array dataset, where the value of each dimension is 0 to 255, corresponding to an 8-bit value in the IP address, and the corresponding storage location is data; Check the data. If the port number to be detected is not found, modify the data value, append the port number and timestamp to the existing data value, and indicate that the record needs to be detected this time. The timestamp is the current timestamp. On the contrary, if the port number to be detected is found, the time interval between the timestamp and the current time is determined. If it is less than the specified threshold, it means that the port has been detected recently and does not need to be detected this time; otherwise, it means that the port has not been detected for more than the specified time interval and needs to be detected this time.

2. The method according to claim 1, characterized in that: The step of performing simulation detection on each record in the data set to be detected includes: Extract a data record from the dataset to be detected; Use IP address and port detection program to simulate the operation of ordinary users directly accessing the web browser to access the URL address composed of the IP address and port; Determine whether the URL provides normal Web services based on the feedback information; If normal Web services are provided, the IP and port combination plus the protocol type are stored in the network service data set; the web page is automatically screenshotted and saved in the web page screenshot data set, and then the first step is jumped to obtain a new piece of data for detection; If the Web service is not provided, jump to the first step to obtain a new piece of data for detection.

3. The method according to claim 1, characterized in that: The step of automatically analyzing the intercepted network service content includes: Using data mining algorithms, the similarities between images are calculated and divided into several categories, thus automatically generating multiple classified sample data sets; Use deep learning neural convolutional networks to train image recognition models, and eventually use the image recognition models to automatically analyze and process more and more website screenshots; Ultimately, the website analysis can be automated and analysis reports can be automatically generated periodically.

4. The method according to claim 1, characterized in that: The method also includes the steps of combining multiple traffic counting methods by combined voting, including: First, through the time series analysis of the total traffic of the IP address, it is found that the traffic volume suddenly increases or decreases suspiciously; Secondly, through the time series analysis of the traffic providing network services, suspicious situations of the network service traffic are discovered; Once again, determine the source of access through the visiting IP address, so as to detect suspicious situations as soon as possible; Through multi-layer judgment, suspicious situations can be discovered in a timely manner and summarized and presented to users for judgment and analysis.

5. The method according to claim 1, characterized in that The following steps are also included: The newly discovered IP address and port combination providing network services is recorded, and the analysis of its traffic and content is presented to the user.

Images