An active internet exposure surface detection method and system

Abstract

The application provides an active Internet exposure surface detection method and system, relates to the technical field of network and security technology, and the method comprises the following steps: generating internal asset data set comprising asset management information and network configuration information and unknown exposure surface keyword data set comprising security control information; according to the internal asset data set and the unknown exposure surface keyword data set, the Internet data source is grabbed through a crawler program, and Internet exposure surface information is extracted; the Internet exposure surface information is effectively verified through a network space mapping engine, the verified Internet exposure surface information is input into a local AI identification model, the risk level and the correlation of the Internet exposure surface information are calculated and sorted, and reliable Internet exposure surface information is generated. The application can automatically, multi-angle and long-time detect unknown exposure surfaces on the Internet that may be associated with enterprises, sort out exposure surface asset information related to enterprises, has a low error rate and high real-time performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network technology and security technology, and particularly relates to an active Internet exposure surface detection method and system. BACKGROUND

[0002] With the continuous development of the Internet, the application of digital technologies such as 5G and cloud computing, and the rapid growth of enterprise asset access to the network, it means that the risk exposure surface of enterprises will grow and become more complex. At the same time, network attack technologies continue to evolve, and attack teams are professional, organized, and tool-based, which makes the network security management of enterprises face more serious pressure. At present, the management of the Internet exposure surface in most enterprises is still in the passive management mode of using internal asset account records and regular manual review, which has the following problems:

[0003] 1. Asset exposure surface management: With the rapid growth of enterprise asset access to the network, how to effectively manage and reduce the exposure surface of enterprises on the Internet to reduce potential network security risks.

[0004] 2. Complexity of network security risks: The continuous evolution of network attack technologies and the professionalization, organization, and toolization of attack teams increase the pressure on enterprise network security management, and some fake shell websites and information leakage caused by non-enterprise subjective factors.

[0005] 3. Limitations of internal management methods: Most enterprises still rely on traditional internal asset account records and manual review methods, which have problems such as outdated updates, missed reviews, inability to fully investigate and trace, etc.

[0006] 4. Passive security protection: Enterprises often adopt passive defense strategies in network security protection, lack the ability to actively discover and respond to security threats, and manual methods cannot spend a lot of resources to fully investigate and trace small-scale leaks of platform entity accounts, high-risk ports, and sensitive information. Some test systems and edge systems are less concerned, and administrators cannot learn about the exposure surface of such systems in a timely manner, resulting in a lack of appropriate monitoring means. Due to their special nature, the security reinforcement of such systems is generally low, and they are prone to various vulnerability risks. The Internet exposure surface investigation service provided by security vendors has good results, but the price is generally high. Enterprises generally only use such solutions as a one-time full-scale Internet exposure surface investigation solution in the short term, which is difficult to solve the long-term Internet exposure surface problem of enterprises.

[0007] 5. Combination of technology and management: Enterprises need to find a balance between technical means and management measures to achieve more effective network security management. Asset account records are not updated in a timely manner, and administrators have missed review problems.

[0008] To address the above issues, enterprises need to take a series of measures, including but not limited to automated asset discovery and management, strengthening security monitoring and response, improving security protection capabilities, strengthening compliance and best practices, improving security awareness and training, using professional security services, establishing emergency response mechanisms, implementing risk management, and using threat intelligence, etc. to build a more comprehensive and effective network security management system. Therefore, there is an urgent need for a technical solution that can actively detect Internet exposure. SUMMARY

[0009] The present application shows an active Internet exposure detection method and system.

[0010] In a first aspect, the present application shows an active Internet exposure detection method, comprising:

[0011] Obtain enterprise local asset management information, network configuration information and security control information, and perform cleaning and sorting operations to generate internal asset dataset including asset management information and network configuration information and unknown exposure key word dataset including security control information;

[0012] According to the internal asset dataset and the unknown exposure key word dataset, the Internet data source is grabbed by the crawler program, and the Internet exposure information is extracted;

[0013] The Internet exposure information is verified, the verified Internet exposure information is input into the local AI recognition model, the internal asset dataset and the unknown exposure key word dataset are combined, the data cleaning and risk assessment are performed, the risk level and the correlation of the Internet exposure information are calculated and sorted by the local AI recognition model, and the reliable Internet exposure information is generated.

[0014] Further, the crawler program uses Selenium tool to render the webpage, and according to the type of the accessed webpage, the Internet data source is grabbed by static page element positioning and simulated webpage automatic clicking.

[0015] Further, the crawler program uses proxy IP pool and browser header address pool for data crawling.

[0016] Further, the extraction of the Internet exposure information includes: interfacing with a spatial mapping engine to collect vulnerability data and threat intelligence information of the Internet exposure.

[0017] Further, the local AI is obtained by training the internal asset dataset and the unknown exposure key word dataset.

[0018] Further, the local AI recognition model is trained through reinforcement learning and reward modeling, and the local AI recognition model is optimized through a PPO algorithm.

[0019] Further, the risk level and relevance of the internet exposure surface information calculated and sorted by the local AI recognition model include: the local AI recognition model assigns weights and sorts the risk level and relevance of the internet exposure surface information according to vulnerability data and threat intelligence information of the internet exposure surface information, and generates a risk report of the internet exposure surface information.

[0020] In a second aspect, the present application shows an active internet exposure surface detection system, characterized in that the system comprises:

[0021] An information acquisition module is configured to acquire asset management information, network configuration information and security control information of an enterprise, and perform cleaning and sorting operations to generate an internal asset dataset including the asset management information and the network configuration information and an unknown exposure surface keyword dataset including the security control information.

[0022] A crawling module is configured to extract internet exposure surface information by crawling an internet data source through a crawler according to the internal asset dataset and the unknown exposure surface keyword dataset.

[0023] A detection module is configured to perform effective verification on the internet exposure surface information through a network space mapping engine, input the verified internet exposure surface information into a local AI recognition model, combine the internal asset dataset and the unknown exposure surface keyword dataset to perform data cleaning and risk assessment, calculate and sort the risk level and relevance of the internet exposure surface information through the local AI recognition model, and generate credible internet exposure surface information.

[0024] The technical solution provided by the present application can include the following beneficial effects:

[0025] The technical solution of the present application performs effective verification on the internet exposure surface information through a network space mapping engine, inputs the verified internet exposure surface information into a local AI recognition model, and combines automatic data collection, advanced crawler technology, AI recognition model and risk assessment method to significantly improve the efficiency, accuracy and security of enterprise internet exposure surface detection. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a step flowchart of the method of the present application;

[0027] Figure 2 is a method flowchart of the first embodiment of the present application;

[0028] Figure 3is an example flowchart of embodiment two of the present application;

[0029] Figure 4 is a structural block diagram of an active Internet exposure surface detection system of the present application;

[0030] Figure 5 is a block diagram of an electronic device of the present application;

[0031] Figure 6 is a block diagram of a computer readable storage medium of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0033] Noun explanation:

[0034] Web crawler: refers to a program that can automatically collect and crawl information in a specific webpage on the Internet according to certain rules after being written by a person.

[0035] Internet exposure surface: refers to domain names, IPs, websites, entity accounts, application systems, high-risk ports, source codes, file data, etc. exposed in the Internet, which can be used or invaded by attackers, such as systems, devices, information, etc. owned or controlled by the operating unit.

[0036] Network space mapping engine: a site or application that detects, collects, analyzes, and processes global network assets in real time to obtain various attributes of the assets in the network space and draw these attribute states.

[0037] AI recognition model: refers to an open source AI model that can automatically obtain Internet exposure surface information and make judgments after local data set fine-tuning training and function development.

[0038] Selenium: Selenium is a tool for automated testing of web applications. It can simulate user operations in the browser, such as clicking, inputting, etc., to detect the functions and performance of the application. Selenium originated in 2004 as a JavaScript-based testing tool called Selenium Core. Later, with the development of technology, Selenium has undergone multiple iterations and improvements, and finally formed the current Selenium WebDriver, also known as Selenium.

[0039] Headless Chrome: Headless Chrome is a headless mode of the Chrome browser, which means running Chrome without a graphical user interface. This mode allows users to run programs using all the features supported by Chrome without opening the browser's GUI, making it particularly suitable for scenarios such as automated testing, web scraping, and server-side tasks.

[0040] PPO Algorithm: PPO (Proximal Policy Optimization) is a reinforcement learning algorithm proposed by OpenAI in 2017. PPO aims to solve the instability of training caused by excessive policy updates in traditional policy gradient methods. By introducing a mechanism to limit the range of policy updates, PPO ensures convergence while improving the stability and efficiency of the algorithm.

[0041] request Library: The request library is a very popular Python HTTP third-party library, mainly used to send various HTTP requests, handle cookies, sessions, connection pools, redirects, multiple authentication methods, etc. It makes it very convenient to handle HTTP requests and is commonly used in crawlers and network requests.

[0042] session: In databases, session refers to an interaction process between a user and a database server, starting from the user successfully connecting to the database server and ending when the user disconnects. Session is mainly used to identify and manage the interaction process between the user and the database, and the user can perform various database operations such as query, insert, update and delete in the session. The database server will record all operations in the session and use it to recover data when necessary.

[0043] JavaScript: JavaScript (JS) is a lightweight interpreted or just-in-time compiled programming language, mainly used for web development to enhance the interactivity and dynamic nature of web pages.

[0044] rabbitMQ: RabbitMQ is an open-source message middleware developed in Erlang language, based on the AMQP (Advanced Message Queuing Protocol) protocol. It is mainly used for asynchronous communication between applications, especially in distributed systems and microservice architectures. RabbitMQ can help solve message delivery problems, improve system scalability, reliability and response speed.

[0045] SHAP: SHAP (SHapley Additive exPlanations) is a tool for explaining the output of machine learning models. It quantifies the contribution of each feature to the model's prediction based on the Shapley value in game theory. The core idea of SHAP is to calculate the marginal contribution of each feature to the model's output, thereby explaining the "black box model" from both global and local perspectives.

[0046] URL: Uniform Resource Locator (URL) is a string used to identify and locate resources on the Internet. Each information resource on the Internet has a unique address, which is the URL. It not only indicates the location of the resource, but also provides the information required to access the resource, such as protocol type, server address, port number and specific location of the resource on the server.

[0047] fake_useragent: is a Python library specifically designed to generate fake User-Agent strings. User-Agent is part of an HTTP request, used to help servers identify the type of browser, version and operating system of the client making the request. In some cases, such as in the development of crawlers, faking User-Agent can simulate different browser environments, thereby avoiding being identified and blocked by websites.

[0048] SSL: SSL (Secure Sockets Layer) is a security protocol designed to protect data transmission security in Internet communication, providing encryption, identity verification and data integrity protection. SSL protocol establishes an encrypted link between the client and server to ensure that the transmitted data cannot be intercepted or tampered with by unauthorized third parties. It establishes a session between the client and server through a handshake process, completing the verification of the identity of both parties, the negotiation of keys and encryption suites, thereby ensuring the security of communication.

[0049] json: (JavaScript Object Notation) is a lightweight data exchange format widely used in Web development for data interfaces, serialization, Token generation and configuration files. It is designed based on a subset of ECMAScript and is independent of language, supporting multiple programming languages. etree: is a library written in python language, used to parse tree type element files.

[0050] Example 1:

[0051] Reference Figure 1, shows an active Internet exposure surface detection method of the application, which can be applied to an electronic device, and the method comprises the following steps:

[0052] Step 101, obtain enterprise local asset management information, network configuration information and security control information, and perform cleaning and sorting operations to generate an internal asset dataset including asset management information and network configuration information and an unknown exposure surface keyword dataset including security control information;

[0053] Step 102, according to the internal asset dataset and the unknown exposure surface keyword dataset, the Internet data source is grabbed by a crawler program to extract the Internet exposure surface information. For example, the corresponding network crawler program can be generated for the regular search engine, the network disk search, the file resource search, the public number, the applet and other network information sources to extract the Internet exposure surface information.

[0054] Specifically, the network crawler program should be classified according to the desktop and the mobile terminal, and can be written in static page element positioning, simulated web page automatic clicking and other ways. The page elements positioned by the crawler should be text characters or double-element positioning mode as much as possible to ensure the uniqueness of the elements. For the web pages with login verification or requiring page operation, the headless browser setting and simulated web page clicking method are used for data submission and page operation to reduce the number of triggering verification codes. For the relatively simple numerical, alphabetical, sliding and clicking verification code types, the optical character recognition can be used to process and recognize the verification code picture, or the return data packet can be intercepted to obtain the verification code, or the third-party verification code recognition platform proxy can be used to recognize the verification code. The crawler program includes the combination of Selenium and Headless Chrome and supports JavaScript rendering to ensure the integrity and accuracy of the Internet data source. According to the demand, the proxy IP pool provided by the cloud vendor and the platform is grabbed, the session sub-library of the request library is used to write test functions to test the IP pool for survival and quality, and the IP with high quality is collected to build the proxy address pool of the crawler. When data crawling is performed, the proxy address pool and the browser head address pool are randomly used for data crawling to reduce the frequency of anti-crawling mechanism triggered by the website when the number of single IP access is too high.

[0055] The key code blocks of element positioning, login verification, data acquisition, and data submission of the network crawler program are modularized. According to the types of accessed web pages, the items that change, such as positioning elements, unknown exposure face keyword data, and submission data types, are established in a redundant pool. Different modules of the network crawler program are written in a classified manner to ensure the stability and reliability of the network crawler program. By adding a message sorting queue middleware such as rabbitMQ or implementing an asynchronous message queue in the code, the request sending and data analysis stages of the crawler are implemented using multi-threading and thread locking technology to achieve multi-threaded crawling of multiple websites. The data crawling and processing stages are asynchronously received through the message queue method, which can significantly reduce the network socket demand during the crawler runtime. In the event of an error due to anti-crawling mechanisms or other reasons, the data obtained by the crawler before the exception can be effectively preserved.

[0056] In step 103, the network programming technology is used to verify the Internet exposure face information through the network space mapping engine of major security vendors. The verified Internet exposure face information is input into the local AI recognition model, combined with the internal asset data set and the unknown exposure face keyword data set, for data cleaning and risk assessment. The risk level and relevance of the Internet exposure face information are calculated and sorted by the local AI recognition model to generate reliable Internet exposure face information, thereby completing the Internet exposure face detection.

[0057] Specifically, a proactive security scanning program is established using programming technology to perform security scanning on unknown Internet exposure faces to collect Web application and framework, middleware, subdomain name, API information, SSL / TLS certificate, port, C segment, vulnerability, CDN, and threat intelligence information. The interface is connected with the network space mapping engine to obtain vulnerability data and threat intelligence information.

[0058] The network space mapping engine is called to collect information such as keywords, icons, return data, domain name certificates, and record nature that may appear on assets. Enterprise information search engines and other information are accessed to obtain possible enterprise information data.

[0059] Specifically, the local AI recognition model is trained using the internal asset data set and the unknown exposure face keyword data set to improve the discrimination accuracy of the Internet exposure face information. The local AI recognition model is trained through reinforcement learning and reward modeling, and optimized through the PPO algorithm to improve the risk assessment accuracy of the Internet exposure face information. The risk level and relevance of the Internet exposure face information are calculated and sorted by the local AI recognition model, including: the local AI recognition model assigns weights and sorts the risk level and relevance of the Internet exposure face information based on the vulnerability data and threat intelligence information of the Internet exposure face information, and generates a risk report of the Internet exposure face information.

[0060] Specifically, the AI recognition module is developed and externally connected with function calls to realize the calling of the network crawler program and the spatial mapping calling program. The main process is that the program provides a set of calling functions to the AI recognition model, the AI recognition model selects to use or not to use or is forced to use the function according to the administrator selection, the application obtains the result according to the function and provides it to the AI recognition model. The AI recognition model can continuously use the function call and new data as the training data set, and after supervised control training, artificial setting of preference and weight, online learning and continuous updating of AI are realized to ensure the stability and availability of AI.

[0061] Specifically, the SHAP model explanation technology is used to analyze the decision-making process of the AI recognition model, track and graph the feature points of the Internet exposure surface data set, and reduce the influence of feature points with low exposure surface correlation on the average prediction value. For example, some system keywords have a greater influence on the average prediction value, but the actual importance is low, and after multiple feature accumulations, the data discrimination is seriously affected.

[0062] The present application can be applied to network and system security system analysis scenarios, especially in the Internet exposure surface risk assessment and security protection scenarios. The Internet exposure surface refers to domain names, IPs, websites, entity accounts, application systems, high-risk ports, source codes, file data, and other data exposed to the Internet and running or remaining, which can be detected by attackers or intruded into systems, devices, information, etc.

[0063] The method and system of the present application are suitable for Internet asset exposure surface management of medium and large enterprises. Based on network crawler and network programming technology, combined with multiple detection technologies, the unknown exposure surface that may be associated with the enterprise on the Internet is automatically detected from multiple angles and for a long time. Through algorithmic data analysis of the unknown exposure surface, the exposure surface asset information related to the enterprise is sorted out, the error rate is low, the real-time performance is high, and the time for manual sorting and checking can be greatly reduced.

[0064] The embodiments of the present application will be described in detail below with reference to the accompanying drawings:

[0065] Embodiment 2:

[0066] The method of the present application is solidified in software form in computer and other physical products, Figure 2 The method flowchart of the first embodiment of the present application.

[0067] Step 1, obtain internal asset source data and unknown exposure surface keyword data. As described in step 1, the internal asset source data can include asset name, hardware information, IP address, port, domain name, service name, service type, software identification, software process, framework fingerprint, certificate file, file name, file type, entity account, and other asset data. The unknown exposure surface keyword data is enumerated and collected from the above internal asset source data in combination with data characteristics, which can include asset keywords, IP address C segment, asset high-frequency port, subdomain name dictionary, directory dictionary, high-frequency service interface, commonly used framework fingerprint, ICP filing name, file name keyword, entity account keyword, and other data. The detail and availability of the unknown exposure surface keyword data determine the effectiveness of the Internet exposure surface detection method. Due to differences in internal asset operating environment and main business, the source data collected is diversified and scattered, and the data collection process and rules need to be classified and solidified to improve the detail and availability of the unknown exposure surface keyword data.

[0068] Step 2, generate unknown exposure surface keyword data based on internal asset data set and asset exposure surface keywords; build a crawler program for search engines, file search, network disk search, public number, and other network information sources to collect unknown Internet exposure surface information. As described in step 2, due to the different types of websites to be crawled, code modularization should be performed when writing the crawler program, and element positioning should be encapsulated into a configuration file to reduce the workload of the crawler program in later rewriting.

[0069] The crawler should be implemented according to the following steps:

[0070] 1) Module split for the crawler, decompose URL management module, access module, parsing module, data storage module, scheduler module, anti-crawling mechanism processing module, exception handling module, configuration management module.

[0071] 2) The URL management module should classify different types of web pages, use fake_useragent and headless browser to set up a fake browser header, improve the success rate of URL connection, and reduce the probability of encountering SSL errors and anti-crawling measures.

[0072] 3) When building the access module, when the webpage is a static page, use request+bs4 to obtain data, when the webpage is a dynamic page, use selenium+request+bs4 to dynamically capture content after rendering the webpage, when the webpage is a heterogeneous data format (such as JSON, XML, etc.), use json, etree to extract data. When the webpage is an application, xray+selenium should be used to render the webpage and capture HTML (HyperText Markup Language), analyze API (Application Programming Interface) endpoints to obtain data.

[0073] 4) Use bs4 (Beautiful Soup 4, a Python library for parsing HTML and XML documents, mainly used for extracting data from web pages), lxml (a high-performance Python library) to parse the collected pages, define the extraction tags according to the collection requirements, and extract the test according to different web pages to generate the corresponding extraction strategy and logic, and label and format the data to ensure the availability and consistency of the data.

[0074] 5) The data storage module should use mysql (an open source relational database management system (RDBMS)) and other relational databases to store the extracted exposure data.

[0075] 6) The scheduler module should control the crawling order and frequency of the crawler, and the scheduling module should work together to achieve multi-threading and asynchronous crawling to improve the crawling rate. When writing multi-threading, attention should be paid to thread locks and thread apoptosis design to avoid thread preemption and thread deadlocks.

[0076] 7) The anti-crawling mechanism processing module mainly realizes user agent randomization, random waiting interval, Captcha (Completely Automated Public Turing test to tell Computers and Humans Apart) bypass, session access, some websites will limit user_agent and IP, use proxy IP address pool and random agent bypass, some websites will detect access frequency, when a large number of access in a short time, anti-crawling measures will be popped up, random waiting interval bypass, some websites will use Captcha for anti-crawling, you can use disable javascript, simulate clicks, and East, optical character recognition and connect third-party Captcha recognition to bypass.

[0077] 8) The exception handling module should include crawler log collection, performance monitoring and exception handling, where log collection records the running state of the crawler, collects crawler error information, performance monitoring monitors the crawling speed, success rate, occupancy rate, etc. of the crawler, and exception handling can adjust and alarm the crawler according to the exception type when the crawler appears an exception, reducing the interruption of the crawler exception.

[0078] 9) The configuration management module should use YAML (YAML Ain't a Markup Language, a format for expressing data serialization) and other formats to store the settings and parameters of the crawler, to reduce direct modification of the crawler source code and realize dynamic update of the crawler.

[0079] Step 3, build a web crawler program to actively perform network detection using unknown exposure face keyword data to obtain unknown exposure face information; build a space mapping call program to operate and read data from the space mapping engine to collect more related vulnerability data and threat intelligence information of unknown Internet exposure faces; as described in step 3, the space mapping call program should be able to call queries such as but not limited to the history of port opening, subdomain name, domain name record subject information, sensitive directory, framework fingerprint, enterprise information, and other vulnerability data of unknown exposure faces. When building this program, attention should be paid to the active security scanning program in the above optional items, and a program that can actively scan the Internet exposure face to obtain real-time exposure face vulnerability data should be built to greatly improve the real-time and effectiveness of Internet exposure face detection.

[0080] Step 4, use unknown exposure face information and unknown exposure face keyword data to call the space mapping engine to effectively verify and obtain more related vulnerability data;

[0081] Build an exposure face AI recognition program to clean and distinguish the obtained unknown Internet exposure face information and related vulnerability data to obtain associated Internet exposure face information; step 5, use the active security scanning program to perform security scanning on the unknown exposure face information and verify and supplement the related vulnerability data;

[0082] The Internet exposure face AI recognition program should be implemented by the following steps:

[0083] 1) Pre-training phase: the AI model uses qwen2 (a large language model developed by the Aliyun Tongyi Qianwen team), uses LLaMA-Factory (Large Language Model Factory) to process internal asset data sets and unknown exposure face keyword data sets, and sets the stage to pt to conform to the pre-training data set format.

[0084] 2) Post-training phase: After the pre-training phase, use supervised fine-tuning, set stage as sft, configure fine-tuning method as lora+, this phase according to the task requirements, separately to a data training, improve the identification quality of AI, at the same time, introduce RLHF (Reinforcement Learning from Human Feedback, Reinforcement Learning from Human Feedback) technology, evaluate and sort the output of pre-training by artificial, optimize the behavior of the model.

[0085] 3) Reward modeling phase: Set stage as rm, set data set as: input, good answer, bad answer, observe output result and training loss rate, modify data set preference, so that its output is more in line with the demand preference.

[0086] 4) Reinforcement learning phase: Use the score of the reward model as the reward signal, optimize the model through PPO algorithm, the AI model accepts the output of the pre-trained model as input, and its output as the input of the reward model, constantly adjust the model parameters, so that the model learns to approach human preference while not losing the original problem-solving ability.

[0087] 5) Evaluation and tuning: Provide test data set in independent test environment, evaluate its output result, adjust learning power, learning depth, exploration rate and other parameters, optimize model performance, ensure stable convergence of strategy and value function, combine model explanation, fine-tune unstable parameters, multiple iterations and improvements.

[0088] When building an AI recognition program, you should write a calling function that adapts the crawler program and the spatial mapping calling program. When the AI makes inferences, you can call the function to get the information you need. When building an AI recognition program, you should analyze the data set type of the exposed surface information you get, build corresponding regular expressions and scoring rewards, score and classify vulnerability data and Internet exposure keywords, and establish first, second, third, and other risk levels. For example, the first risk level should include file certificates, ICP filing names, asset keywords, IP addresses, file name keywords, and port options that have strong associations with internal asset data sources. The second risk level should include IP address C segment, entity account management detection, and fingerprint framework options that have weak associations with internal asset data sources. And so on.

[0089] Step 6, AI identifies unknown Internet exposure data and determines the relevance and risk level of unknown Internet exposure.

[0090] Wherein, the exposure surface AI recognition program is optionally constructed, data cleaning and discrimination are performed on the obtained unknown Internet exposure surface information and related vulnerability data, and associated Internet exposure surface information is obtained. Specifically, the SHAP model interpretation technology can be used to analyze the decision-making process of the AI recognition model, track and graph the feature points of the Internet exposure surface data set, and reduce the influence of feature points with low correlation to the exposure surface on the average prediction value. For example, some system keywords have a greater influence on the average prediction value, but their actual importance is low. After multiple feature accumulations, the data discrimination is seriously affected. For example, the more the keyword "telecommunications" appears, the higher the ranking of the exposure surface in the model after multiple feature accumulations. However, the shap value of other feature points is actually negative, which reduces the consistency of the model.

[0091] In summary, the technical solution of the present application significantly improves the efficiency, accuracy and security of enterprise Internet exposure surface detection by combining automatic data collection, advanced crawler technology, AI recognition model and risk assessment method, as follows:

[0092] Improve detection efficiency and accuracy: By automatically obtaining enterprise asset management information, network configuration information and security control information, and generating internal asset data set and unknown exposure surface keyword data set, the enterprise Internet exposure surface can be more efficiently managed and identified.

[0093] Enhance data integrity and accuracy: The crawler program combines Selenium and Headless Chrome, and supports JavaScript rendering, ensuring the integrity and accuracy of Internet data sources.

[0094] Improve data crawling efficiency: The crawler program uses static page element positioning and simulated web page automatic clicking to capture data according to the type of web page, and uses proxy IP pool and browser header address pool to reduce the triggering of anti-crawling mechanism, improving the efficient crawling of Internet data sources.

[0095] Strengthen vulnerability data and threat intelligence acquisition: Through the interface with the network space mapping engine, vulnerability data and threat intelligence information are obtained, enhancing the risk assessment of Internet exposure surface information.

[0096] Improve the discrimination accuracy of AI recognition model: The local AI recognition model is trained by the internal asset data set and the unknown exposure surface keyword data set, improving the discrimination accuracy of Internet exposure surface information.

[0097] Improve risk assessment accuracy: The local AI recognition model is trained by reinforcement learning and reward modeling, and optimized by PPO algorithm, improving the risk assessment accuracy of Internet exposure surface information.

[0098] Risk level and relevance assessment: The local AI recognition model can weight and rank the risk level and relevance of internet-exposed information based on vulnerability data and threat intelligence information, generate risk reports, and provide enterprises with more accurate risk management basis.

[0099] Reducing false positives and improving threat detection efficiency: The application of AI technology in network security can reduce false positives and improve the efficiency of threat detection, allowing security teams to focus more on real threats.

[0100] Save resources and improve test stability: Tests in Selenium Headless mode do not pop up a visible browser window and can run silently in the background, saving resources and improving test stability.

[0101] Improve network security protection capabilities: Through proactive Internet exposure detection, enterprises can more effectively identify and defend against network attacks, reduce security risks, and improve network security protection capabilities.

[0102] Example 3:

[0103] This embodiment is applicable to critical environments with multiple systems and assets. Systems, comprised of multiple servers, databases, network equipment, and related application software, are crucial enterprise assets, directly impacting the smooth operation of business operations and data security. Because these systems involve large amounts of sensitive data and are exposed to the internet, they are vulnerable to hacker attacks and vulnerability exploitation. Therefore, asset consolidation, exposure monitoring, vulnerability monitoring, and system hardening are essential for these systems. Figure 3 This is a schematic diagram of an example flow chart of the second embodiment.

[0104] As step 1, obtain internal asset types and information to generate internal resource data; specifically, obtain internal asset data and special exposure surface keywords through internal resource data sorting, including: obtaining enterprise local asset management information, network configuration information and security control information, which can use automated scanning tools to scan the entire network environment for information collection. Using the natural language processing capabilities in AI technology, automatically parse asset information (such as servers, databases, network devices, and applications) and extract detailed information, including: IP address, port number, operating system and version, installed software and version, device network topology relationship and connection method. This not only improves the speed of data collection, but also reduces errors caused by manual operation. Data integration and cleaning includes cleaning and organizing the collected information, and the cleaning and organizing operation specifically enters the information into the asset management system, applies data correlation analysis technology (such as graph database or relational database), and logically classifies and correlates the assets. Automatic classification of data not only depends on static characteristics, but also on dynamic information such as business traffic and access permissions, to achieve intelligent classification of assets. Asset classification uses Scikit-learn classification algorithms to classify assets according to their importance, such as "key assets", "secondary assets", "auxiliary assets", and mark assets that need to be protected, so as to prioritize processing in subsequent processes.

[0105] Generate internal asset data sets including asset management information and network configuration information, and unknown exposure surface keyword data sets including security control information; specifically, exposure surface keyword setting includes setting keywords related to enterprise asset systems, using spaCy natural language processing (NLP) technology, setting keywords related to enterprise asset systems, such as asset name, IP address, domain name, port number, service name, etc. Generate associated keywords for exposure surface monitoring. In order to more accurately monitor the Internet exposure surface, to ensure that all possible security risks are covered.

[0106] As step 2, unknown exposure surface keyword data is generated according to internal asset source data and asset exposure surface keywords; the security team monitors the Internet exposure surface through asset exposure surface keywords and configures monitoring tools (web crawler technology, search engine), uses web crawler technology, supports large-scale and efficient data crawling through module programming and distributed crawler technology, searches the Internet exposure surface, specifically: real-time monitoring using AI-driven web crawler technology, real-time monitoring of the Internet exposure surface, searching whether there are assets exposed on the Internet. Pay special attention to port exposure, disclosure of application version information and the state of open services. Exposure surface analysis uses ElasticSearch to compare with the enterprise internal asset account book to analyze whether the exposed assets are the assets actually used by the enterprise, combines the exposure surface monitoring result with the asset classification, uses ElasticSearch to risk assess the confirmed exposure situation to determine the possible security threats. Finally, generate an "Internet exposure surface monitoring report", use visual analysis tools (such as Grafana or Tableau) for display, make the report more intuitive and easy to understand, and propose targeted risk response strategies, including exposed asset list, exposed services, possible risk points and suggested countermeasures.

[0107] As step 3, build network monitoring program, interface security vendor vulnerability disclosure platform, intelligence threat platform data interface, get the latest vulnerability information and threat intelligence information; specifically, daily vulnerability information vulnerability disclosure monitoring and analysis, on the network of major security vendors vulnerability disclosure platform, intelligence threat platform interface docking, realize data call and read, get related vulnerability information and threat intelligence information; Among them, the vulnerability monitoring includes real-time monitoring of vulnerability database (such as CVE, CNVD) and 0day vulnerability disclosure platform, obtains the industry related vulnerability information, pays special attention to the high risk vulnerability of the operating system, database, application program used in the core system. AI model automatically filters and classifies vulnerability information to quickly identify the potential impact on enterprise assets. Vulnerability analysis is to use VulnWhisperer tool to compare the obtained vulnerability information with the data in enterprise asset management system, evaluate the impact of newly disclosed vulnerabilities on existing systems, and mark the affected assets. Vulnerability marking is to mark vulnerability points according to vulnerability information using Scikit-learn classification model, and generate vulnerability priority list. According to the harmfulness, exploitability, existence time and other factors of the vulnerability, the vulnerability is divided into high, medium and low priority. Impact assessment is to analyze the potential impact of each vulnerability, including data leakage, system downtime, unauthorized access, etc. Generate "vulnerability impact analysis report", including detailed vulnerability description, affected asset list, potential business impact and recommended repair strategy. As step 4, analyze the comparison of the obtained vulnerability information and asset management system data, evaluate the impact of newly disclosed vulnerabilities on existing systems, mark the affected assets, and classify high, medium and low priority.

[0108] Among them, the system administrator, security operation team, according to the vulnerability analysis report, enters the repair period, carries out vulnerability repair and reinforcement.

[0109] As step 5, carry out vulnerability repair and reinforcement, develop repair plan, arrange system maintenance window, allocate repair resources, and verify that the vulnerability has been repaired and the system is running stably; Specifically includes:

[0110] 1. Repair plan development: according to "vulnerability impact analysis report", through automatic patch management tool (such as Ansible, Chef), develop repair plan for different priority vulnerabilities, arrange system maintenance window, and ensure that repair process does not affect business continuity.

[0111] 2. Repair implementation: for high priority vulnerabilities, take immediate repair measures such as patching, updating system version, disabling or removing high-risk services. For medium and low priority vulnerabilities, repair in combination with system daily maintenance. In addition to repairing known vulnerabilities, system reinforcement measures such as multi-factor authentication, updating firewall strategy, deploying Web Application Firewall (WAF) are also implemented to further improve the overall protection capability of the system.

[0112] 3. Reinforcement measures: While repairing vulnerabilities, implement system reinforcement measures, such as deploying WAF (Web Application Firewall) to prevent common attacks. Increase multi-factor authentication and strengthen user access control. Update and reinforce firewall policies, close unnecessary ports and services. Implement regular security audits and penetration tests to discover new vulnerabilities in a timely manner. Use AI to monitor system behavior after implementation and identify abnormal activities through user behavior analysis.

[0113] 4. Verification of repair: After implementing repair and reinforcement measures, conduct system testing to ensure that vulnerabilities have been completely repaired and the system is stable. Record the repair process and results, and generate a "Repair and Reinforcement Verification Report".

[0114] As step 6, update the system asset information after repair and reinforcement to the asset management system, update the keywords and strategies of Internet exposure surface monitoring.

[0115] Specifically, it includes updating asset and monitoring data: updating the system asset information after repair and reinforcement to the asset management system. Update the keywords and strategies of Internet exposure surface monitoring to reflect the current system security status.

[0116] Example 4:

[0117] As Figure 4 , a proactive Internet exposure surface detection system is shown, which comprises:

[0118] An information acquisition module is used to acquire enterprise local asset management information, network configuration information and security control information, and perform cleaning and sorting operations to generate an internal asset dataset including asset management information and network configuration information, and an unknown exposure surface keyword dataset including security control information;

[0119] A crawling module is used to extract Internet exposure surface information by crawling Internet data sources through a crawler program according to the internal asset dataset and the unknown exposure surface keyword dataset;

[0120] A detection module is used to perform effective verification on the Internet exposure surface information through a network space mapping engine, input the verified Internet exposure surface information into a local AI recognition model, combine the internal asset dataset and the unknown exposure surface keyword dataset, perform data cleaning and risk assessment, calculate and sort the risk level and relevance of the Internet exposure surface information through the local AI recognition model, and generate credible Internet exposure surface information, thereby completing the Internet exposure surface detection.

[0121] For system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0122] Optionally, the embodiment of the present application further provides an electronic device, comprising: a processor, a memory, a computer program stored in the memory and executable in the processor, which realizes each process of the method embodiment and achieves the same technical effects when executed by the processor. To avoid repetition, details are not described here.

[0123] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to realize each process of the method embodiment and achieve the same technical effects. To avoid repetition, details are not described here. The computer readable storage medium includes a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0124] Figure 5 is a block diagram of an electronic device 800 according to an embodiment of the present application. The electronic device 800 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0125] Referring to Figure 5 , the electronic device 800 can include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0126] The processing component 802 usually controls overall operations of the electronic device 800, such as operations associated with displaying, making phone calls, data communications, camera operations and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete all or part of steps of the above method. In addition, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0127] The memory 804 is configured to store various types of data to support the operation of the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or nonvolatile memory, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disc, or optical disc.

[0128] The power supply component 806 supplies power for various components of the electronic device 800. The power supply component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0129] The multimedia component 808 includes a screen providing an output interface between the electronic device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. The front camera and / or the rear camera can receive external multimedia data when the device 800 is in an operation mode, such as a shooting mode or a video mode. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0130] The audio component 810 is configured to output and / or input an audio signal. For example, the audio component 810 includes a microphone (MIC) configured to receive an external audio signal when the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting an audio signal.

[0131] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0132] The sensor component 814 includes one or more sensors for providing status assessments for various aspects of the electronic device 800. For example, the sensor component 814 can detect an open / closed position of the device 800, relative positioning of components, such as a display and a keypad of the electronic device 800, a change in position of the electronic device 800 or a component of the electronic device 800, presence or absence of user contact with the electronic device 800, orientation or acceleration / deceleration / g-force and temperature of the electronic device 800. The sensor component 814 can include an optical sensor for detecting ambient light, a proximity sensor for detecting nearby objects without any physical touch, a CMOS or CCD image sensor for use in imaging applications, or an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor in some embodiments.

[0133] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a cellular network (e.g., 2G, 3G, 4G, or 5G), or a combination thereof. In an example embodiment, the communication component 816 receives broadcast signals or broadcast operation information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 816 can further include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0134] In an example embodiment, the electronic device 800 can be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements for performing the above-described methods.

[0135] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the electronic device 800 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0136] Figure 6is a block diagram of a computer-readable storage medium 1900 shown in the present application. For example, the computer-readable storage medium 1900 can be provided as a server.

[0137] Referring to Figure 6 The computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-mentioned method.

[0138] The computer-readable storage medium 1900 can also include a power supply component 1926 configured to perform power management of the computer-readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer-readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer-readable storage medium 1900 can operate based on an operating system stored in the memory 1932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

[0139] It should be noted that in this paper, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0140] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods described in various embodiments of the present application.

[0141] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and all of them belong to the protection of the present application.

[0142] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the embodiments of the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software mode depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0144] In the embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented by other ways. For example, the device embodiments described above are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0145] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected to achieve the purpose of the embodiment of the present application according to actual needs.

[0146] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0147] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc.

[0148] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An active Internet exposure surface detection method, characterized by, The method comprises: obtaining enterprise local asset management information, network configuration information and security control information, and performing cleaning and sorting operations to generate an internal asset dataset comprising the asset management information and the network configuration information and an unknown exposure surface keyword dataset comprising the security control information; according to the internal asset dataset and the unknown exposure surface keyword dataset, extracting internet exposure surface information by crawling an internet data source through a crawler program; validating the internet exposure surface information, inputting the validated internet exposure surface information into a local AI recognition model, combining the internal asset dataset and the unknown exposure surface keyword dataset to perform data cleaning and risk assessment, calculating and sorting the risk level and relevance of the internet exposure surface information through the local AI recognition model, and generating credible internet exposure surface information.

2. The active Internet exposure surface probing method of claim 1, wherein, The crawler program uses a Selenium tool to render a webpage, and according to the type of the accessed webpage, the crawler program extracts the internet data source through static page element positioning and simulates webpage automatic clicking.

3. The active Internet exposure surface probing method of claim 1, wherein, The crawler program uses a proxy IP pool and a browser header address pool to perform data crawling.

4. The active Internet exposure surface probing method of claim 1, wherein, The extraction of the internet exposure surface information comprises: interfacing with a spatial mapping engine to collect vulnerability data and threat intelligence information of the internet exposure surface.

5. The active Internet exposure surface probing method of claim 1, wherein, The local AI is trained through the internal asset dataset and the unknown exposure surface keyword dataset.

6. An active Internet exposure surface probing method as claimed in claim 5, characterized in that, The local AI recognition model is trained through reinforcement learning and reward modeling, and is optimized through a PPO algorithm.

7. An active Internet exposure surface probing method as claimed in claim 6, characterized in that, The calculation and sorting of the risk level and relevance of the internet exposure surface information through the local AI recognition model comprises: the local AI recognition model assigns weights to and sorts the risk level and relevance of the internet exposure surface information according to the vulnerability data and the threat intelligence information of the internet exposure surface information, and generates a risk report of the internet exposure surface information.

8. An active Internet exposure surface detection system, characterized by, The system comprises: an information acquisition module configured to obtain enterprise local asset management information, network configuration information and security control information, and perform cleaning and sorting operations to generate an internal asset dataset comprising the asset management information and the network configuration information and an unknown exposure surface keyword dataset comprising the security control information; a crawling module configured to extract internet exposure surface information by crawling an internet data source through a crawler program according to the internal asset dataset and the unknown exposure surface keyword dataset; a detection module configured to validate the internet exposure surface information through a network spatial mapping engine, input the validated internet exposure surface information into a local AI recognition model, combine the internal asset dataset and the unknown exposure surface keyword dataset to perform data cleaning and risk assessment, calculate and sort the risk level and relevance of the internet exposure surface information through the local AI recognition model, and generate credible internet exposure surface information.

9. An electronic device, comprising: The system comprises: a processor, a memory, and a computer program stored on the memory and executable on the processor, wherein the computer program is executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1 to 7.

Images