An information collection method, system, device, and computer-readable storage medium
By introducing a large language model into penetration testing to block information gathering tasks, the problems of resource waste and inefficiency are solved, and more accurate information gathering and continuous and effective asset detection are achieved.
Patent Information
- Application Number
- CN202611208990.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-10
- Publication Date
- 2026-11-10
AI Technical Summary
Large language models may lead to resource waste and task execution timeouts during penetration testing information gathering due to inference bias or getting stuck in loops, thus affecting penetration testing efficiency.
A large language model is introduced to handle the blocking of information collection tasks. By obtaining the task context and outputting the blocking response strategy, a reasonable blocking strategy is formulated to handle the task after blocking, avoiding simple truncation or rate limiting.
It improved the accuracy of information gathering, reduced resource waste, ensured the continuous and effective operation of asset exploration during penetration testing, and avoided overall mission failure.
Smart Images

Figure CN123074775A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of cybersecurity technology, and more specifically, to an information collection method, system, device, and computer-readable storage medium. Background Technology
[0002] Penetration testing is a method of assessing the security of computer systems, networks, or web applications by simulating attacker behavior to discover potential security vulnerabilities. Information gathering, as a preliminary stage of penetration testing, directly determines the effectiveness of vulnerability discovery and attack path planning.
[0003] In recent years, Large Language Models (LLMs) have demonstrated powerful capabilities in natural language understanding and intelligent decision-making, providing a new technical approach for the intelligent scheduling of penetration testing information gathering. For example, LLMs can be used as scheduling engines to invoke various security tools to execute information gathering tasks. However, LLMs may repeatedly invoke the same tool due to inference bias or getting stuck in loops, or continuously try invalid paths, leading to wasted system resources, task timeouts, and even triggering alarms from the target security system, thus affecting penetration testing efficiency.
[0004] In conclusion, improving the accuracy of information collection and reducing resource waste are problems that urgently need to be solved by those skilled in the art. Summary of the Invention
[0005] The purpose of this application is to provide an information collection method that can, to some extent, solve the technical problem of how to improve the accuracy of information collection and reduce resource waste. This application also provides an information collection system, an electronic device, and a computer-readable storage medium.
[0006] To achieve the above objectives, firstly, an information collection method is provided, comprising: Obtain a list of candidate assets after crawling the assets of the target object; According to the candidate asset list, perform the corresponding information collection tasks; In response to the blocking that triggers the information gathering task, the context of the information gathering task is input into the large language model; Blocking response strategy for obtaining the information collection task output by the large language model; The information collection task is processed according to the blocking response strategy to obtain the information collection results of the target object.
[0007] Preferably, obtaining the candidate asset list after crawling the target object includes: Obtain the identifier of the target object; Based on the identifier of the target object, a full asset crawl is performed using various search syntaxes to obtain the original asset list; Based on the ICP filing information, the original asset list is filtered and expanded to obtain an initial asset list; Based on IP attribution information, the initial asset list is filtered to obtain a filtered asset list; Based on the sub-objects associated with the target object, the filtered asset list is expanded and deduplicated to obtain a candidate asset list; The various search syntaxes include domain name search, certificate search, title search, body search, icon search, and C-segment search.
[0008] Preferably, the step of performing the corresponding information collection task according to the candidate asset list includes: The candidate asset list is divided into blocks to obtain block asset data; Input the segmented asset data into the large language model; The value level is obtained by the large language model after performing semantic analysis, contextual reasoning, and cross-association on the segmented asset data. The asset priority of the candidate asset list is determined based on the value level. According to the asset priority, perform the corresponding information collection task on the candidate asset list; The semantic analysis is used to infer the business meaning of the asset identifier by combining the industry context of the target object; the contextual reasoning is used to determine the value level by comprehensively considering the multi-dimensional signals of the asset, including HTTP response features, page title, response header information and JS reference information; the cross-association is used to adjust the value level of the asset according to the association between assets.
[0009] Preferably, the information collection task includes an active information collection task, and the step of executing the corresponding information collection task according to the candidate asset list includes: For the assets to be processed in the candidate asset list, they are processed according to a preset expansion method to obtain asset baselines and vulnerability clues; By using a browser proxy, we perform site database analysis, request crawling, and front-end framework extraction on the assets to be processed to obtain browser information. In response to obtaining the login credentials for the asset to be processed, authentication information is extracted based on the login credentials; The asset baseline is subjected to fingerprint recognition, WAF detection, and authentication catalog scanning to obtain asset detection results; Perform static auditing on unprocessed JavaScript files in the assets to be processed to obtain static audit information; Based on the asset baseline, the vulnerability clues, the browser information, the authentication information, the asset detection results, and the static audit information, the proactive information collection results for generating the candidate asset list are generated. The preset extension methods include subdomain enumeration, liveness detection, directory brute-force, JavaScript analysis, port scanning, unauthorized access detection, and attack surface analysis.
[0010] Preferably, the step of performing site database analysis on the assets to be processed through a browser proxy includes: Launch a browser instance via a browser proxy and load the Uniform Resource Locator (URL) of the target page in the asset to be processed; Starting from the entry Uniform Resource Locator (URL) of the target page, traverse the links of the target page to construct a page hierarchy graph of the site corresponding to the target page; Analyze the Uniform Resource Locator (URL) path patterns, navigation structures, and interactive elements of each page in the page hierarchy diagram to identify the functional modules of the site; Identify the forms in each page of the page hierarchy diagram, analyze the forms, and obtain form information, which includes submission address, field name, field type, and validation rules; Based on the form information, the page type is labeled for each page in the page hierarchy diagram to obtain the page type result; Track the data flow between pages in the page hierarchy diagram and establish the calling relationship between the functional modules; Based on the page hierarchy diagram, the functional modules, and the calling relationships, a five-dimensional site structure profile of the asset to be processed is constructed.
[0011] Preferably, the information collection task includes a passive information collection task, and the step of executing the corresponding information collection task according to the candidate asset list includes: For the pending assets in the candidate asset list, obtain the traffic file of the pending assets; The traffic files are subjected to asset inventory, value classification, JS analysis, API inference, port scanning, vulnerability detection, authentication testing and summarization to obtain the passive information collection results of the candidate asset list.
[0012] Preferably, the information collection task is processed according to the blocking response strategy, including: If the blocking response strategy indicates that passage is allowed, then the information collection task continues. If the blocking response strategy is characterized as a downgrade, then the scanning intensity or range of the information collection task is reduced and the task continues to be executed. If the blocking response strategy is skipped, the current step of the information collection task is skipped and the subsequent steps are executed. If the blocking response strategy indicates redirection, then the execution will continue after switching to an alternative path for the information gathering task; If the blocking response strategy is terminated, the information collection task ends.
[0013] Secondly, an information collection system is provided, including: The asset crawling module is used to obtain a list of candidate assets after crawling the target object. The information collection module is used to perform corresponding information collection tasks according to the candidate asset list; The model input module is used to input the context of the information gathering task into the large language model in response to the blocking that triggers the information gathering task. The strategy acquisition module is used to acquire the blocking response strategy of the information collection task output by the large language model; The blocking processing module is used to process the information collection task according to the blocking response strategy in order to obtain the information collection result of the target object.
[0014] Thirdly, an electronic device is provided, comprising: Memory, used to store computer programs; A processor for executing the computer program to implement any of the information collection methods described above.
[0015] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements any of the information collection methods described above.
[0016] This application provides an information collection method that involves obtaining a candidate asset list after crawling assets of a target object; executing corresponding information collection tasks according to the candidate asset list; inputting the context of the information collection task into a large language model in response to the blocking of the information collection task; obtaining the blocking response strategy of the information collection task output by the large language model; and processing the information collection task according to the blocking response strategy to obtain the information collection results of the target object. In this application, a large language model is introduced into the blocking processing stage of the information collection task. The reasoning ability of the large language model is used to perform blocking response processing on the blocked information collection task, realizing intelligent closed-loop blocking processing of triggering, reasoning, and decision-making. Compared with simple hard truncation or rate limiting of the information collection task, the reasoning ability of the large language model can be used to formulate a more reasonable blocking response strategy, which can more accurately process the information collection task after blocking, avoiding premature or slow termination of information collection, improving the accuracy of information collection and reducing resource waste. Based on this, when conducting asset penetration testing on a target object, if information collection for a specific asset triggers a blockage, the process doesn't simply involve disconnection or waiting. Instead, it utilizes LLM (Limited Least Metrics) to infer the current asset's processing status. This allows for precise differentiation between deep enumeration of a particular asset and invalid collisions, ensuring continuous and effective detection of various assets throughout the entire penetration cycle and preventing the overall penetration task from failing due to a single point of failure. The information collection system, electronic device, and computer-readable storage medium provided in this application also solve the corresponding technical problems. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0018] Figure 1 A flowchart illustrating an information collection method provided in this application embodiment; Figure 2 A flowchart for information collection based on asset filtering and augmentation; Figure 3 A flowchart for prioritizing information collection; Figure 4 A flowchart for proactive information collection; Figure 5 This is a schematic diagram of the structure of an information collection system provided in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of the electronic device according to an embodiment of this application. Detailed Implementation
[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] Penetration testing is a method of assessing the security of computer systems, networks, or web applications by simulating attacker behavior to discover potential security vulnerabilities. Information gathering, as a preliminary stage of penetration testing, directly determines the effectiveness of vulnerability discovery and attack path planning.
[0021] In recent years, large language models have demonstrated powerful capabilities in natural language understanding and intelligent decision-making, providing a new technical approach for the intelligent scheduling of penetration testing information gathering. For example, large language models can be used as scheduling engines to invoke various security tools to execute information gathering tasks. However, large language models may repeatedly invoke the same tool due to inference bias or getting stuck in loops, or continuously try invalid paths, leading to wasted system resources, task timeouts, and even triggering alarms from the target security system, thus affecting penetration testing efficiency. The information gathering scheme provided in this application can improve the accuracy of information gathering and reduce resource waste.
[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating an information collection method provided in an embodiment of this application.
[0023] An information collection method provided in this application embodiment may include the following steps: Step S101: Obtain the candidate asset list after crawling the target object.
[0024] In practical applications, during penetration testing of a target object, a candidate asset list can be obtained first after asset crawling of the target object. This list can then be used to conduct penetration testing on the target object later. The target object can be a user-specified target for penetration testing, such as a company, brand, product, or domain name. Asset crawling refers to the process of automatically collecting network asset information associated with the target object from the internet. This asset information can include domain names, IP addresses, open ports, and web services.
[0025] Step S102: Perform the corresponding information collection task according to the candidate asset list.
[0026] In practical applications, after obtaining the candidate asset list, it is necessary to collect information on the assets in the candidate asset list. That is, according to the candidate asset list, the corresponding information collection task needs to be executed to obtain the corresponding asset information. The information collection task refers to the process of in-depth exploration and analysis of the assets in the candidate asset list.
[0027] Step S103: In response to the blocking of the information gathering task, the context of the information gathering task is input into the large language model.
[0028] Step S104: Obtain the blocking response strategy for the information collection task output by the large language model.
[0029] In practical applications, to avoid continuous information collection from a specific asset, it is necessary to set up corresponding blocking mechanisms for information collection tasks to protect the information collection process. These blocking mechanisms can utilize tool call counts, stage execution rounds, and global execution rounds. For example, during the execution of an information collection task, the number of calls to the information collection tool and / or the number of execution rounds in the information collection stage can be counted. When the number of calls or the number of execution rounds exceed a preset threshold, blocking is triggered. Taking tool call counts as an example, during the execution of an information collection task, the number of calls to the information collection tool within a single stage can be detected. When the number of calls to the same information collection tool within a single stage exceeds a first preset threshold, blocking is triggered. The first preset threshold can be dynamically configured based on the tool type and target scale. For example, for a subdomain enumeration tool... For tools, a higher threshold can be set, such as 20 times; for port scanning tools, a lower threshold can be set, such as 5 times. Taking the number of execution rounds in a stage as an example, the number of execution rounds in each stage can be collected during the execution of each information collection task. When the number of execution rounds in a certain stage exceeds the second preset threshold, a blocking is triggered. The number of execution rounds in a stage can be the number of times different parameters or different tool combinations are repeatedly tried for information collection in the same stage. The second preset threshold can be set to 3 rounds or 5 rounds, etc. Taking the number of global execution rounds as an example, the number of decision loops in the entire information collection task can be collected as the number of global execution rounds. When the number of global rounds reaches the third preset threshold, a blocking is triggered. The third preset threshold can be 3 rounds or 5 rounds, etc.
[0030] When any of the above count values exceeds the corresponding preset threshold, a blocking mechanism is triggered, pausing the currently executing information collection task. The context of the current information collection task is then input into the large language model. The type of large language model can be flexibly selected as needed. The context of the current information collection task includes, but is not limited to: the sequence of executed tool calls and the input parameters and output results of each call, the execution status of the current stage, the completed information collection results, error logs and exception information, the processing progress of the current asset, and the scale information of the target asset. Afterwards, the large language model infers based on the input task context to determine whether the reason for the blocking is a reasonable repeated call (e.g., due to a large amount of information to be collected, requiring multiple calls to the same tool to complete the enumeration), or an invalid miscall or infinite loop (e.g., the large language model repeatedly tries the same path, each attempt fails, or the output is zero). Based on the inference result, the optimal strategy is selected from five blocking response strategies. The first strategy is to allow, i.e., determining that the current call scenario is reasonable and allowing continued execution; the second strategy is to downgrade, i.e., determining that the current tool call is necessary but too intense, reducing the scanning intensity or range before continuing. For example, the dictionary used for directory brute-force attacks can be changed from a large dictionary to a smaller dictionary, or the scope of port scanning can be reduced from all ports to common ports. The third strategy is skipping, which means skipping the current step if the current path is deemed low-yield or has no more information to discover, and continuing with the subsequent stages using existing data. The fourth strategy is redirection, which means switching to a more efficient alternative path if the current information gathering task is deemed inefficient, such as switching from active scanning to passive analysis, or switching from the current asset to other high-value assets. The fifth strategy is termination, which means forcibly terminating information gathering if continuing the current information gathering task is deemed meaningless or too risky. Correspondingly, it is necessary to obtain the blocking response strategy for information gathering tasks output by the large language model to process the information gathering tasks accordingly.
[0031] Step S105: Process the information collection task according to the blocking response strategy to obtain the information collection results of the target object.
[0032] In practical applications, once the blocking response strategy is obtained, information gathering tasks can be processed according to this strategy to obtain the information gathering results of the target object. In this way, the target object can be processed accordingly, such as evaluating the target object or conducting penetration testing.
[0033] In practical applications, the type of blocking response strategy can be flexibly set as needed. For example, during the information collection task processing according to the blocking response strategy, if the blocking response strategy indicates permission, the information collection task continues to be executed, that is, the information collection tool continues to be called to collect information. After the information collection is completed, the information collection result of the target object is obtained. If the blocking response strategy indicates degrading, the scanning intensity or range of the information collection task is reduced and then execution continues. For example, the scanning concurrency of the current tool is reduced, the scanning dictionary is switched from fine mode to coarse mode, or the scanning timeout is shortened. After the information collection is completed, the information collection result of the target object is obtained. If the blocking response strategy indicates skipping, the current step of the information collection task is skipped and the subsequent steps are executed. After the information collection is completed, the information collection result of the target object is obtained. If the blocking response strategy indicates redirection, the process switches to an alternative path for the information collection task and continues execution. After the information collection is completed, the information collection result of the target object is obtained. If the blocking response strategy indicates termination, the information collection task is ended, and the collected information is used as the information collection result of the target object. In this way, these five strategies cover all stages of the blocking scenario. The degradation strategy can reduce the risk of being blocked by the target without completely abandoning the information collection task. The skip strategy avoids ineffective waiting on low-yield paths. The termination strategy ensures that the collected information data can be saved under any abnormal circumstances, avoiding the complete loss of the information collection results of the target object.
[0034] This application provides an information collection method that involves obtaining a candidate asset list after crawling assets of a target object; executing corresponding information collection tasks according to the candidate asset list; inputting the context of the information collection task into a large language model in response to the blocking of the information collection task; obtaining the blocking response strategy of the information collection task output by the large language model; and processing the information collection task according to the blocking response strategy to obtain the information collection results of the target object. In this application, a large language model is introduced into the blocking processing stage of the information collection task. The reasoning ability of the large language model is used to perform blocking response processing on the blocked information collection task, realizing intelligent closed-loop blocking processing of triggering, reasoning, and decision-making. Compared with simple hard truncation or rate limiting of the information collection task, the reasoning ability of the large language model can be used to formulate a more reasonable blocking response strategy, which can more accurately process the information collection task after blocking, avoiding premature or slow termination of information collection, improving the accuracy of information collection and reducing resource waste. Based on this, when conducting asset penetration on a target object, if the collection of information on a certain asset of the target object triggers a blockage, it is not simply a matter of disconnecting or waiting. Instead, it uses LLM to infer the current processing status of the asset, which can accurately distinguish between deep enumeration of a certain asset and invalid collisions. This ensures that the detection of various assets continues to operate effectively throughout the entire penetration cycle, avoiding the failure of the overall penetration task due to a single point of failure.
[0035] Building upon the above embodiments, crawling as many target assets as possible will undoubtedly provide more options for information gathering, thereby providing more information for subsequent penetration testing and ensuring the accuracy of the penetration test. To achieve this goal, please refer to... Figure 2 The information collection method provided in this application embodiment may include the following steps: Step S201: Obtain the identifier of the target object.
[0036] In practical applications, relevant information can be collected based on the identifier of the target object. This identifier can be a descriptive information entered by the user to uniquely identify the target object, such as the full name of the target company, brand name, product name, or domain name. Furthermore, to improve the accuracy of subsequent information searches, the user-input identifier can be standardized, for example, by removing extra spaces, standardizing capitalization, and supplementing with the complete company name, thus obtaining the target object's identifier.
[0037] Step S202: Based on the identifier of the target object, crawl the entire asset list using various search syntaxes to obtain the original asset list. The various search syntaxes include domain search, certificate search, title search, body search, icon search, and C-segment search.
[0038] In practical applications, a full asset crawl can be performed using a variety of search syntax combinations through the API (Application Programming Interface) of a spatial search engine. A spatial search engine is a search engine capable of automatically querying information about devices, websites, and services on the internet via an API interface. Examples of spatial search engines include FOFA, Shodan, and Censys. Various search syntaxes include domain name search, certificate search, title search, body search, icon search, and C-segment search. Additional features include Amass / Subfield aggregation of CT logs and passive data sources such as DNS to enumerate subdomains, httpx batch active site information detection, and Masscan scanning to achieve C-segment association.
[0039] In practical applications, asset crawling using domain name search involves using the target object's identifier as a keyword to search for domains containing that name or its spelling variations. When using certificate search, one can search for related domains and subdomains through the organization name field in the Transport Layer Security (TLS) certificate. This is because when an organization applies for TLS certificates for multiple subdomains or different domains, the organization name field usually contains the same organization information, thus allowing for reverse indexing to other domains of the same organization. Finally, asset crawling using title search involves matching the content of the title tags in the HTTP response. Since website titles often contain brand names or product names, title search can discover sites whose domains do not directly contain the target object's identifier but whose page titles are related to the target. In asset crawling via Body Search, matching searches can be performed using the HTTP response body content. This is because the body content of a webpage may contain information such as the target object's identifier, even if the site's domain and title do not directly reflect the target object's information. Body Search can still discover such related assets. In asset crawling via Icon Search, matching searches can be performed using the hash value of the website icon. This is because different sites under the same organization often use the same or similar website icons, and matching icon hash values can discover other assets under the same entity. In asset crawling via C-segment search, C-segment association searches can be performed on discovered IP addresses. The C-segment refers to the first 24 bits of an IP address; IP addresses within the same C-segment usually belong to the same network infrastructure or the same organization. These six search syntaxes can be executed independently; simply taking the union of the search results generates the original asset list. In this way, through this multi-syntax fusion crawling strategy, asset crawling can be performed from multiple perspectives, ensuring comprehensive asset discovery.
[0040] It should be noted that for mobile applications such as mini-programs of the target object, the backend API interface address of the mini-program can be extracted by decompiling or packet capture and added to the original asset list of the target object.
[0041] Step S203: Based on the ICP filing information, filter and expand the original asset list to obtain an initial asset list.
[0042] In practical applications, the original asset list may contain assets that do not belong to the target, or may be missing some assets of the target. In order to avoid assets that do not belong to the target affecting the penetration test and to collect as much information as possible for the penetration test, the original asset list can be filtered and expanded.
[0043] In this process, ICP filing is a mandatory filing procedure for websites. Each filed domain name corresponds to a filing entity, and the filing information includes the entity name. Therefore, the ICP filing entity information corresponding to each domain name in the original asset list can be queried through the ICP filing query interface. For domain names with found filing information, the filing entity name is matched with the target object's identifier. If the filing entity matches the target object's identifier, the domain name is retained; if the filing entity does not match the target object's identifier, the domain name is marked as a non-target asset and removed from the original asset list. At the same time, for domain names whose filing entity matches the target object's identifier, all other filed domain names under that filing entity are further queried and included in the original asset list to obtain the initial asset list, thereby achieving asset expansion.
[0044] Step S204: Based on IP attribution information, filter the initial asset list to obtain a filtered asset list.
[0045] In practical applications, after ICP filing filtering and expansion, the initial asset list may still contain some assets for which filing information cannot be found or is incomplete, such as servers deployed overseas or unfiled test environments. Based on this, the IP address of each asset in the initial asset list can be queried through the IP attribution query interface to identify the organization name to which the IP belongs. In other words, the organization to which the IP belongs is matched with the identifier of the target object. If they match, the asset is retained; if they do not match, the asset is removed from the initial asset list, resulting in a filtered asset list.
[0046] Step S205: Based on the sub-objects associated with the target object, expand and deduplicate the filtered asset list to obtain a candidate asset list.
[0047] In practical applications, a target object may have multiple associated sub-objects. These sub-objects can also assist in penetration testing of the target object. Taking a company as an example, companies often control multiple subsidiaries through holding or participating in shares. The assets of these subsidiaries can also become the attack surface for penetration testing. Based on this, the filtered asset list can be expanded and deduplicated according to the sub-objects associated with the target object to obtain a candidate asset list, thereby achieving further screening of the target object's assets. Still using a company as an example, the equity structure of the target company can be queried through the company's business registration information query interface to find the subsidiaries that the target company holds or participates in. Then, for each subsidiary found, the name of the subsidiary is used as a new identifier to crawl the corresponding assets. The assets of the subsidiaries are then merged into the filtered asset list and deduplicated to obtain the candidate asset list.
[0048] Step S206: Perform the corresponding information collection task according to the candidate asset list.
[0049] Step S207: In response to the blocking of the information gathering task, the context of the information gathering task is input into the large language model.
[0050] Step S208: Obtain the blocking response strategy for the information collection task output by the large language model.
[0051] Step S209: Process the information collection task according to the blocking response strategy to obtain the information collection results of the target object.
[0052] As can be seen from the implementation process, this application obtains the target object's identifier during the process of acquiring the candidate asset list after crawling the target object's assets; based on the target object's identifier, it performs full asset crawling using various search syntaxes to obtain the original asset list; based on ICP filing information, it filters and expands the initial asset list to obtain the initial asset list; based on IP attribution information, it filters the initial asset list to obtain the filtered asset list; based on the sub-objects associated with the target object, it expands and deduplicates the filtered asset list to obtain the candidate asset list; among these, the various search syntaxes include domain name search, certificate search, title search, body search, icon search, and C-segment search. It can achieve full coverage of assets through six different search syntaxes; automatically remove domains unrelated to the target object by matching ICP filing entities, and expand other filed domains under the same entity in reverse; automatically exclude IP assets other than the target object by IP attribution query; discover easily overlooked related assets such as sub-objects through association; realize end-to-end automated conversion from fuzzy target object identification to accurate asset list, and achieve automated removal of noisy assets while expanding assets, thus expanding the attack surface coverage and improving the accuracy of subsequent penetration testing.
[0053] Building upon the above embodiments, considering the large number of assets, information needs to be collected for each asset. However, the impact of information on each asset differs for penetration testing. Therefore, reasonable information collection will undoubtedly improve the validity of subsequent penetration testing. To achieve this goal, please refer to [link to relevant documentation]. Figure 3 The information collection method provided in this application embodiment may include the following steps: Step S301: Obtain the candidate asset list after crawling the target object.
[0054] Step S302: Divide the candidate asset list into blocks to obtain block asset data.
[0055] In practical applications, since the candidate asset list may contain hundreds to thousands of asset records, and the single context window capacity of the large language model is limited, it is impossible to process all asset data at once. Therefore, a block processing strategy can be adopted to group the asset data in the candidate asset list by domain name or IP segment to obtain block asset data, each block containing a preset number of asset records.
[0056] Step S303: Input the segmented asset data into the large language model.
[0057] Step S304: Obtain the value level obtained by the large language model after performing semantic analysis, contextual reasoning and cross-correlation processing on the segmented asset data.
[0058] In practical applications, segmented asset data can be input into a large language model to obtain the value level of the asset data. When submitting each segmented asset data to the large language model, industry information and business type information of the target object can be attached as contextual background for hierarchical reasoning of the large language model.
[0059] In practical applications, the Big Language Model (MLM) can classify assets into value categories through a three-layer mechanism. The first layer is semantic analysis, where the MLM performs semantic analysis on each asset's identifier to understand its meaning within the target business context. Unlike fixed keyword matching, semantic analysis can infer based on the semantics and context of words. For example, "crm" might not be included as a high-value keyword in a general keyword list, but the MLM can infer from its semantic knowledge that "crm" is an abbreviation for Customer Relationship Management System. Combined with the industry context of the target enterprise, it can infer that the subdomain may carry customer data management functions. Similarly, "portal" might refer to an online banking portal in the financial industry context, but only an information portal in the manufacturing industry context, thus belonging to a high-value target. The second layer is contextual reasoning, where the MLM comprehensively considers multi-dimensional probe signals of the asset for inference and judgment. These multi-dimensional probe signals include HTTP response characteristics, page titles, response header information, and JavaScript reference information. For example, if the literal name of a subdomain does not contain... The first layer involves identifying assets with management keywords, but if HTTP probing reveals response headers containing management backend framework characteristics such as "X-Admin-Session," or if the page title contains non-English management keywords like "console" or "backend," the big language model classifies the asset as a high-value target. For example, if a subdomain is a static resource site (static.example.com), but JS analysis reveals it references the complete address of an internal API, the big language model elevates it from a general target to a high-value target. The third layer involves cross-associations, where the big language model discovers relationships between assets and adjusts the value level of individual assets based on these relationships. For instance, if multiple subdomains point to the same IP range, the big language model infers that this IP range might be the target's internal infrastructure and prioritizes all assets under that IP range. Similarly, if cdn.example.com is a general target, but the big language model finds that its CDN origin address points to the API gateway of the internal management system, it classifies it as a high-value target. Finally, after summarizing the processing results of each asset, the big language model can also classify cross-block conflicts, that is, perform secondary reasoning and adjudication on the different levels assigned to the same asset in different blocks, and output the final deduplicated classification list. The classification list output by the big language model can be a structured list, including asset identifier, value level (high value / general), classification basis, confidence level, and suggested follow-up collection actions.
[0060] Step S305: Determine the asset priority of the candidate asset list based on the value level.
[0061] Step S306: Execute the corresponding information collection task for the candidate asset list according to asset priority.
[0062] In practical applications, assets can be sorted according to their value level from high to low, forming a priority queue for information collection. For example, high-value assets are placed at the front of the queue to receive information collection resources first, while general assets are placed at the back of the queue and processed after the high-value assets are completed. This ensures that the attack surface information of the most critical assets is obtained first within a limited time window. Then, information collection tasks are executed on the candidate asset list according to priority. For example, the main intelligent agent executes information collection tasks through information collection tools using a blocking call method. This blocking call ensures strict sequential execution between stages and correct maintenance of data dependencies. Furthermore, during the execution of information collection tasks, the number of calls to each information collection tool and the number of execution rounds for each information collection stage are counted in real time for subsequent blocking trigger judgments.
[0063] Step S307: In response to the blocking of the information gathering task, the context of the information gathering task is input into the large language model.
[0064] Step S308: Obtain the blocking response strategy for the information collection task output by the large language model.
[0065] Step S309: Process the information collection task according to the blocking response strategy to obtain the information collection results of the target object.
[0066] As can be seen from the implementation process, this application, in the process of performing corresponding information collection tasks according to the candidate asset list, divides the candidate asset list into blocks to obtain block asset data; inputs the block asset data into a large language model; obtains the value level obtained by the large language model after semantic analysis, contextual reasoning, and cross-association processing of the block asset data; determines the asset priority of the candidate asset list according to the value level; and performs corresponding information collection tasks on the candidate asset list according to the asset priority. Semantic analysis is used to infer the business meaning of asset identifiers in conjunction with the industry context of the target object; contextual reasoning is used to comprehensively infer and determine the value level of assets based on multi-dimensional signals, including HTTP response features, page titles, response header information, and JS reference information; cross-association is used to adjust the asset value level based on the relationships between assets. In this way, the semantic understanding capability of the large language model can overcome the limitations of fixed keywords and perform business meaning reasoning in conjunction with industry context; cross-validation is performed by the contextual reasoning layer to comprehensively analyze multi-dimensional signals such as HTTP response headers, page titles, and JS references to avoid misjudgment based on a single dimension; and hidden relationships such as CDN origin pulls and same IP segments are discovered through the cross-association layer, and the grading is dynamically adjusted accordingly. This enables accurate asset classification, facilitating subsequent value-based prioritization and ensuring that limited information gathering resources are allocated to high-value targets, maximizing the effectiveness of penetration testing within a limited window of opportunity.
[0067] Based on the above embodiments, in order to collect as much information as possible and as accurately as possible, please refer to... Figure 4 The information collection method provided in this application embodiment may include the following steps: Step S401: Obtain the candidate asset list after crawling the target object.
[0068] Step S402: For the assets to be processed in the candidate asset list, process them according to the preset extension methods to obtain the asset baseline and vulnerability clues. The preset extension methods include subdomain enumeration, liveness detection, directory brute-force, JavaScript analysis, port scanning, unauthorized access detection and attack surface analysis.
[0069] In practical applications, information collection tasks can include proactive information collection tasks, which means collecting information by actively sending probe requests to the target object. Based on this, in the process of executing the corresponding information collection tasks according to the candidate asset list, the assets to be processed in the candidate asset list can be processed first according to the preset expansion method to obtain the asset baseline and vulnerability clues.
[0070] In practical applications, preset extension methods can include sub-tasks such as subdomain enumeration, liveness detection, directory brute-force, JavaScript analysis, port scanning, unauthorized access detection, and attack surface analysis. In subdomain enumeration, subdomains can be incrementally discovered through dictionary enumeration, DNS zone transfer detection, certificate transparency log queries, and search engine crawling. In liveness detection, live targets can be filtered through HTTP / HTTPS connectivity probes, and the availability of discovered subdomains and IPs can be verified to exclude offline or unreachable assets. In directory brute-force, the URL paths of live targets can be brute-forced using dictionaries to discover hidden directories and files. In JavaScript analysis, JavaScript files referenced by the target site can be downloaded and analyzed to extract API endpoints and sensitive information. In port scanning, common ports of the target IP can be scanned to discover open services. In unauthorized access detection, unauthorized access can be detected on common service ports to identify potential risks of sensitive data exposure. In attack surface analysis, the outputs of the above sub-tasks can be combined to analyze and label potential attack surfaces. Furthermore, each subtask can be executed in parallel pipeline mode, and subtasks can be started in parallel. The intermediate output of a subtask can be passed to subsequent subtasks that depend on that output in real time. In addition, the scanning order can follow the asset value classification results, with high-value targets being scanned first. After all subtasks are completed, a complete asset baseline and vulnerability clues are generated. The asset baseline can include domain name, IP, port, service, URL, etc., and the vulnerability clues can include unauthorized access, sensitive paths, etc.
[0071] Step S403: Through the browser proxy, perform site database analysis, request crawling, and front-end framework extraction on the assets to be processed to obtain browser information.
[0072] In practical applications, browser proxies, such as Browser Agent, can be used to perform site database analysis, request crawling, and front-end framework extraction on the assets to be processed, thereby obtaining browser information. Site database analysis refers to establishing a complete profile of the site's page hierarchy, functional modules, and data flow by systematically crawling and analyzing the website's pages.
[0073] In an exemplary embodiment, during the site database analysis of the assets to be processed via a browser proxy, a browser instance can be launched through the browser proxy to load the Uniform Resource Locator (URL) of the target page in the assets to be processed; starting from the entry URL of the target page, the links of the target page are traversed to construct a page hierarchy graph of the site corresponding to the target page; the URL path patterns, navigation structures, and interactive elements of each page in the page hierarchy graph are analyzed to identify the functional modules of the site, which may include user management, data query, file upload, etc.; forms in each page of the page hierarchy graph are identified and processed. The process involves analyzing form information, including submission address, field names, field types, and validation rules. Based on this information, page types are labeled in the page hierarchy diagram, resulting in page type results, such as login page, list page, detail page, edit page, and API documentation. Data flow between pages in the page hierarchy diagram is traced, including data flow via links, form submissions, and AJAX requests, establishing call relationships between functional modules. Based on the page hierarchy diagram, functional modules, and call relationships, a five-dimensional site structure profile of the asset to be processed is constructed. This five-dimensional profile systematically describes the site structure from five dimensions, achieving a leap from URL discovery to site understanding. It provides business-semantic understanding for subsequent Planner attack strategy development, uses form information to provide precise input point location and field validation rules for SQL injection and XSS testing, and uses data flow to provide call chain guidance for exploiting privilege escalation and business logic vulnerabilities, facilitating accurate and efficient subsequent penetration testing.
[0074] During the process of crawling assets through a browser proxy, the network interception function of browser automation tools can be used to capture all XHR (XML HTTP Request) and fetch requests issued during the execution of browser pages, including asynchronously loaded API interfaces (XHR interception); record the URL, request method, request header, request body, and response content of each XHR request, and extract the parameter structure and data format of dynamic interfaces; for list-type pages, automatically trigger interactive operations such as pagination and load more, and capture the interface and parameter patterns of pagination requests; monitor WebSocket connections and record URLs and communication data; for SPA applications, trigger front-end route switching to discover virtual pages based on front-end routes and corresponding asynchronous data requests; and use all the data obtained above as the crawling results.
[0075] During the process of extracting the front-end framework of the assets to be processed through a browser proxy, it can identify the front-end frameworks and versions used by the page, such as React, Vue, and Angular; detect state management tools such as Redux, Vuex, and Pinia; extract sensitive data such as user information and permission configuration contained in the global state; extract front-end configuration information, including API base paths and function switches, from global objects such as window.__CONFIG__ and process.env; and use all the data obtained above as the front-end framework extraction result.
[0076] Step S404: In response to obtaining the login credentials for the asset to be processed, the authentication information is extracted based on the login credentials.
[0077] In practical applications, when the task configuration for information collection provides login credentials for the target site, these credentials can be a username and password, etc., allowing authentication information to be extracted. Specifically, a browser instance in BrowserAgent can be used to simulate a login operation, filling in login credentials and submitting a login form. After successful login, cookies and tokens from the browser, including Session Tokens, JWT Tokens, CSRF Tokens, etc., can be extracted. XHR requests during the login process can be intercepted, capturing the request format and token generation logic of the authentication interface. The extracted authentication information can then be stored in a shared state for use in subsequent stages.
[0078] Step S405: Perform fingerprint recognition, WAF detection, and authentication catalog scanning on the asset baseline to obtain the asset detection results.
[0079] In practical applications, during the fingerprinting of asset baselines, web fingerprinting can be performed on live assets. By analyzing specific patterns in HTTP response headers and bodies, cookie names, and files in specific paths, the target's CMS, framework, web server, programming language, and version information can be detected. During WAF (Web Application Firewall) detection, it's possible to detect whether a target site has deployed a WAF, identify the WAF type and rule characteristics, and provide a basis for subsequent attack strategy formulation. During authentication directory scanning, after successfully obtaining authentication information, directory brute-force tools such as ffuf can be used to scan the authenticated directory with cookies or tokens to discover hidden paths and functions requiring login. Furthermore, fingerprinting, WAF detection, and authentication directory scanning operations in this step can only be performed once and cannot be retried. This ensures that repeated requests during the information collection phase do not trigger the target's security protection mechanisms. If any operation fails, the blocking mechanism can be delegated to a larger model to determine whether downgrade processing is necessary.
[0080] Step S406: Perform static auditing on the unprocessed JavaScript files in the assets to be processed to obtain static audit information.
[0081] In practical applications, deep static auditing can be performed on uncovered JavaScript files in the assets to be processed. For example, based on new URLs and resource references discovered by the browser proxy, the missing JavaScript files can be downloaded; hard-coded API endpoint paths in JavaScript code can be extracted through regular expression matching and abstract syntax tree analysis; sensitive credentials such as leaked API keys, secret keys, and access tokens can be detected in JavaScript code; internal infrastructure information such as internal service domains and test environment domains can be discovered from JavaScript code; and all registered route paths and corresponding component information can be extracted from the front-end framework's routing configuration files. This expands the attack surface.
[0082] Step S407: Based on asset baselines, vulnerability clues, browser information, authentication information, asset detection results, and static audit information, generate a candidate asset list as a result of proactive information collection.
[0083] Step S408: In response to the blocking of the information gathering task, the context of the information gathering task is input into the large language model.
[0084] Step S409: Obtain the blocking response strategy for the information collection task output by the large language model.
[0085] Step S410: Process the information collection task according to the blocking response strategy to obtain the information collection results of the target object.
[0086] As can be seen from the implementation process, this application executes subdomain enumeration, liveness detection, directory brute-force, JS analysis, port scanning, unauthorized access detection, and attack surface analysis in a parallel pipeline manner during the proactive information collection process. Subtasks can be started in parallel and intermediate outputs can be transmitted in real time, improving execution efficiency. In the browser proxy site database analysis stage, JavaScript rendering is executed through a real browser, solving the problem of traditional scanners ignoring applications. The authentication information extraction stage and authentication directory scanning stage realize the post-authentication deep collection capabilities that existing tools do not have, significantly expanding the attack surface coverage and improving the effectiveness of subsequent penetration testing.
[0087] Understandably, in addition to active information gathering, passive information gathering can also be performed. Passive information gathering is suitable for scenarios where existing HTTP traffic files exist, such as pcap files captured by packet sniffing tools or HTTP session files exported by proxy tools. Passive mode starts from existing HTTP traffic files without actively sending any probe requests to the target, offering a high degree of stealth. In other words, information gathering tasks include passive information gathering tasks. During the execution of corresponding information gathering tasks according to the candidate asset list, for assets to be processed in the candidate asset list, the traffic files of the assets to be processed are obtained; the traffic files are then used for asset inventory, value classification, JS analysis, API inference, port scanning, vulnerability detection, authentication testing, and summarization to obtain the passive information gathering results for the candidate asset list.
[0088] Specifically, during asset inventory, HTTP traffic files can be parsed to extract target domains from the HTTP request Host header, paths and query parameters from the request URL, source and destination IPs from IP layer information, and domains from certificates in TLS handshake information to build an asset list. In value grading, similar to value grading in proactive mode, the large language model categorizes targets into high-value and general targets based on extracted domain naming patterns, request characteristics, and business context in the traffic, prioritizing analysis of high-value target traffic in subsequent stages. During fingerprinting, features such as HTTP response headers, cookies, and page content can be analyzed to identify the technology stack and middleware version information used by the target. During JS analysis, the content of JavaScript files can be extracted from the traffic for analysis. This includes API endpoints, sensitive information, and internal domains. During API inference, the functionality and parameter structure of API interfaces can be inferred from the URL path, parameters, and response format of HTTP requests. During port scanning, ports associated with target IPs can be identified from traffic, supplementing the port profile with known service information. During vulnerability detection, HTTP requests and responses can be analyzed to identify potential security vulnerabilities, such as error message leaks, sensitive path exposure, and insecure HTTP methods. During authentication testing, authentication-related requests, such as login and token refresh, can be extracted from traffic to analyze the implementation and potential weaknesses of authentication mechanisms. During aggregation, the outputs from all stages can be merged and compiled into a standardized report, labeled with target value levels, and written to structured storage. The passive mode also has a built-in blocking mechanism, limiting the number of execution rounds and tool calls for each analysis stage, and using a large language model to determine the blocking response strategy when blocking is triggered.
[0089] In an exemplary embodiment, after obtaining the information collection results of the target object, the collected information can be compiled into a standardized report. The report may include all discovered assets and their value levels, fingerprint information of each asset, discovered open ports and services, API endpoints and sensitive information extracted from JavaScript files, a list of captured dynamic interfaces, and potential attack surface analysis. Furthermore, the report content can be written to the `recon_result` field of a JSON-formatted status file via a specific script for structured persistent storage, to be used for subsequent penetration testing task planning. In addition, the information collection method of this application can be scheduled and executed by an intelligent agent, without specific limitations.
[0090] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an information collection system provided in an embodiment of this application.
[0091] This application provides an information collection system applicable to products such as automated penetration testing platforms, security assessment tools, and red team attack and defense drill systems, and may include: The asset crawling module 101 is used to obtain a list of candidate assets after crawling the target object. The information collection module 102 is used to perform corresponding information collection tasks according to the candidate asset list; The model input module 103 is used to input the context of the information collection task into the large language model in response to the blocking that triggers the information collection task; The strategy acquisition module 104 is used to acquire the blocking response strategy of the information collection task output by the large language model; The blocking processing module 105 is used to process the information collection task according to the blocking response strategy in order to obtain the information collection results of the target object.
[0092] This application provides an information collection system in which an asset crawling module is specifically used to: obtain the identifier of a target object; crawl all assets using various search syntaxes based on the identifier of the target object to obtain an original asset list; filter and expand the initial asset list based on ICP filing information to obtain an initial asset list; filter the initial asset list based on IP attribution information to obtain a filtered asset list; and expand and deduplicate the filtered asset list based on the sub-objects associated with the target object to obtain a candidate asset list. The various search syntaxes include domain name search, certificate search, title search, body search, icon search, and C-segment search.
[0093] This application provides an information collection system. The information collection module is specifically used for: segmenting a candidate asset list to obtain segmented asset data; inputting the segmented asset data into a large language model; obtaining the value level obtained by the large language model after semantic analysis, contextual reasoning, and cross-association processing of the segmented asset data; determining the asset priority of the candidate asset list based on the value level; and executing corresponding information collection tasks on the candidate asset list according to the asset priority. Semantic analysis is used to infer the business meaning of asset identifiers by combining the industry context of the target object; contextual reasoning is used to comprehensively infer and determine the value level based on multi-dimensional signals of the asset, including HTTP response features, page titles, response header information, and JS reference information; and cross-association is used to adjust the asset value level based on the relationships between assets.
[0094] This application provides an information collection system. The information collection task includes an active information collection task. The information collection module is specifically used for: processing assets to be processed in the candidate asset list according to a preset extension method to obtain asset baselines and vulnerability clues; performing site database analysis, request crawling, and front-end framework extraction on the assets to be processed through a browser proxy to obtain browser information; extracting authentication information based on the login credentials obtained for the assets to be processed; performing fingerprint recognition, WAF detection, and authentication directory scanning on the asset baseline to obtain asset detection results; performing static auditing on unprocessed JavaScript files in the assets to be processed to obtain static audit information; and generating an active information collection result of the candidate asset list based on the asset baseline, vulnerability clues, browser information, authentication information, asset detection results, and static audit information. The preset extension methods include subdomain enumeration, liveness detection, directory brute-force, JavaScript analysis, port scanning, unauthorized access detection, and attack surface analysis.
[0095] This application provides an information collection system. The information collection module is specifically used for: launching a browser instance through a browser proxy and loading the Uniform Resource Locator (URL) of the target page in the asset to be processed; starting from the entry URL of the target page, traversing the links of the target page to construct a page hierarchy diagram of the site corresponding to the target page; analyzing the URL path patterns, navigation structures, and interactive elements of each page in the page hierarchy diagram to identify the site's functional modules; identifying forms in each page of the page hierarchy diagram and analyzing the forms to obtain form information, including submission address, field name, field type, and validation rules; labeling each page in the page hierarchy diagram with page types based on the form information to obtain page type results; tracking the data flow between pages in the page hierarchy diagram to establish the call relationship between functional modules; and constructing a five-dimensional site structure profile of the asset to be processed based on the page hierarchy diagram, functional modules, and call relationships.
[0096] This application provides an information collection system, in which the information collection task includes a passive information collection task. The information collection module is specifically used to: obtain the traffic files of the assets to be processed in the candidate asset list; perform asset inventory, value classification, JS analysis, API inference, port scanning, vulnerability detection, authentication testing and summarization on the traffic files to obtain the passive information collection results of the candidate asset list.
[0097] This application provides an information collection system in which a blocking processing module is specifically configured to: if the blocking response policy representation allows, then continue executing the information collection task; if the blocking response policy representation is downgraded, then reduce the scanning intensity or range of the information collection task and continue executing; if the blocking response policy representation is skipped, then skip the current step of the information collection task and continue executing subsequent steps; if the blocking response policy representation is redirected, then switch to an alternative path for the information collection task and continue executing; if the blocking response policy representation is terminated, then end the information collection task.
[0098] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, the embodiments of this application also provide an electronic device. Figure 6 This is a schematic diagram of the hardware structure of the electronic device according to an embodiment of this application, as shown below. Figure 6 As shown, the electronic device includes: Communication interface 1 enables information exchange with other devices, such as network devices; Processor 2 is connected to communication interface 1 to enable information interaction with other devices and, when running a computer program, executes the information collection methods provided by one or more of the above-mentioned technical solutions. The computer program is stored in memory 3.
[0099] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 4 The general will label all buses as Bus System 4.
[0100] The memory 3 in this embodiment is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program used to operate on the electronic device.
[0101] It is understood that memory 3 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 3 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.
[0102] The methods disclosed in the embodiments of this application can be applied to processor 2, or implemented by processor 2. Processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 2 or by instructions in the form of software. The processor 2 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 2 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 3. Processor 2 reads the program in memory 3 and completes the steps of the aforementioned method in combination with its hardware.
[0103] When processor 2 executes the program, it implements the corresponding processes in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.
[0104] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 3 that stores a computer program, which can be executed by a processor 2 to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.
[0105] In an exemplary embodiment, this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the preceding embodiments.
[0106] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, terminal, and method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.
[0107] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0108] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0109] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.
[0110] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.
[0111] For descriptions of relevant parts of the information collection system, electronic device, and computer-readable storage medium provided in the embodiments of this application, please refer to the detailed descriptions of the corresponding parts in the information collection method provided in the embodiments of this application, and they will not be repeated here. Furthermore, parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.
[0112] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0113] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An information collection method, characterized in that, include: Obtain a list of candidate assets after crawling the assets of the target object; According to the candidate asset list, perform the corresponding information collection tasks; In response to the blocking that triggers the information gathering task, the context of the information gathering task is input into the large language model; Blocking response strategy for obtaining the information collection task output by the large language model; The information collection task is processed according to the blocking response strategy to obtain the information collection results of the target object.
2. The method according to claim 1, characterized in that, The process of obtaining the candidate asset list after crawling the assets of the target object includes: Obtain the identifier of the target object; Based on the identifier of the target object, a full asset crawl is performed using various search syntaxes to obtain the original asset list; Based on the ICP filing information, the original asset list is filtered and expanded to obtain an initial asset list; Based on IP attribution information, the initial asset list is filtered to obtain a filtered asset list; Based on the sub-objects associated with the target object, the filtered asset list is expanded and deduplicated to obtain a candidate asset list; The various search syntaxes include domain name search, certificate search, title search, body search, icon search, and C-segment search.
3. The method according to claim 1, characterized in that, The step of performing the corresponding information collection task according to the candidate asset list includes: The candidate asset list is divided into blocks to obtain block asset data; Input the segmented asset data into the large language model; The value level is obtained by the large language model after performing semantic analysis, contextual reasoning, and cross-association on the segmented asset data. The asset priority of the candidate asset list is determined based on the value level. According to the asset priority, perform the corresponding information collection task on the candidate asset list; The semantic analysis is used to infer the business meaning of the asset identifier by combining the industry context of the target object; the contextual reasoning is used to determine the value level by comprehensively considering the multi-dimensional signals of the asset, including HTTP response features, page title, response header information and JS reference information; the cross-association is used to adjust the value level of the asset according to the association between assets.
4. The method according to claim 1, characterized in that, The information collection task includes an active information collection task, and the step of executing the corresponding information collection task according to the candidate asset list includes: For the assets to be processed in the candidate asset list, they are processed according to a preset expansion method to obtain asset baselines and vulnerability clues; By using a browser proxy, we perform site database analysis, request crawling, and front-end framework extraction on the assets to be processed to obtain browser information. In response to obtaining the login credentials for the asset to be processed, authentication information is extracted based on the login credentials; The asset baseline is subjected to fingerprint recognition, WAF detection, and authentication catalog scanning to obtain asset detection results; Perform static auditing on unprocessed JavaScript files in the assets to be processed to obtain static audit information; Based on the asset baseline, the vulnerability clues, the browser information, the authentication information, the asset detection results, and the static audit information, the proactive information collection results for generating the candidate asset list are generated. The preset extension methods include subdomain enumeration, liveness detection, directory brute-force, JavaScript analysis, port scanning, unauthorized access detection, and attack surface analysis.
5. The method according to claim 4, characterized in that, The process of performing site database analysis on the assets to be processed via a browser proxy includes: Launch a browser instance via a browser proxy and load the Uniform Resource Locator (URL) of the target page in the asset to be processed; Starting from the entry Uniform Resource Locator (URL) of the target page, traverse the links of the target page to construct a page hierarchy graph of the site corresponding to the target page; Analyze the Uniform Resource Locator (URL) path patterns, navigation structures, and interactive elements of each page in the page hierarchy diagram to identify the functional modules of the site; Identify the forms in each page of the page hierarchy diagram, analyze the forms, and obtain form information, which includes submission address, field name, field type, and validation rules; Based on the form information, the page type is labeled for each page in the page hierarchy diagram to obtain the page type result; Track the data flow between pages in the page hierarchy diagram and establish the calling relationship between the functional modules; Based on the page hierarchy diagram, the functional modules, and the calling relationships, a five-dimensional site structure profile of the asset to be processed is constructed.
6. The method according to claim 1, characterized in that, The information collection task includes a passive information collection task, and the execution of the corresponding information collection task according to the candidate asset list includes: For the pending assets in the candidate asset list, obtain the traffic file of the pending assets; The traffic files are subjected to asset inventory, value classification, JS analysis, API inference, port scanning, vulnerability detection, authentication testing and summarization to obtain the passive information collection results of the candidate asset list.
7. The method according to claim 1, characterized in that, The information collection task is processed according to the aforementioned blocking response strategy, including: If the blocking response strategy indicates that passage is allowed, then the information collection task continues. If the blocking response strategy is characterized as a downgrade, then the scanning intensity or range of the information collection task is reduced and the task continues to be executed. If the blocking response strategy is skipped, the current step of the information collection task is skipped and the subsequent steps are executed. If the blocking response strategy indicates redirection, then the execution will continue after switching to an alternative path for the information gathering task; If the blocking response strategy is terminated, the information collection task ends.
8. An information collection system, characterized in that, include: The asset crawling module is used to obtain a list of candidate assets after crawling the target object. The information collection module is used to perform corresponding information collection tasks according to the candidate asset list; The model input module is used to input the context of the information gathering task into the large language model in response to the blocking that triggers the information gathering task. The strategy acquisition module is used to acquire the blocking response strategy of the information collection task output by the large language model; The blocking processing module is used to process the information collection task according to the blocking response strategy in order to obtain the information collection result of the target object.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the information collection method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the information collection method as described in any one of claims 1 to 7.
