A directory scanning method, device and storage medium in a penetration testing scenario
By generating dictionary configurations and thread counts adapted to the target site, adjusting the dictionary strategy, and building a hit rate training model, the problems of high false positive rate, high false negative rate, and slow speed of directory scanning tools were solved, achieving efficient and accurate directory scanning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INTEGRITY TECH GRP INC
- Filing Date
- 2023-10-24
- Publication Date
- 2026-06-02
AI Technical Summary
Existing directory scanning tools have high false positive and false negative rates and slow scanning speed, which affects testing efficiency.
Based on the target site's operating system, generate dictionary configuration items and thread count. Adjust the dictionary strategy using the initial dictionary library and dictionary configuration items, generate a new dictionary library, build a hit rate training model for scanning, and output the scanning results of sensitive files.
It effectively reduces false alarms and false negatives, improves scanning efficiency, and ensures the accuracy and speed of scanning results.
Smart Images
Figure CN117592047B_ABST
Abstract
Description
Technical Field
[0001] This invention mainly relates to the field of information collection technology, specifically to a directory scanning method, device, and storage medium in a penetration testing scenario. Background Technology
[0002] Directory scanning, also known as sensitive file crawling, is a security testing method designed to scan the directory structure of a website or server to find potential vulnerabilities or security risks. Currently, most mainstream directory scanning principles combine dictionary scanning and fuzzing. Dictionary scanning is one of the most common methods. It uses a pre-prepared dictionary file to attempt to scan each directory and file on the target site to determine which directories and files exist. The dictionary file typically includes common directory and filenames, as well as other potentially existing directories and filenames. The advantages of dictionary scanning are its simplicity and speed, but it may miss some directories and files not in the dictionary, resulting in a large number of false positives. Dictionary scanning often uses multi-threaded scanning techniques, loading directory dictionaries and filtering rules to brute-force the results. The filtering rules are largely similar, mostly involving control languages, timeout periods, and response status codes (whether to follow a redirect).
[0003] Fuzz scanning is a method of fuzzy path matching. Unlike traditional dictionary scanning, fuzzing uses a dictionary file containing fuzzy matching patterns to discover unknown directories and files. These patterns typically include: 1. Fuzzy matching, such as using wildcards () and placeholders (?) to discover unknown directories and files like " / admin / " and " / user / ?? / profile". 2. Mutation matching, such as using case variations, character substitutions, deletions, and insertions to mutate existing directory and filenames to discover similar but unknown directories and files. 3. Translation matching, using translations in different languages to expand the dictionary file to discover directories and files that may exist but are in different languages.
[0004] The shortcomings of existing technology are:
[0005] 1. High false positive rate: Directory scanning tools may generate a large number of false positives during the scanning process, misclassifying non-existent directories or files as existing ones. This can cause testers to waste a lot of time and effort verifying the false positives, thus affecting testing efficiency.
[0006] 2. High false negative rate: Directory scanning tools may miss some existing directories or files during the scanning process, misclassifying some real directories or files as non-existent. This may prevent testers from discovering some real vulnerabilities or security issues.
[0007] 3. Slow speed: Directory scanning tools require multiple HTTP requests and responses when scanning a large number of directories and files, resulting in a relatively slow speed. This may cause testers to spend a considerable amount of time completing a directory scan. Summary of the Invention
[0008] The technical problem to be solved by the present invention is to provide a directory scanning method, apparatus and storage medium for penetration testing scenarios, which addresses the shortcomings of the prior art.
[0009] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A directory scanning method in a penetration testing scenario, comprising the following steps:
[0010] Connect to the target site based on the specified site entry;
[0011] The penetration testing environment in the target site is verified. If the verification is successful, the scanning mode is set, and a single configuration item is loaded under the set scanning mode.
[0012] Generate dictionary configuration items and target thread count based on the current operating system of the target site;
[0013] Import an initial dictionary database, scan and analyze the directories in the target site based on the initial dictionary database and the dictionary configuration items, and adjust the configurable dictionary strategy items in the initial dictionary database according to the dictionary hit rate in the initial dictionary database to generate a new dictionary database;
[0014] The system determines whether there is a next scan task in the task waiting queue based on the target number of threads. If there is, the system scans and analyzes the directories in the target site based on the new dictionary, and adjusts the dictionary variables in the new dictionary based on the dictionary hit rate to generate a new dictionary again. If there is no scan task, the final new dictionary is obtained.
[0015] A hit rate training model is constructed using the final new dictionary database. The directories in the target site are then scanned based on the hit rate training model, and the scan results of sensitive files are output.
[0016] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: A catalog scanning device for penetration testing scenarios, comprising:
[0017] The site connection module is used to connect to the target site based on the specified site entry point;
[0018] The mode setting module is used to verify the penetration testing environment in the target site. When the verification is successful, the scanning mode is set, and a single configuration item is loaded in the set scanning mode.
[0019] The configuration module is used to generate dictionary configuration items and the target number of threads based on the current operating system of the target site;
[0020] The scanning processing module is used to import an initial dictionary database, scan and analyze the directories in the target site based on the initial dictionary database and the dictionary configuration items, and adjust the configurable dictionary strategy items in the initial dictionary database according to the dictionary hit rate in the initial dictionary database to generate a new dictionary database.
[0021] The system determines whether there is a next scan task in the task waiting queue based on the target number of threads. If there is, the system scans and analyzes the directories in the target site based on the new dictionary, and adjusts the dictionary variables in the new dictionary based on the dictionary hit rate to generate a new dictionary again. If there is no scan task, the final new dictionary is obtained.
[0022] The training model building module is used to build a hit rate training model through the final new dictionary, scan the directories in the target site according to the hit rate training model, and output the scanning results of sensitive files.
[0023] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a directory scanning device for penetration testing scenarios, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the directory scanning method for penetration testing scenarios as described above.
[0024] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a computer-readable storage medium storing a computer program, which, when executed by a processor, performs directory scanning in the penetration testing scenario described above.
[0025] The beneficial effects of this invention are: it can generate dictionary configuration items and target thread count based on the current operating system of the target site, load a single configuration item, and continuously adjust dictionary variables based on the dictionary hit rate to gradually match whether the scan configuration meets the expected effect, effectively improving the directory scanning efficiency and effectively reducing false alarms and false negatives. Attached Figure Description
[0026] Figure 1 This is one of the flowcharts illustrating the catalog scanning method provided in an embodiment of the present invention;
[0027] Figure 2A second schematic flowchart of the catalog scanning method provided in an embodiment of the present invention;
[0028] Figure 3 This is a schematic diagram of the functional modules of the catalog scanning device provided in an embodiment of the present invention. Detailed Implementation
[0029] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.
[0030] like Figure 1-2 As shown in the figure, an embodiment of the present invention provides a directory scanning method in a penetration testing scenario, which includes the following steps:
[0031] Connect to the target site based on the specified site entry;
[0032] The penetration testing environment in the target site is verified. If the verification is successful, the scanning mode is set, and a single configuration item is loaded under the set scanning mode.
[0033] Generate dictionary configuration items and target thread count based on the current operating system of the target site;
[0034] Import an initial dictionary database, scan and analyze the directories in the target site based on the initial dictionary database and the dictionary configuration items, and adjust the configurable dictionary strategy items in the initial dictionary database according to the dictionary hit rate in the initial dictionary database to generate a new dictionary database;
[0035] The system determines whether there is a next scan task in the task waiting queue based on the target number of threads. If there is, the system scans and analyzes the directories in the target site based on the new dictionary, and adjusts the dictionary variables in the new dictionary based on the dictionary hit rate to generate a new dictionary again. If there is no scan task, the final new dictionary is obtained.
[0036] A hit rate training model is constructed using the final new dictionary database. The directories in the target site are then scanned based on the hit rate training model, and the scan results of sensitive files are output.
[0037] Optionally, the penetration testing environment verification includes identification via WAF (Web Application Firewall). If anomalies are found, the scan is terminated; otherwise, FLAG probe detection and Locality Sensitive Hash (LSH) algorithm are used to detect pseudo-static URLs and status codes. If anomalies are found, the scan is terminated. This can improve the security and reliability of the penetration testing environment.
[0038] Specifically, the scanning modes include standard scan, fast scan, full scan, and custom scan.
[0039] The training model is a specific tree-shaped data structure, and the scan results are stored in a pre-built result library.
[0040] The analysis and display of the scan results mainly consist of tagging and simulated tree-structured results. Tagging is primarily used to identify sensitive files.
[0041] For example, the scan results are divided into two sections: simulation tests in the strategy configuration and analysis and display of the scan results, highlighting the sensitive files present in the scan results.
[0042] The simulation test performs a filter preview for the currently selected strategy configuration, and continuously adjusts the dictionary filtering strategy based on the dictionary hit rate to gradually match whether the scanned configuration meets the expected results.
[0043] In this embodiment of the invention, a dictionary configuration item and target thread number can be generated based on the current operating system of the target site, and a single configuration item can be loaded. The dictionary variables, i.e. the filtering strategy, are continuously adjusted based on the dictionary hit rate to gradually match whether the scanning configuration meets the expected effect, effectively improving the directory scanning efficiency and effectively reducing the false alarm and false negative rates.
[0044] Optionally, the single configuration item for loading includes configuration items for request method, timeout, thread count threshold, append dictionary, scanned object, dictionary tag, file extension, target language, number of retries, and blacklist.
[0045] Optionally, after loading a single configuration, load the dynamic configuration.
[0046] The step of generating dictionary configuration items and the target thread count based on the current operating system of the target site includes:
[0047] Based on the current operating system, dictionary naming rules and dictionary policy configurable items are defined for dictionary configuration items. The dictionary policy configurable items include dictionary variables, CDN status, program scripting language, recursion level, scan priority, website language, file path, whether to support leaked suffix checks, parent directory path, and directory category.
[0048] The target number of threads is defined based on the thread count threshold in the single configuration item.
[0049] Specifically, the dictionary naming rules are as follows:
[0050] 70% of file paths (derived through boundary value testing) contain "prefix + path + suffix", where the path is covered by common characters [a-z0-a].
[0051] The following characters are not allowed in filenames created in the Windows operating system: "<>|[Enter Return]:*? / \.
[0052] Optionally, the above configuration process can be simulated and tested in advance to improve the selection and setting of dictionary configuration items.
[0053] Optionally, the dictionary variables include multiple rule variables and multiple keywords;
[0054] The multiple rule variables include the prefixes and suffixes of directory file names, the numbers and / or letters in directory file names, camelCase character norms, and regular expression matching norms;
[0055] The keywords include database keywords, installation file keywords, user file keywords, statistics file keywords, and download file keywords.
[0056] Specifically, the principles of dictionary variable analysis are as follows:
[0057] 1) Supports prefixes and suffixes: for example: [wp_]+[admin]+[_].
[0058] 2) Supports camelCase, for example: [aA]dmin, where the character enclosed in a is a camelCase character.
[0059] 3) Supports regular expression matching, for example: admin[00-99]; that is, traversing admin0-admin9, making it easier to write dictionaries.
[0060] 4) Pure numbers (repetition, 00, 000, 0000) (continuous words, 12, 123, 1234).
[0061] 5) Pure letters (reduplicated words, a, aa, aaa) (continuous words, ab, abc, abcd, abcde) (English words bin, admin) (Chinese pinyin denglu, houtai).
[0062] 6) Numbers + Letters: The above permutations and combinations of pure numbers and pure letters.
[0063] 7) Camel case data analysis, word frequency analysis (which letters or words will cause camel case problems), position analysis (which positions will cause camel case problems), pure uppercase letter and special character analysis, and file classification.
[0064] The principles of operating system analysis are:
[0065] 1) Assign files to operating systems based on their file extensions. Some files exist in Linux but not in Windows, such as www.rar.
[0066] 2) Operating systems are also related to scripting languages. For example, the Linux operating system certainly does not include ASP or ASPX scripts. Specifically:
[0067] windows: ".asp", ".aspx", ".rar", ".bat", ".mdb", ".mdf", ".xml".
[0068] linux: ".gz", ".tar", ".tag.gz", ".bz2", ".sh".
[0069] 3) The operating system is mainly used to determine camelCase detection issues (because Windows is case-insensitive, while Linux is relatively case-sensitive, the dictionary will be larger when the operating system is Linux).
[0070] Keyword settings:
[0071] 1) Database keywords: db (alone), base, Backup, data, access.
[0072] 2) Installation file keywords: install, setup.
[0073] 3) User file keywords: register, UserReg, reg_user, Region, (reg alone), member, user.
[0074] 4) Keyword for statistical documents: count.
[0075] 5) Download file keyword: down.
[0076] Optionally, the scanning and analysis of directories in the target site includes:
[0077] Perform dictionary variable analysis, operating system calibration, and keyword analysis on the directory;
[0078] The generation of the new dictionary database includes:
[0079] Based on the number of times each variable in the dictionary variables is hit in the dictionary variable analysis, variables with higher hit rates are loaded first, and a root task is generated based on the loaded variables;
[0080] The execution status of the root task is determined. If the execution status is normal, a request packet is generated and sent. If the execution status is abnormal, the scanning task is reloaded.
[0081] The path of the requested packet is determined. If it is a new path, a new scan and analysis are performed, and blacklist and fuzzy compressed file filtering are applied to generate a new dictionary. If it is not a new path and meets the hierarchical requirements, a new dictionary is generated directly.
[0082] Specifically, when performing blacklist and fuzzy compressed file filtering, blacklist filtering is performed first, followed by fuzzy compressed file filtering. If a fuzzy compressed file exists, a FUZZ scan is loaded and a new dictionary is generated. If no fuzzy compressed file exists, a new dictionary is generated.
[0083] In this embodiment of the invention, the generated new dictionary may include status code, response body, request method, scan time, size, path value, response type, and response title.
[0084] Optionally, the process of generating and sending a request packet further includes a step of determining the request packet sending status:
[0085] If there are request restrictions in the request packet sending status, the target number of threads will be dynamically adjusted. After adjustment, if the number of retries configured in the single configuration item is reached, the scan will be terminated.
[0086] In this embodiment of the invention, the stability of scanning directory files can be improved, and scanning anomalies can be prevented.
[0087] like Figure 3 As shown in the figure, an embodiment of the present invention provides a catalog scanning device for a penetration testing scenario, comprising:
[0088] The site connection module is used to connect to the target site based on the specified site entry point;
[0089] The mode setting module is used to verify the penetration testing environment in the target site. When the verification is successful, the scanning mode is set, and a single configuration item is loaded in the set scanning mode.
[0090] The configuration module is used to generate dictionary configuration items and the target number of threads based on the current operating system of the target site;
[0091] The scanning processing module is used to import an initial dictionary database, scan and analyze the directories in the target site based on the initial dictionary database and the dictionary configuration items, and adjust the configurable dictionary strategy items in the initial dictionary database according to the dictionary hit rate in the initial dictionary database to generate a new dictionary database.
[0092] The system determines whether there is a next scan task in the task waiting queue based on the target number of threads. If there is, the system scans and analyzes the directories in the target site based on the new dictionary, and adjusts the dictionary variables in the new dictionary based on the dictionary hit rate to generate a new dictionary again. If there is no scan task, the final new dictionary is obtained.
[0093] The training model building module is used to build a hit rate training model through the final new dictionary, scan the directories in the target site according to the hit rate training model, and output the scanning results of sensitive files.
[0094] This invention provides a directory scanning device for a penetration testing scenario, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the directory scanning method for a penetration testing scenario as described above.
[0095] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it performs a directory scan in the penetration testing scenario described above.
[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0097] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
[0098] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.
[0099] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0100] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A directory scanning method in a penetration testing scenario, characterized in that, Includes the following steps: Connect to the target site based on the specified site entry; The penetration testing environment in the target site is verified. If the verification is successful, the scanning mode is set, and a single configuration item is loaded under the set scanning mode. Generate dictionary configuration items and target thread count based on the current operating system of the target site; Import an initial dictionary database, scan and analyze the directories in the target site based on the initial dictionary database and the dictionary configuration items, and adjust the configurable dictionary strategy items in the initial dictionary database according to the dictionary hit rate in the initial dictionary database to generate a new dictionary database; The system determines whether there is a next scan task in the task waiting queue based on the target number of threads. If there is, the system scans and analyzes the directories in the target site based on the new dictionary, and adjusts the dictionary variables in the new dictionary based on the dictionary hit rate to generate a new dictionary again. If there is no scan task, the final new dictionary is obtained. A hit rate training model is constructed using the final new dictionary database. The directories in the target site are scanned based on the hit rate training model, and the scan results of sensitive files are output. The step of scanning and analyzing the directories in the target site includes: Perform dictionary variable analysis, operating system calibration, and keyword analysis on the directory; The generation of the new dictionary database includes: Based on the number of times each variable in the dictionary variables is hit in the dictionary variable analysis, variables with higher hit rates are loaded first, and a root task is generated based on the loaded variables; The execution status of the root task is determined. If the execution status is normal, a request packet is generated and sent. If the execution status is abnormal, the scanning task is reloaded. The path of the requested packet is determined. If it is a new path, a new scan and analysis are performed, and blacklist and fuzzy compressed file filtering are applied to generate a new dictionary. If it is not a new path and meets the hierarchical requirements, a new dictionary is generated directly.
2. The catalog scanning method according to claim 1, characterized in that, The single configuration item for loading includes configuration items for request method, timeout, thread count threshold, append dictionary, scanned object, dictionary tag, file extension, target language, number of retries, and blacklist.
3. The catalog scanning method according to claim 1, characterized in that, The step of generating dictionary configuration items and the target thread count based on the current operating system of the target site includes: Based on the current operating system, dictionary naming rules and dictionary policy configurable items are defined for dictionary configuration items. The dictionary policy configurable items include dictionary variables, CDN status, program scripting language, recursion level, scan priority, website language, file path, whether to support leaked suffix checks, parent directory path, and directory category. The target number of threads is defined based on the thread count threshold in the single configuration item.
4. The catalog scanning method according to claim 1, characterized in that, The dictionary variables include multiple rule variables and multiple keywords; The multiple rule variables include the prefixes and suffixes of directory file names, the numbers and / or letters in directory file names, camelCase character norms, and regular expression matching norms; The keywords include database keywords, installation file keywords, user file keywords, statistics file keywords, and download file keywords.
5. The catalog scanning method according to claim 1, characterized in that, The process of generating and sending a request packet also includes a step of determining the request packet sending status: If there are request restrictions in the request packet sending status, the target number of threads will be dynamically adjusted. After adjustment, if the number of retries configured in the single configuration item is reached, the scan will be terminated.
6. A directory scanning device for a penetration testing scenario, employing the directory scanning method according to any one of claims 1 to 5, characterized in that, include: The site connection module is used to connect to the target site based on the specified site entry point; The mode setting module is used to verify the penetration testing environment in the target site. When the verification is successful, the scanning mode is set, and a single configuration item is loaded in the set scanning mode. The configuration module is used to generate dictionary configuration items and the target number of threads based on the current operating system of the target site; The scanning processing module is used to import an initial dictionary database, scan and analyze the directories in the target site based on the initial dictionary database and the dictionary configuration items, and adjust the configurable dictionary strategy items in the initial dictionary database according to the dictionary hit rate in the initial dictionary database to generate a new dictionary database. The number of target threads determines whether there is a next scan task in the task waiting queue. If there is, the directories in the target site are scanned and analyzed according to the new dictionary. The dictionary variables in the new dictionary are adjusted according to the dictionary hit rate in the new dictionary to generate a new dictionary again. If no scanning task exists, the final new dictionary is obtained; The training model building module is used to build a hit rate training model through the final new dictionary, scan the directories in the target site according to the hit rate training model, and output the scanning results of sensitive files. The step of scanning and analyzing the directories in the target site includes: Perform dictionary variable analysis, operating system calibration, and keyword analysis on the directory; The generation of the new dictionary database includes: Based on the number of times each variable in the dictionary variables is hit in the dictionary variable analysis, variables with higher hit rates are loaded first, and a root task is generated based on the loaded variables; The execution status of the root task is determined. If the execution status is normal, a request packet is generated and sent. If the execution status is abnormal, the scanning task is reloaded. The path of the requested packet is determined. If it is a new path, a new scan and analysis are performed, and blacklist and fuzzy compressed file filtering are applied to generate a new dictionary. If it is not a new path and meets the hierarchical requirements, a new dictionary is generated directly.
7. A catalog scanning device for a penetration testing scenario, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the directory scanning method in the penetration testing scenario as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it performs a directory scan in the penetration testing scenario as described in any one of claims 1 to 5.