Dynamic DAST scanning method and system and readable storage medium

Abstract

The invention provides a dynamic DAST scanning method and system and a readable storage medium, and belongs to the field of network security and software engineering automation, and the method comprises the steps: automatically triggering a scanning task when a Developops process runs to a sec security node, and obtaining an interface or URL list; the method comprises the following steps of: processing script rendering, event triggering and redirection by adopting a dynamic crawler acquisition entrance which takes Scrapy as a core and is combined with Pyppeteer, and reducing acquisition failure caused by forbidding through strategies such as a proxy pool, a User-Agent, a Referer and session maintenance; after fingerprint identification and port scanning are carried out on the collected URL, tasks are distributed to xray passive scanning and AWVS active scanning by utilizing Celery scheduling; and after field extraction and format unification are carried out on different engine results, the results are written into MongoDB and duplicate removal is carried out, a customized scanning report is generated and returned to a Developops process to serve as sec node output, and therefore automatic safety scanning and early warning repairing are achieved in the development full life cycle.

Description

A dynamic DAST scanning method, system, and readable storage medium Technical Field

[0001] This invention belongs to the fields of network security and software engineering automation, and in particular relates to a dynamic DAST scanning method, system and readable storage medium. Background Technology

[0002] Currently, mainstream Developer Ops development processes typically do not include a security scanning phase. Developers often prioritize feature implementation, with security testing usually conducted after development is complete. This leads to delayed vulnerability discovery, high remediation costs, and significant maintenance difficulties. Embedding a DAST scanner into the Developer Ops process, performing a DAST security scan on new features and code modules during testing, can expose vulnerabilities earlier and allow for remediation during the development phase, thus improving product security.

[0003] On the other hand, scanners developed based on open-source frameworks may have shortcomings in terms of stability. When embedded in the process to scan interfaces or URLs, they may also introduce the risk of dirty data injection. Furthermore, it is difficult to achieve fully automated embedded scanning, and some functions require manual clicking to complete, making them unsuitable for high-frequency iteration and automated release scenarios.

[0004] With the increasing adoption of DevOps, Agile development, and automated container deployments, enterprises typically need to integrate security capabilities into the entire development lifecycle, creating automated security scanning checkpoints to reduce manpower investment in security departments and cover potential risks in more non-core business modules. Summary of the Invention

[0005] In view of this, the present invention aims to provide a dynamic DAST scanning method, system, and readable storage medium to at least solve one of the problems in the background art.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows: A dynamic DAST scanning method, comprising: S1, when the Developers process reaches a preset security node, automatically triggering a DAST scanning task to obtain a list of interfaces or URLs of the target to be tested; the list of interfaces or URLs is obtained by manual input, or generated and automatically passed in by the previous process node of the security node; S2, sending the list of interfaces or URLs to a dynamic crawler module for dynamic crawling, the dynamic crawler module using Scrapy as the crawler core, and calling a rendering component based on a headless browser in the downloader middleware to perform event triggering, script rendering and redirection processing to obtain a set of URL candidates; S3, applying anti-crawler processing strategies to access requests of the URL candidate set in response to access blocking, login session failure or abnormal request characteristics during the crawling process, the anti-crawler processing strategies at least including proxy pool switching and User-Agen S4. Perform fingerprinting and port scanning on the URL set to generate asset feature information corresponding to the URL set, and write the URL set and asset feature information into the task queue; S5. Distribute the URL set in the task queue to the passive scanning engine and the active scanning engine for security scanning through the asynchronous task scheduling component to obtain a scan result set; S6. Extract fields and normalize the results of the scan result set, and perform deduplication storage in the database according to the preset uniqueness rules; S7. Monitor the scanning status of the passive scanning engine and the active scanning engine. After the scanning is detected to be completed, read the deduplicated results from the database to generate a scan report, and send the scan report back to the Developops process as the output of the sec security node, so that the Developops process can enter the next stage based on the scan report.

[0007] Furthermore, the rendering component mentioned in step S2 is Pyppeteer. After the browser environment is initialized, the dynamic crawler module injects key functions to control the script running environment and complete event triggering and link collection.

[0008] Furthermore, in step S2, the Scrapy Spider extracts new requests based on regular expression rules, CSS selectors, or XPath rules when parsing the response and submits them to the scheduler, causing the engine and scheduler to run in a loop until the task queue is empty.

[0009] Furthermore, the login session maintenance mentioned in step S3 includes registering multiple accounts and rotating the accounts in a manner that simulates normal operation, or automatically refreshing the session information when the session expires.

[0010] Furthermore, step S3 further includes: when page data is dynamically loaded using a script, calling the script execution component to execute the target script to obtain dynamically loaded data, and using the obtained data to supplement the URL candidate set.

[0011] Furthermore, in step S4, the URL set is written to Redis, and in step S6, the deduplicated scan results are written to MongoDB.

[0012] Furthermore, in step S5, the asynchronous task scheduling component is Celery, the passive scanning engine is xray, and the active scanning engine is AWVS.

[0013] Furthermore, the uniqueness rule in step S6 generates a unique key based at least on the URL, vulnerability type, parameter location, and vulnerability feature summary, and performs deduplication storage using the unique key as an index.

[0014] Furthermore, in step S1, the process instance identifier, submission version identifier, or build number corresponding to this scanning task are recorded and displayed in conjunction with the deduplication storage in step S6 and the report in step S7 to achieve traceability.

[0015] Furthermore, step S7 further includes: when the scan report meets the preset release conditions, a release signal is returned to the Developops process; otherwise, a blocking signal is returned along with vulnerability details, causing the process to remain at the sec security node.

[0016] Furthermore, this solution discloses a dynamic DAST scanning system, including: a triggering and access module, a dynamic crawler module, an anti-crawler processing module, an asset identification module, a task scheduling module, a scan execution module, a result normalization and deduplication module, and a report generation and feedback module. The triggering and access module automatically triggers scanning at the sec security node of the Developops process and obtains an interface list or URL list. The dynamic crawler module performs dynamic crawling based on Scrapy and combined with a headless browser rendering component to form a URL candidate set. The anti-crawler processing module applies proxy pool switching, User-Agent switching, Referer setting, and session maintenance strategies to access requests. The asset identification module performs fingerprint recognition and port scanning to generate asset characteristic information. The task scheduling module distributes tasks to the passive scanning engine and the active scanning engine. The scan execution module performs security scanning to obtain a scan result set. The result normalization and deduplication module extracts scan result fields, unifies the format, and deduplicates them according to uniqueness rules before storing them in the database. The report generation and feedback module generates a scan report after scanning and feeds it back to the Developops process as the output of the sec security node.

[0017] Furthermore, this solution discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described.

[0018] Compared with the prior art, the dynamic DAST scanning method, system and readable storage medium of the present invention have the following advantages: (1) The present invention automatically triggers the DAST scanning task when the Developops process runs to the sec security node, and sends the scanning report back to the Developops process as the sec node output after the scan is completed, so that the process can automatically enter the next stage according to the report, reducing manual intervention and improving process consistency and execution efficiency; (2) The present invention adopts the dynamic crawling mechanism of Scrapy combined with Pyppeteer, which can handle dynamic scenarios such as script rendering, event triggering, form processing and redirection, avoiding The problem of insufficient entry point discovery in front-end framework pages such as Vue is addressed by the non-static crawler, thereby increasing the number of discoverable URLs and function entry points, and providing a more complete target set for subsequent vulnerability scanning; (3) This invention introduces strategies such as proxy pool switching, User-Agent switching, Referer setting and login session maintenance during the crawling stage, and can supplement the data obtained by script execution when the data is dynamically loaded by the script, thereby reducing the probability of crawling failure caused by factors such as blocking and session failure, and improving the stability of DAST scanning tasks in continuous iteration scenarios; (4) This invention can generate resources outside the URL set by limiting fingerprint recognition and port scanning. (5) This invention uses Celery to distribute tasks to xray passive scanning and AWVS active scanning for parallel execution, so that the system can cover both passive detection risks and active detection risks, improve the ability to discover different types of vulnerabilities, and facilitate security self-checks during the development phase; (6) This invention extracts fields and standardizes results from different formats output by engines such as xray and AWVS, and performs comparison and deduplication on the database side to ensure the uniqueness of the data entering the database and reduce duplicate vulnerabilities. (7) This invention generates a customized report by reading the deduplicated results from the database after scanning and sending it back to the Developops process. This can unify the report fields and presentation structure, and can combine organizational information to ensure that the report is accurately delivered to the developers, thereby improving the operability of vulnerability location, reproduction and repair. (8) This invention records the process instance identifier, submission version identifier or build number when the scan is triggered and displays them in association with the results and reports. This can achieve full traceability from process to scan results, which is convenient for quickly locating the corresponding version and change source in subsequent retesting, regression verification and problem location.(9) This invention binds the output of the sec security node to the allowance conditions. When the report meets the preset allowance conditions, it returns an allowance signal; otherwise, it returns a blockage signal carrying vulnerability details. This enables the Developops process to form an executable security threshold control before release, reducing the probability of high-risk vulnerabilities entering subsequent stages and improving security assurance capabilities from the engineering process level. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 is a schematic flowchart of the method described in an embodiment of the invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0021] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0022] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0023] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] Example 1: Dynamic DAST Scanning Embedded in the Developops Process - Overall Implementation Example. This example provides the implementation method. The Developops process includes at least build, test, and sec security scan nodes. The DAST scanner is embedded into the sec node of Developops as an interface service. When the process runs to the sec node, a DAST scan task is automatically triggered, ensuring that new features or code modules undergo at least one security scan during the testing phase, so as to discover vulnerabilities as early as possible and complete fixes during development.

[0025] In this embodiment, the input for the DAST scanning task is an interface list or a URL list. The interface list or URL list can be manually entered on the front-end page and then distributed, or it can be automatically generated by the node above the sec node based on the scanner's required fields and then passed to the scanner as a scanning interface document, which is automatically crawled by the scanner to form the initial entry set.

[0026] To support automated operation within the process, the scanner maintains task records and task status, including at least the following fields: task identifier, target domain name / target URL, trigger source, creation time, current stage status, and stage result summary. Upon triggering, the scanner executes the following sequentially: dynamic crawler data collection, anti-anti-crawler processing, fingerprint recognition and port scanning, passive and active scanning, result normalization and deduplication, and report generation and feedback.

[0027] Once the report is generated, the scanner sends the report (or report address, summary of key conclusions) back to the Sec node of Developops. Developops detects that the DAST scan has ended, closes the scan, and proceeds to the next stage. If security thresholds need to be set in the engineering implementation, the number of high-risk vulnerabilities and the marking of blocking vulnerabilities in the report can be used as part of the Sec node's allowance conditions, so that the process stops at the Sec node and prompts for rectification when there are high-risk issues.

[0028] Example 2: Dynamic Web Crawler Module Implementation (Scrapy combined with Pyppeteer) This example provides a specific implementation of a dynamic web crawler module. The crawler module uses Scrapy as its core framework, receives task information from the front end or reads task information from the task database, calls start_requests() to generate the initial request URL, and specifies a callback parsing function for the initial request.

[0029] Scrapy's Engine is responsible for communication, signal and data transmission with various middleware components, sending requests generated by the Spider to the Downloader for resource requests. The Downloader internally contains a download processor and a downloader middleware manager. The downloader middleware manager processes requests / responses in a predetermined order. To handle dynamic scenarios such as event triggering, JS rendering, and redirection in web pages, a Pyppeteer rendering component is added at the end of the downloader middleware chain: when the requested page involves script rendering, asynchronous loading, redirection, or needs to trigger front-end events, Pyppeteer drives the headless browser to load the page and perform necessary event triggering, rendering, and redirection processing, obtaining the rendered Response which is then returned to the Scrapy Engine, which in turn hands it to the Spider for parsing.

[0030] Spider focuses on writing callback methods such as parse to parse the response content according to rule rules (regular expressions, CSS selectors, XPath, etc.), extract new requests, and submit them to the Engine and Scheduler for scheduling. The Engine and Scheduler run in a loop until there are no new requests in the task queue, thus forming a set of URL candidates.

[0031] In an optional implementation, to improve the ability to discover entry points for complex front-end pages, Pyppeteer can hook or inject scripts into key functions after initializing the browser environment to simulate user clicks, form submissions, and other operations, collect page redirection links, and supplement the discovery of hidden entry points; at the same time, it parses and records the backend redirection links to ensure that the final URL set is more complete.

[0032] Example 3: Anti-anti-crawler processing example. Dynamic crawlers may encounter situations such as blocking, access frequency restrictions, session failure, or dynamic data loading when running on the target system, resulting in collection failure. This embodiment introduces a set of anti-anti-crawler strategies during the crawling phase, executing at least one or more strategies in combination for access requests to the URL candidate set: 1) Proxy pool strategy: Constructing different IPs through a proxy pool to initiate requests to the target URL, thereby reducing the risk of IP blocking due to abnormal access volume; 2) Request header feature strategy: Setting or randomly rotating the User-Agent; when generating the next request, setting the current URL as the Referer field, or setting it as a unified entry point, thereby reducing the probability of being identified as a crawler; 3) Login session maintenance strategy: When the target system requires login, session expiration may occur; multiple accounts can be registered and normal operations can be simulated for rotation, or the session information can be automatically refreshed after detecting session expiration and the request can be retried; 4) Dynamic loading and script execution strategy: For pages with dynamically loaded data, automated testing tools can be used to drive the browser to simulate user operations and parse the webpage to obtain data; or a library that can execute JS code can be used to execute JS to obtain dynamic data; if the data is encrypted or decrypted, the JS encryption method can be located and the JS code can be run in the Python environment to obtain the corresponding encoding, thereby continuing to complete the data acquisition.

[0033] In engineering implementation, to reduce the risk of "dirty data injection," dynamic crawling output can be subjected to security verification and standardization: for example, only URLs matching the target domain name or target port range are retained; non-HTTP / HTTPS protocols and URLs with abnormal formats are filtered; upper limits are set on parameter length, number of repeated redirects, crawling depth, and maximum number of requests to avoid abnormal pages triggering resource consumption or introducing uncontrollable targets. This part is used to enhance the stability and controllability of embedded operation without changing the overall process of the scanner.

[0034] Example 4: Fingerprint Recognition and Port Scanning After completing dynamic crawling and anti-anti-crawler processing, fingerprint recognition and port scanning are performed on the verified set of URLs to generate asset characteristic information. Fingerprint recognition can be used to identify the middleware type, web framework, or component characteristics of the target system; port scanning is used to identify exposed ports and service characteristics. The URL set and asset characteristic information are written into the task queue as context input for the scanning task, providing a basis for subsequent scanning strategy selection, result attribution, and reporting. This step is used to compensate for the lack of fingerprint recognition and port scanning capabilities in traditional scanners.

[0035] In an optional implementation, the set of URLs to be scanned is written to the Redis storage system to form a queue, which is then pulled by the asynchronous scanning module as needed.

[0036] Example 5: Scan Scheduling, Result Normalization and Deduplication, and Report Generation This example describes the implementation of scan execution and report output. After completing the crawler and fingerprint recognition, AWVS and xray are invoked to perform a security scan on the collected URL set. Since the scan results from AWVS and xray are not in a uniform format and do not meet the required format for report output, this example performs field extraction and result normalization on the scan results: only the fields required for the report are extracted and organized according to a preset structure before being stored in a MongoDB database.

[0037] When writing to MongoDB, each record is compared with existing records in the database to remove duplicates, ensuring the uniqueness of the stored data. Optionally, uniqueness rules can be generated based on fields such as URL, vulnerability type, parameter location, and vulnerability feature summary to achieve deduplication storage across engines and duplicate scans.

[0038] For scan scheduling, Celery can be used to implement asynchronous task scheduling, allocating tasks to xray passive scanning and AWVS active scanning: passive scanning facilitates self-checking of functional safety by developers during development, while active scanning covers a wider range of detection types. The scanner monitors the scanning status of xray and AWVS, and after all scanning actions are completed, it reads deduplicated data from MongoDB to generate a scan report.

[0039] In terms of report output, customized report formats can be generated, and department and personnel information can be attached to ensure that the report can be accurately delivered to the corresponding developers; the report content should at least include vulnerability type, vulnerability level, URL, trigger point or parameter location, brief reproduction description and remediation suggestions.

[0040] Finally, once the Developops process detects that the DAST scan has finished, it closes the scan and enters the next stage. The scanner waits for the next interface or URL task to arrive and repeats the above process, thus forming a reusable and automatically running security scanning capability throughout the entire Developops lifecycle.

[0041] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic DAST scanning method, characterized in that, include: S1. When the Developers process reaches the preset security node, a DAST scan task is automatically triggered to obtain the interface list or URL list of the target to be tested. The interface list or URL list is obtained by manual input or generated and automatically passed in by the previous process node of the security node. S2. The interface list or URL list is sent to the dynamic crawler module to perform dynamic crawling. The dynamic crawler module uses Scrapy as the crawler core and calls the rendering component based on the headless browser in the downloader middleware to perform event triggering, script rendering and redirection processing to obtain a set of URL candidates. S3. For access blocking, login session failure, or abnormal request characteristics during the crawling process, apply anti-crawler processing strategies to the access requests of the URL candidate set. The anti-crawler processing strategies include at least one or more of the following: proxy pool switching, User-Agent switching, Referer setting, and login session maintenance. Output the set of URLs that pass the verification. S4. Perform fingerprint recognition and port scanning on the URL set to generate asset feature information corresponding to the URL set. Write the URL set and asset feature information into the task queue. S5. Distribute the URL set in the task queue to the passive scanning engine and the active scanning engine for security scanning through the asynchronous task scheduling component to obtain a set of scanning results. S6. Extract fields and standardize results from the scan result set, and perform deduplication storage in the database according to preset uniqueness rules; S7. Monitor the scanning status of the passive scanning engine and the active scanning engine. After the scanning is completed, read the deduplicated results from the database to generate a scanning report. Then, send the scanning report back to the Developops process as the output of the sec security node, so that the Developops process can enter the next stage based on the scanning report.

2. The method according to claim 1, characterized in that, The rendering component mentioned in step S2 is Pyppeteer. After the browser environment is initialized, the dynamic crawler module injects key functions to control the script running environment and complete event triggering and link collection.

3. The method according to claim 1, characterized in that, In step S2, Scrapy's Spider extracts new requests based on regular expression rules, CSS selectors, or XPath rules when parsing the response and submits them to the scheduler, causing the engine and scheduler to run in a loop until the task queue is empty.

4. The method according to claim 1, characterized in that, The login session maintenance mentioned in step S3 includes registering multiple accounts and rotating the accounts in a manner that simulates normal operation, or automatically refreshing the session information when the session expires.

5. The method according to claim 1, characterized in that, Step S3 further includes: when page data is dynamically loaded using a script, calling the script execution component to execute the target script to obtain dynamically loaded data, and using the obtained data to supplement the URL candidate set.

6. The method according to claim 1, characterized in that, In step S4, the URL set is written to Redis, and in step S6, the deduplicated scan results are written to MongoDB.

7. The method according to claim 1, characterized in that, In step S1, the process instance identifier, submission version identifier, or build number corresponding to this scanning task are further recorded and displayed in conjunction with the deduplication storage in step S6 and the report in step S7 to achieve traceability.

8. The method according to claim 1, characterized in that, Step S7 further includes: when the scan report meets the preset release conditions, a release signal is returned to the Developops process; otherwise, a blocking signal is returned along with vulnerability details, causing the process to stop at the sec security node.

9. A dynamic DAST scanning system embedded in the Developmentops development process, characterized in that, include: The system comprises a triggering and access module, a dynamic crawler module, an anti-crawler processing module, an asset identification module, a task scheduling module, a scan execution module, a result normalization and deduplication module, and a report generation and feedback module. The triggering and access module automatically triggers scans at the security nodes of the Developops process to obtain a list of interfaces or URLs. The dynamic crawler module uses Scrapy and a headless browser rendering component to perform dynamic crawling to form a set of URL candidates. The anti-crawler processing module applies proxy pool switching, User-Agent switching, Referer settings, and session persistence strategies to access requests. The asset identification module performs fingerprint recognition and port scanning to generate asset characteristic information. The task scheduling module distributes tasks to the passive and active scanning engines; the scan execution module performs security scans to obtain a set of scan results; the result normalization and deduplication module extracts fields from the scan results, standardizes the format, and deduplicates them according to uniqueness rules before storing them in the database; the report generation and feedback module generates a scan report after the scan is completed and sends it back to the Developops process as the output of the sec security node.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 8.

Images