A distributed scanner system implementation method and implementation device supporting orchestration

Abstract

The present invention relates to a method for implementing a distributed scanner system that supports orchestration. The method includes: encapsulating a single scanner into a node in a DAG graph and storing the node in a component library; creating a scan plan in the form of a DAG graph, sequentially creating nodes according to the execution order of the scanners, and selecting the input data of downstream nodes from the output data of upstream nodes; an orchestration engine parsing the scan plan, creating specific scan task records, and inserting them into a task queue based on the execution time of the scan task; the orchestration engine determining the execution order and execution conditions of the nodes in the DAG graph, starting and executing the scan task; and after all scanner nodes have completed scanning, the orchestration engine aggregates and organizes the scan results to generate a scan report and stores it in a database. The method of the present invention can significantly improve the execution efficiency of traditional scanning solutions, and while improving performance, robustness, and flexibility, it also provides a better solution for automation and reducing maintenance costs.

Description

Technical Field

[0001] The present invention belongs to the technical field of network inspection scanners, and in particular relates to a distributed scanner system supporting orchestration, and an implementation method and an implementation device thereof. Background Art

[0002] Asset discovery and vulnerability scanning are essential operations for network administrators during their daily network inspections, especially in private cloud environments. However, conventional scanners often have many limitations when dealing with such large networks, such as:

[0003] 1. Insufficient performance: Most conventional scanners run in a process mode. When facing large networks such as clouds, it is often necessary to split a large network segment into small segments, start multiple processes to scan, and then merge the scan results. This requires certain preliminary preparation work and can easily lead to low scanning efficiency due to insufficient performance.

[0004] 2. Low robustness: Conventional scanners often consume significant resources, such as CPU, memory, and network bandwidth, during operation. When running as a process, the operating system itself does not limit its resource usage, which can easily affect the normal operation of other processes. If a hardware failure or other abnormal interruption in the scanning process occurs, previous scanning results may be lost, wasting all previous efforts.

[0005] 3. Single Functionality and Poor Flexibility: Conventional scanners have relatively limited functionality, with some specializing in port probing, others in vulnerability scanning, others in website scanning, and others in weak password detection. However, in daily network inspections, administrators need a comprehensive understanding of assets, vulnerabilities, and other aspects, so they rely on a combination of multiple scanners. However, without a unified framework for scheduling, administrators must expend considerable effort on coordinating these different scanners. Furthermore, changes to scanning strategies require adjustments to corresponding scripts and other automated tools, resulting in high maintenance costs. Summary of the Invention

[0006] In order to overcome the above-mentioned defects of conventional scanners when facing large networks, the present invention proposes a new distributed scanner system based on an orchestration framework. The distributed scanner system of the present invention can support orchestration, support multi-target and multi-scenario scanning tasks, and can expand new scanners. The present invention realizes independent control of a single scanner and can realize dynamic allocation of resources according to the task execution speed.

[0007] The distributed scanner system of the present invention is deployed and operated based on the Kubernetes architecture, running the scanning process in a pod. Kubernetes enables horizontal scalability and resource management of the scanner. This system can integrate various scanner types, including but not limited to port scanners, vulnerability scanners, weak password scanners, and website scanners. The integrated scanners are encapsulated as nodes in a component library. Administrators create scan plans on the UI based on their needs. These plans are presented as directed acyclic graphs (DAGs). The orchestration framework initiates scan tasks based on the scan plans and outputs scan reports for administrators to view and download.

[0008] Specifically, the present invention provides a distributed scanner system implementation method that supports orchestration, such as Figure 2 As shown, the method includes the following steps:

[0009] S1. Encapsulate scanner nodes: The administrator encapsulates a single scanner into a node in a DAG (directed acyclic graph) on the UI and stores it in the component library.

[0010] S2. Create a scan plan: The administrator creates a scan plan in the form of a DAG graph on the UI. Nodes are created sequentially according to the scanner's execution order. The input data of downstream nodes is selected from the output data of upstream nodes.

[0011] S3. Parsing the scan plan: After receiving the scan plan, the orchestration engine parses it, first traversing the nodes in the DAG graph, determining all node types, creating corresponding Kubernetes Deployments resource files, setting their replicas to 0, and saving them in the database. After the Kubernetes Deployments resource files for the nodes are created, a specific scan task record is created and inserted into the task queue based on the scan task execution time.

[0012] S4. Start and execute a scan task: The administrator starts a scan task on the UI. When the scan task begins, the orchestration engine first traverses the nodes in the DAG graph in a breadth-first manner to determine the execution order and conditions. The started node automatically pulls the scan target from the designated task queue. Once all tasks on the current node are completed, the downstream nodes are immediately notified, and the orchestration engine starts the downstream nodes.

[0013] S5. Generate scan report: After all scanner nodes complete the scan, the orchestration engine aggregates and organizes the scan results to generate a scan report and stores it in the database for administrators to view or download online.

[0014] Furthermore, the present invention supports the implementation of a distributed scanner system in which the scanner nodes encapsulated in step S1 implement a unified node interface class, and the input and output data between the scanner nodes can be flexibly connected;

[0015] The single scanner includes but is not limited to: a port scanner, a vulnerability scanner, a weak password scanner, and a website scanner.

[0016] Furthermore, the present invention supports the orchestration of the distributed scanner system implementation method step S1 also includes: when the administrator adds a new scanner on the UI interface, configures the startup command of the scanner node, the received scanning parameters, and the scanning result formatting parameters. After the task is created and submitted, the background program builds the scanner image according to the node configuration information. After the image is built, it is saved in the node warehouse for use when creating a scanning plan.

[0017] Furthermore, the present invention supports individually setting the parallelism of each node created in step S2 of the orchestrated distributed scanner system implementation method to speed up the scanning speed.

[0018] Furthermore, step S4 of the distributed scanner system implementation method supporting orchestration of the present invention also includes: for the scheduled execution scanning plan, the orchestration engine regularly generates scanning tasks, each scanning task is configured with a separate task queue, and all scanning targets are placed in the corresponding task queue waiting for execution; after the scanning task initialization is completed, the orchestration engine starts the scanning task.

[0019] Furthermore, the scanning targets in the distributed scanner system implementation method supported by the present invention include but are not limited to: IP, domain name, and website.

[0020] Furthermore, step S4 of the distributed scanner system implementation method supported by the present invention also includes: before each node executes, the number of node replicas is modified according to the configured parallelism, and a pod with a corresponding number of replicas is started; after the current node task is executed, the Kubernetes interface is called to set the number of pod replicas to 0 to achieve the effect of elastic resource expansion.

[0021] Furthermore, the scanning report generated in step S5 of the distributed scanner system implementation method supported by the present invention includes but is not limited to: service detection results, weak password detection results, vulnerability information and repair suggestions;

[0022] The file formats supported by the scan report include but are not limited to: doc, excel, html, pdf, and wps.

[0023] On the other hand, the present invention further provides a device for implementing a distributed scanner system that supports orchestration, which implements the steps of the above-mentioned method for implementing a distributed scanner system that supports orchestration when the device is running.

[0024] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the above-mentioned method for implementing a distributed scanner system supporting orchestration are implemented.

[0025] In summary, the present invention provides a distributed scanner system based on an orchestration framework that significantly improves the efficiency of traditional scanning solutions. While improving performance, robustness, and flexibility, it also provides a better solution for automation and reduced maintenance costs. The distributed scanner system supported by the present invention has the following features:

[0026] 1) It is a scanner framework that supports orchestration, supports multi-target and multi-scenario scanning tasks, and can expand new scanners.

[0027] 2) It realizes the independent control of a single scanner and can realize the dynamic allocation of resources according to the task execution speed.

[0028] 3) This scanner system runs on a k8s cluster. The orchestration system only needs to control the replicas of the pod to achieve horizontal expansion. The larger the scale, the higher the performance. Its performance advantage is more obvious when facing large networks.

[0029] 4) This scanner system can limit the resource usage of the scanner through k8s. The isolation between containers also ensures that the scanner and other services do not compete for hardware resources. In addition, when the scanning task is abnormally interrupted, there is no need to worry about the scanning task being interrupted and abandoned halfway. The newly tuned pod can take over the unfinished task and continue scanning.

[0030] 5) When faced with different task scenarios, administrators only need to draw different DAG graphs, without the need to maintain additional scripts and codes. New scanners can also be integrated, making it highly scalable. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the description of the present invention. Obviously, the following drawings are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 Schematic diagram of the overall operation architecture of the distributed scanner system of the present invention.

[0033] Figure 2 This is a flow chart of the distributed scanner system implementation method of the present invention. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions, and advantages of the present invention more clearly apparent, the technical solutions of the present invention will be described clearly and completely below with reference to specific embodiments. Obviously, the described embodiments are only a portion of the embodiments of the present invention, not all of them. The present invention may also be implemented or applied through different specific implementation methods, and the details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention.

[0035] At the same time, it should be understood that the scope of protection of the present invention is not limited to the specific embodiments described below; it should also be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the scope of protection of the present invention.

[0036] Embodiment: A distributed scanner system implementation method supporting orchestration

[0037] like Figure 1 As shown, the administrator can perform operations such as encapsulating scanner nodes, creating scan plans, starting scan tasks, and downloading scan reports on the UI interface.

[0038] The implementation process of this distributed scanner system is as follows:

[0039] 1) Encapsulate Scanner Nodes: First, encapsulate individual scanners into nodes in the DAG graph and store them in the component library. Encapsulated scanners implement a unified node interface class to ensure flexible connection of input and output data between scanners. When administrators add a new scanner in the UI, they must configure the scanner node's startup command, received scanning parameters, and scan result formatting. After creating and submitting the task, the background program will build a scanner image based on the node's configuration information. Once the image is built, it will be saved in the node repository for easy use when creating subsequent scan plans.

[0040] 2) Create a scan plan: Administrators create a scan plan in the UI using a Directed Graph (DAG) graph. When creating a scan plan in the UI, nodes are created sequentially according to the scanner's execution order. Downstream nodes' input data can be selected from upstream nodes' output data. Each node can have its own parallelism setting to speed up the scan. For example, a scan plan might first use masscan for port detection, then use nmap to detect port services and operating system information, and finally use nuclei to perform a vulnerability scan on exposed services.

[0041] 3) Scan Plan Parsing: After receiving the scan plan, the orchestration engine parses it, first traversing the nodes in the DAG graph. After determining all node types, it creates the corresponding Kubernetes Deployments resource file, sets its replicas to 0, and saves it in the database. After the node's Kubernetes Deployments resource file is created, a specific scan task record is created and inserted into the task queue based on the scan task's execution time.

[0042] 4) Start and execute a scan task: The administrator starts a scan task on the UI, primarily for immediate scan scenarios. If the scan is scheduled, the orchestration engine will periodically generate scan tasks. Each scan task is assigned a separate task queue, and all scan targets (IP addresses, domain names, websites, etc.) are queued for execution. After the scan task is initialized, the orchestration engine starts the task.

[0043] When the scanning task begins, the orchestration engine first traverses the nodes in the DAG graph in a breadth-first manner to determine the execution order and execution conditions of the nodes in the graph. Before each node is executed, the number of node replicas is modified according to the configured parallelism, and the pod with the corresponding number of replicas is started. The started node automatically pulls the scanning target from the specified task queue, and the scanning results are updated to the central database by the data bus. After the current node is executed, the Kubernetes interface is called to set the number of pod replicas to 0, so as to achieve the effect of elastic resource expansion. After all the tasks of the current node are completed, the downstream nodes are notified immediately. After all the upstream nodes are executed, the orchestration engine will pull up the downstream nodes and set the number of replicas of the current node to 0. This cycle repeats until the scanning task is completed. Finally, the orchestration engine aggregates the scanning results into a scanning report and stores it in the database for the administrator to download.

[0044] 5) Scan Report Generation: After all scanners complete a scan, the report generation service automatically generates a scan report, which administrators can view online or download as a file. The scan report includes service detection results, weak password detection results, vulnerability information, and remediation suggestions. Scan reports support multiple file formats, including but not limited to: doc, excel, html, pdf, wps, etc.

[0045] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession can make some changes or modifications to the technical content disclosed above without departing from the scope of the technical solution of the present invention to obtain equivalent embodiments with equivalent changes. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A distributed scanner system implementation method supporting orchestration, characterized in that: The method comprises: S1. Encapsulate Scanner Node: The administrator encapsulates a single scanner into a node in the DAG graph on the UI and stores it in the component library. S2. Create a scan plan: The administrator creates a scan plan in the form of a DAG graph on the UI. Nodes are created sequentially according to the scanner's execution order. The input data of downstream nodes is selected from the output data of upstream nodes. S3. Parsing the scan plan: After receiving the scan plan, the orchestration engine parses it, first traversing the nodes in the DAG graph, determining all node types, creating corresponding Kubernetes Deployments resource files, setting their replicas to 0, and saving them in the database. After the Kubernetes Deployments resource files for the nodes are created, a specific scan task record is created and inserted into the task queue based on the scan task execution time. S4. Start and execute a scan task: The administrator starts a scan task on the UI. When the scan task begins, the orchestration engine first traverses the nodes in the DAG graph in a breadth-first manner to determine the execution order and conditions. The started node automatically pulls the scan target from the designated task queue. Once all tasks on the current node are completed, the downstream nodes are immediately notified, and the orchestration engine starts the downstream nodes. S5. Generate scan report: After all scanner nodes complete the scan, the orchestration engine aggregates and organizes the scan results to generate a scan report and stores it in the database for administrators to view or download online.

2. The method for implementing a distributed scanner system supporting orchestration according to claim 1, wherein: The scanner nodes encapsulated in step S1 implement a unified node interface class, and the input and output data between the scanner nodes can be flexibly connected; The single scanner includes: port scanner, vulnerability scanner, weak password scanner, and website scanner.

3. The method for implementing a distributed scanner system supporting orchestration according to claim 1, wherein: Step S1 also includes: when the administrator adds a new scanner on the UI interface, configures the startup command of the scanner node, the received scanning parameters, and the scanning result formatting parameters. After the task is created and submitted, the background program builds the scanner image according to the node configuration information. After the image is built, it is saved in the node warehouse for use when creating a scanning plan.

4. The method for implementing a distributed scanner system supporting orchestration according to claim 1, wherein: Each node created in step S2 is individually set with a parallelism to speed up the scanning.

5. The method for implementing a distributed scanner system supporting orchestration according to claim 1, wherein: Step S4 also includes: for the scheduled scan plan, the orchestration engine regularly generates scan tasks, each scan task is configured with a separate task queue, and all scan targets are placed in the corresponding task queue waiting for execution; after the scan task initialization is completed, the orchestration engine starts the scan task.

6. The method for implementing a distributed scanner system supporting orchestration according to claim 5, wherein: The scanning targets include: IP, domain name, website.

7. The method for implementing a distributed scanner system supporting orchestration according to claim 1, wherein: Step S4 also includes: before each node executes, the number of node replicas is modified according to the configured parallelism, and a pod with a corresponding number of replicas is started; after the current node task is executed, the Kubernetes interface is called to set the number of pod replicas to 0.

8. The method for implementing a distributed scanner system supporting orchestration according to claim 1, wherein: The scan report generated in step S5 includes: service detection results, weak password detection results, vulnerability information and repair suggestions; The scan report supports the following file formats: doc, excel, html, pdf, wps.

9. A distributed scanner system implementation device supporting orchestration, characterized in that: When the device is running, the steps of the method for implementing a distributed scanner system supporting orchestration according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the method for implementing a distributed scanner system supporting orchestration according to any one of claims 1 to 8 are implemented.

Images