Method and device for realizing distributed scanner system supporting arrangement

Abstract

The invention relates to an implementation method of a distributed scanner system supporting arrangement. The method comprises the following steps of: packaging a single scanner into nodes in a DAG (Directed Acyclic Graph) and storing the nodes in a component library; a scanning plan is created in the form of a DAG, nodes are sequentially created according to the execution sequence of a scanner, and input data of downstream nodes are selected from output data of upstream nodes; analyzing the scanning plan by the arrangement engine, creating a specific scanning task record, and inserting the scanning task record into a task queue according to the execution time of the scanning task; determining an execution sequence and execution conditions of nodes in the DAG by the arrangement engine, and starting and executing a scanning task; after all scanner nodes complete scanning, the arrangement engine gathers and arranges scanning results to generate a scanning report, and the scanning report is stored in a database. According to the method, the execution efficiency of a traditional scanning scheme can be remarkably improved, and a better solution is provided for automation and maintenance cost reduction while the performance, robustness and flexibility are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of network inspection scanners, and in particular to a distributed scanner system supporting scheduling and an implementation method and an implementation device thereof. Background Art

[0002] Asset discovery and vulnerability scanning are essential operations for network administrators in their daily network inspection activities, especially for private cloud environments. However, conventional scanners often have many limitations when facing such large networks, such as: 1. Insufficient performance: Most conventional scanners run as processes. When facing large networks such as clouds, it is often necessary to split a large network segment into small segments, start multiple processes for scanning, and then merge the scan results. This requires certain preliminary preparation work and it is easy to cause low scanning efficiency due to insufficient performance.

[0003] 2. Low robustness: Conventional scanners often consume a lot of resources such as CPU, memory and network bandwidth during operation. When running in process mode, the operating system itself does not limit its resource usage, which can easily affect the normal operation of other processes. If there is a hardware failure or other abnormal interruption of the scanning process, the previous scanning results may be lost, and all previous efforts will be wasted.

[0004] 3. Single function and poor flexibility: Conventional scanners have relatively single functions. Some are good at port detection, some are good at vulnerability scanning, some are good at website scanning, some are good at weak password detection, etc. However, in daily network inspection activities, administrators need to have a comprehensive understanding of assets, vulnerabilities, etc., so they can only use a variety of scanners for scanning, but there is a lack of a unified framework for scheduling. Therefore, administrators need to spend a lot of energy to solve the coordination problems between different scanners. Moreover, once the scanning strategy changes, the corresponding scripts and other automation tools also need to be adjusted accordingly, and the maintenance cost is high. Summary of the invention

[0005] In order to overcome the above-mentioned defects of conventional scanners when facing large networks, the present invention proposes a new distributed scanner system based on an orchestration framework. The distributed scanner system of the present invention can support orchestration, support multi-target and multi-scenario scanning tasks, and can expand new scanners. The present invention realizes independent control of a single scanner and can realize dynamic allocation of resources according to the task execution speed.

[0006] The distributed scanner system of the present invention is deployed and operated based on the k8s architecture, runs the scanning process in the pod, and realizes the horizontal expansion and resource management of the scanner through k8s. This system can integrate various types of scanners, including but not limited to port scanners, vulnerability scanners, weak password scanners, web site scanners, etc. The integrated scanners are encapsulated in the component library in the form of nodes. The administrator creates a scanning plan on the UI interface according to the needs. The scanning plan exists in the form of a directed acyclic graph (abbreviated as: DAG graph). The orchestration framework starts the scanning task according to the scanning plan, and finally outputs the scanning report for the administrator to view and download.

[0007] Specifically, the present invention provides a distributed scanner system implementation method supporting orchestration, such as Figure 2 As shown, the method comprises the following steps: S1. Encapsulate scanner nodes: The administrator encapsulates a single scanner into a node in a DAG (directed acyclic graph) on the UI and stores it in the component library; S2. Create a scan plan: The administrator creates a scan plan in the form of a DAG graph on the UI interface, creates nodes in sequence according to the execution order of the scanner, and the input data of the downstream node is selected from the output data of the upstream node; S3. Parsing the scanning plan: After receiving the scanning plan, the orchestration engine parses it, first traverses the nodes in the DAG graph, determines all node types, creates the corresponding Kubernetes Deployments resource file, sets its replicas to 0, and then saves it in the database; after the Kubernetes Deployments resource file of the node is created, it creates a specific scanning task record and inserts it into the task queue according to the execution time of the scanning task; S4. Start and execute scanning tasks: The administrator starts a scanning task on the UI interface. When the scanning task starts, the orchestration engine first traverses the nodes in the DAG graph in a breadth-first manner to determine the execution order and execution conditions of the nodes in the graph; the started node automatically pulls the scanning target from the specified task queue; after all the tasks of the current node are executed, the downstream nodes are immediately notified, and the orchestration engine pulls up the downstream nodes; S5. Generate scan report: After all scanner nodes complete the scan, the orchestration engine aggregates and organizes the scan results to generate a scan report and stores it in the database for administrators to view or download online.

[0008] Furthermore, the present invention supports the arrangement of the distributed scanner system implementation method. The scanner nodes encapsulated in step S1 implement a unified node interface class, and the input and output data between the scanner nodes can be flexibly connected; The single scanner includes but is not limited to: a port scanner, a vulnerability scanner, a weak password scanner, and a web site scanner.

[0009] Furthermore, the distributed scanner system implementation method supported by the present invention also includes step S1: when the administrator adds a new scanner on the UI interface, the startup command of the scanner node, the received scanning parameters, and the scanning result formatting parameters are configured. After the task is created and submitted, the background program builds the scanner image according to the configuration information of the node. After the image is built, it is saved in the node warehouse for use when creating a scanning plan.

[0010] Furthermore, the present invention supports the arrangement of a distributed scanner system to implement the method of step S2, wherein each node created is individually set to have a parallelism to speed up the scanning speed.

[0011] Furthermore, the distributed scanner system implementation method supporting orchestration of the present invention also includes in step S4: for the scanning plan to be executed on a scheduled basis, the orchestration engine periodically generates scanning tasks, each scanning task is configured with a separate task queue, and all scanning targets are placed in the corresponding task queue waiting for execution; after the scanning task is initialized, the orchestration engine starts the scanning task.

[0012] Furthermore, the scanning targets in the distributed scanner system implementation method supported by the present invention include but are not limited to: IP, domain name, website.

[0013] Furthermore, the distributed scanner system implementation method supporting orchestration of the present invention also includes in step S4: before each node executes, the number of replicas of the node is modified according to the configured parallelism, and a pod with a corresponding number of replicas is started; after the current node task is executed, the Kubernetes interface is called to set the number of replicas of the pod to 0, so as to achieve the effect of elastic expansion of resources.

[0014] Furthermore, the scanning report generated in step S5 of the distributed scanner system implementation method supported by the present invention includes but is not limited to: service detection results, weak password detection results, vulnerability information and repair suggestions; The file formats supported by the scan report include but are not limited to: doc, excel, html, pdf, wps.

[0015] On the other hand, the present invention further provides a distributed scanner system implementation device supporting orchestration, which implements the steps of the above-mentioned distributed scanner system implementation method supporting orchestration when running.

[0016] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the above-mentioned distributed scanner system implementation method supporting orchestration are implemented.

[0017] In summary, the present invention provides a distributed scanner system based on an orchestration framework, which can significantly improve the execution efficiency of traditional scanning solutions, and while improving performance, robustness and flexibility, it also provides a better solution for automation and reduced maintenance costs. The distributed scanner system supported by the present invention has the following characteristics: 1) It is a scanner framework that can support orchestration, support multi-target and multi-scenario scanning tasks, and can expand new scanners.

[0018] 2) It realizes independent control of a single scanner and can dynamically allocate resources according to the task execution speed.

[0019] 3) This scanner system runs on a k8s cluster. The orchestration system only needs to control the copies of the pod to achieve horizontal expansion. The larger the scale, the higher the performance. When facing a large network, its performance advantage is more obvious.

[0020] 4) This scanner system can limit the resource usage of the scanner through k8s. The isolation between containers can also ensure that the scanner and other services will not compete for hardware resources. When the scanning task is abnormally interrupted, there is no need to worry about the scanning task being interrupted and abandoned halfway. The newly tuned pod can take over the unfinished task and continue scanning.

[0021] 5) When faced with different task scenarios, administrators only need to draw different DAG graphs without maintaining additional scripts and codes. In addition, new scanners can be integrated, which is highly scalable. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the description of the present invention. Obviously, the following drawings are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is a schematic diagram of the overall operation architecture of the distributed scanner system of the present invention.

[0024] Figure 2 The figure is a flow chart of the distributed scanner system implementation method of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, and the present invention can also be implemented or applied through other different specific implementation methods. The details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0026] At the same time, it should be understood that the protection scope of the present invention is not limited to the specific embodiments described below; it should also be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the protection scope of the present invention.

[0027] Embodiment: A distributed scanner system implementation method supporting orchestration like Figure 1 As shown, the administrator can perform operations such as encapsulating scanner nodes, creating scan plans, starting scan tasks, and downloading scan reports on the UI interface.

[0028] The implementation process of this distributed scanner system is as follows: 1) Encapsulate scanner nodes: First, encapsulate a single scanner into a node in the DAG graph and store it in the component library. The encapsulated scanner will implement a unified node interface class to ensure that the input and output data between scanners can be flexibly connected. When the administrator adds a scanner on the UI interface, it is necessary to configure the scanner node startup command, received scanning parameters, scanning result formatting, etc. After the task is created and submitted, the background program will build a scanner image according to the node configuration information. After the image is built, it will be saved in the node warehouse for use when creating a scanning plan later.

[0029] 2) Create a scan plan: The administrator creates a scan plan in the form of a DAG graph on the UI interface. When the administrator creates a scan plan on the UI interface, nodes are created in sequence according to the execution order of the scanner. The input data of the downstream node can be selected from the output data of the upstream node. Each node can set the parallelism separately to speed up the scan. For example, create a scan plan like this: first use masscan to detect ports, then use nmap to detect port services and operating system information, and then use nuclei to perform vulnerability scans for exposed services.

[0030] 3) Parsing the scanning plan: After receiving the scanning plan, the orchestration engine parses it, first traverses the nodes in the DAG graph, determines all the node types, creates the corresponding Kubernetes Deployments resource file, sets its replicas to 0, and then saves it in the database. After the Kubernetes Deployments resource file of the node is created, a specific scanning task record is created and inserted into the task queue according to the execution time of the scanning task.

[0031] 4) Start and execute scanning tasks: The administrator starts a scanning task on the UI interface, which is mainly used for instant scanning scenarios. If the scanning plan is scheduled, the orchestration engine will generate scanning tasks regularly. Each scanning task will be configured with a separate task queue, and all scanning targets (IP / domain name / website, etc.) will be placed in the queue and wait for execution. After the scanning task is initialized, the orchestration engine starts the task.

[0032] When the scanning task starts, the orchestration engine first traverses the nodes in the DAG graph in a breadth-first manner to determine the execution order and execution conditions of the nodes in the graph. Before each node is executed, the number of node replicas is modified according to the configured parallelism, and the pod with the corresponding number of replicas is started. The started node automatically pulls the scanning target from the specified task queue, and the scanning results are updated to the central database by the data bus. After the current node is executed, the Kubernetes interface is called to set the number of pod replicas to 0, so as to achieve the effect of elastic resource expansion. After all the tasks of the current node are completed, the downstream nodes are notified immediately. After all the upstream nodes are executed, the orchestration engine will pull up the downstream nodes and set the number of replicas of the current node to 0. This cycle repeats until the scanning task is completed. Finally, the orchestration engine aggregates the scanning results into a scanning report and stores it in the database for the administrator to download.

[0033] 5) Generate scan report: After all scanners have completed the scan, the report generation service will automatically generate a scan report, which the administrator can view online or download as a file. The scan report includes service detection results, weak password detection results, vulnerability information, and repair suggestions. The scan report supports multiple file formats, including but not limited to: doc, excel, html, pdf, wps, etc.

[0034] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession can make some changes or modifications to the technical contents disclosed above without departing from the scope of the technical solution of the present invention to obtain equivalent embodiments of equivalent changes. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A distributed scanner system implementation method supporting orchestration, characterized in that: The method comprises: S1. Encapsulate scanner nodes: The administrator encapsulates a single scanner into a node in the DAG graph on the UI and stores it in the component library; S2. Create a scan plan: The administrator creates a scan plan in the form of a DAG graph on the UI interface, creates nodes in sequence according to the execution order of the scanner, and the input data of the downstream node is selected from the output data of the upstream node; S3. Parsing the scanning plan: After receiving the scanning plan, the orchestration engine parses it, first traverses the nodes in the DAG graph, determines all node types, creates the corresponding Kubernetes Deployments resource file, sets its replicas to 0, and then saves it in the database; after the Kubernetes Deployments resource file of the node is created, it creates a specific scanning task record and inserts it into the task queue according to the execution time of the scanning task; S4. Start and execute scanning tasks: The administrator starts a scanning task on the UI interface. When the scanning task starts, the orchestration engine first traverses the nodes in the DAG graph in a breadth-first manner to determine the execution order and execution conditions of the nodes in the graph; the started node automatically pulls the scanning target from the specified task queue; after all the tasks of the current node are executed, the downstream nodes are immediately notified, and the orchestration engine pulls up the downstream nodes; S5. Generate scan report: After all scanner nodes complete the scan, the orchestration engine aggregates and organizes the scan results to generate a scan report and stores it in the database for administrators to view or download online.

2. The method for implementing a distributed scanner system supporting orchestration according to claim 1, characterized in that: The scanner nodes encapsulated in step S1 implement a unified node interface class, and the input and output data between the scanner nodes can be flexibly connected; The single scanner includes: port scanner, vulnerability scanner, weak password scanner, and web site scanner.

3. The method for implementing a distributed scanner system supporting orchestration according to claim 1, characterized in that: Step S1 also includes: when the administrator adds a scanner on the UI interface, configures the scanner node's startup command, received scanning parameters, and scanning result formatting parameters. After the task is created and submitted, the background program builds the scanner image according to the node's configuration information. After the image is built, it is saved in the node warehouse for use when creating a scanning plan.

4. The method for implementing a distributed scanner system supporting orchestration according to claim 1, characterized in that: Each node created in step S2 sets the parallelism individually to speed up the scanning.

5. The method for implementing a distributed scanner system supporting orchestration according to claim 1, characterized in that: Step S4 also includes: for the scan plan to be executed at a scheduled time, the orchestration engine regularly generates scan tasks, each scan task is configured with a separate task queue, and all scan targets are placed in the corresponding task queue waiting for execution; after the scan task is initialized, the orchestration engine starts the scan task.

6. The method for implementing a distributed scanner system supporting orchestration according to claim 5, characterized in that: The scanning targets include: IP, domain name, website.

7. The method for implementing a distributed scanner system supporting orchestration according to claim 1, characterized in that: Step S4 also includes: before each node is executed, the number of replicas of the node is modified according to the configured parallelism, and the pod with the corresponding number of replicas is started; after the current node task is executed, the Kubernetes interface is called to set the number of replicas of the pod to 0.

8. The method for implementing a distributed scanner system supporting orchestration according to claim 1, characterized in that: The scanning report generated in step S5 includes: service detection results, weak password detection results, vulnerability information and repair suggestions; The scan report supports the following file formats: doc, excel, html, pdf, wps.

9. A distributed scanner system implementation device supporting orchestration, characterized in that: When the device is running, the steps of the method for implementing a distributed scanner system supporting orchestration described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the method for implementing a distributed scanner system supporting orchestration according to any one of claims 1 to 8 are implemented.

Images