Vulnerability scanning method, device and distributed vulnerability scanning system

Abstract

The embodiment of the present invention relates to a vulnerability scanning method, device and distributed vulnerability scanning system. The method is applied to any scanning node in the distributed vulnerability scanning system. The distributed vulnerability scanning system includes multiple scanning nodes, a task scheduling center, and a main message queue. The task scheduling center sends a scanning task to the main message queue. The scanning task includes at least one scanning target, and the scanning target includes at least one scanning object. The method includes: reading the scanning object from the main message queue through a first main coroutine; writing a scanning request for indicating a vulnerability scan for the scanning object into a sub-message queue corresponding to the first scanning target to which the scanning object belongs; and speed-limiting the scanning request in the sub-message queue corresponding to the second scanning target bound to the first main coroutine through the first main coroutine including a first sub-coroutine. The scanning speed limit with the scanning target as the granularity can be realized, the accuracy of the scanning speed limit can be improved, and the distributed parallel speed limit can be realized.

Description

Technical Field

[0001] The embodiments of the present invention relate to the computer field, and in particular to a vulnerability scanning method, device and distributed vulnerability scanning system. Background Art

[0002] With the continuous development of information technology and the Internet, cyberspace has become the fifth largest space after land, sea, sky and outer space, and the increasingly frequent and complex network attacks have made the security issues of Web applications increasingly prominent. Manual testing and auditing of the security of Web applications is an extremely complex and time-consuming task for security operations and security testers, while automated Web vulnerability scanners can greatly simplify manual detection of security risks, helping security engineers shift their focus to how to deal with security risks.

[0003] In a web vulnerability scanner, scanning speed limit is a crucial function. If a web vulnerability scanner does not have a speed limit function, it is easy to cause excessive load pressure on online business during the web vulnerability scanning process, which may cause denial of service at the least, or even cause the server to crash in serious cases. In some hidden usage scenarios (such as attack and defense confrontation), too fast scanning speed can easily expose its own goals in advance, or easily trigger the IP access limit set by the web application firewall. Therefore, how to achieve accurate and real-time rate limit and concurrency limit while taking into account scanning efficiency is an urgent problem to be solved. Summary of the invention

[0004] In view of this, in order to solve the technical problem in the prior art of how to achieve accurate and real-time rate limiting and concurrency limiting while taking into account scanning efficiency in a vulnerability scanning scenario, the embodiments of the present invention provide a vulnerability scanning method, device and distributed vulnerability scanning system.

[0005] In a first aspect, an embodiment of the present invention provides a vulnerability scanning method, which is applied to any scanning node in a distributed vulnerability scanning system, wherein the distributed vulnerability scanning system includes a plurality of the scanning nodes, a task scheduling center, and a main message queue, wherein N main coroutines are configured on the scanning node, wherein the main coroutine includes M sub-coroutines, wherein the task scheduling center sends a plurality of scanning tasks to the main message queue, wherein the scanning task includes at least one scanning target, wherein the scanning target includes at least one scanning object, and wherein the method includes:

[0006] Reading a scan object from the main message queue through a first main coroutine, wherein the first main coroutine is any one of the N main coroutines;

[0007] Generate a scan request for instructing to perform a vulnerability scan on the scan object read, and write the scan request into a sub-message queue corresponding to a first scan target, wherein the first scan target is the scan target to which the scan object read belongs, and different scan targets correspond to different sub-message queues;

[0008] The scanning request in the sub-message queue corresponding to the second scanning target is executed by the first sub-coroutine according to the set speed limit method, the first sub-coroutine is any one of the M sub-coroutines included in the first main coroutine, and the second scanning target is the scanning target bound to the first main coroutine.

[0009] In a possible implementation, executing the scan request in the sub-message queue corresponding to the second scan target according to the set speed limit method by the first sub-coroutine includes:

[0010] Obtain a token from the token bucket corresponding to the second scanning target through the first sub-routine;

[0011] When the token is obtained, the scan request is read from the sub-message queue corresponding to the second scan target through the first sub-routine, and the scan request is executed.

[0012] In a possible implementation, the method further includes:

[0013] If the token is not obtained, a preset sleep function is called to control the first main coroutine to sleep and start another main coroutine.

[0014] In a possible implementation, after reading the scan object from the main message queue by the first main coroutine, the method further includes:

[0015] Parsing the scanned object to obtain an identification code of the scan target to which the scanned object belongs;

[0016] When it is determined that the scan target identified by the identification code obtained through analysis is not bound to other main coroutines, a binding relationship is established between the first main coroutine and the scan target identified by the identification code.

[0017] In a possible implementation, the filling rate of the token bucket is set by the task scheduling center in the following manner:

[0018] Determining the number of surviving scanning nodes in the distributed vulnerability scanning system;

[0019] Determine a maximum value of the scanning rates corresponding to the plurality of scanning tasks;

[0020] The maximum value is divided by the number to obtain the filling rate of the token bucket.

[0021] In a possible implementation, the method further includes:

[0022] When it is monitored that the number of tokens in the token bucket corresponding to the second scanning target meets the first set condition, the number of the sub-coroutines included in the first main coroutine is increased.

[0023] In a possible implementation, the method further includes:

[0024] When the token is acquired, determining the number of remaining tokens in the token bucket corresponding to the second scanning target;

[0025] If the number of the remaining tokens meets the second setting condition, the preset count value is increased by 1; if the number of the remaining tokens does not meet the second setting condition, the count value is set to 0;

[0026] When it is determined that the count value reaches the set threshold, it is determined that the number of tokens in the token bucket corresponding to the second scanning target meets a first set condition.

[0027] In a possible implementation, increasing the number of the sub-coroutines included in the first main coroutine includes:

[0028] Increasing the number of the sub-coroutines included in the first main coroutine according to a preset step value;

[0029] Determine whether the number of the increased sub-coroutines exceeds a set number threshold;

[0030] If yes, setting the number of the sub-coroutines included in the first main coroutine to the set number threshold;

[0031] If not, keep the current number of the sub-coroutines unchanged.

[0032] In a second aspect, an embodiment of the present invention provides a distributed vulnerability scanning system, the distributed vulnerability scanning system comprising a plurality of scanning nodes, a task scheduling center, and a main message queue; N main coroutines are configured on the scanning node, and the main coroutine includes M sub-coroutines;

[0033] The task scheduling center sends a plurality of scanning tasks to the main message queue, wherein the scanning tasks include at least one scanning target, and the scanning target includes at least one scanning object;

[0034] The scanning node reads the scanning object from the main message queue through a first main coroutine, wherein the first main coroutine is any one of the N main coroutines;

[0035] Generate a scan request for instructing to perform a vulnerability scan on the scan object read, and write the scan request into a sub-message queue corresponding to a first scan target, wherein the first scan target is the scan target to which the scan object read belongs, and different scan targets correspond to different sub-message queues;

[0036] The scanning request in the sub-message queue corresponding to the second scanning target is speed-limited through the first sub-coroutine, the first sub-coroutine is any one of the M sub-coroutines included in the first main coroutine, and the second scanning target is the scanning target bound to the first main coroutine.

[0037] In a second aspect, an embodiment of the present invention provides a vulnerability scanning device, which is applied to any scanning node in a distributed vulnerability scanning system, wherein the distributed vulnerability scanning system includes a plurality of the scanning nodes, a task scheduling center, and a main message queue, wherein the scanning node is configured with N main coroutines, wherein the main coroutine includes M sub-coroutines, and the task scheduling center sends a plurality of scanning tasks to the main message queue, wherein the scanning task includes at least one scanning target, and the scanning target includes at least one scanning object, and the device includes:

[0038] A reading module, configured to read a scan object from the main message queue through a first main coroutine, wherein the first main coroutine is any one of the N main coroutines;

[0039] A request generation module, used to generate a scan request for instructing to perform a vulnerability scan on the scan object read, and write the scan request into a sub-message queue corresponding to a first scan target, wherein the first scan target is the scan target to which the scan object read belongs, and different scan targets correspond to different sub-message queues;

[0040] A speed limit module, used to execute the scan request in the sub-message queue corresponding to the second scan target by a first sub-coroutine, wherein the first sub-coroutine is any one of the M sub-coroutines included in the first main coroutine, and the second scan target is the scan target bound to the first main coroutine.

[0041] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising: a processor and a memory, wherein the processor is used to execute a vulnerability scanning program stored in the memory to implement the vulnerability scanning method described in any one of the first aspects.

[0042] In a fourth aspect, an embodiment of the present invention provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the vulnerability scanning method described in any one of the first aspects.

[0043] The technical solution provided by the embodiment of the present invention configures a main message queue in a distributed vulnerability scanning system, and the task scheduling center sends multiple scanning tasks to the main message queue, wherein the scanning task includes at least one scanning target, and the scanning target includes at least one scanning object; N main coroutines are configured on any scanning node of the distributed vulnerability scanning system, and the main coroutine includes M sub-coroutines, and any scanning node reads the scanning object from the main message queue through the first main coroutine, generates a scanning request for instructing to perform a vulnerability scan on the read scanning object, and writes the scanning request into the sub-message queue corresponding to the first scanning target to which the scanning object belongs, thereby realizing the division of the scanning object based on the scanning target granularity. Afterwards, the first sub-coroutine executes the scan request in the sub-message queue corresponding to the second scan target according to the set speed limit method, thereby realizing vulnerability scanning and scanning speed limit based on the scan target. Compared with the prior art that can only limit the scan task, this realizes a finer-grained scan speed limit, so that the scan rate can be limited more accurately. At the same time, the distributed vulnerability scanning system provided by the embodiment of the present invention realizes an asynchronous scanning architecture, adopts a multi-coroutine parallel approach, and can support a larger scanning concurrency and reduce the thread scheduling overhead compared with ordinary multi-threaded vulnerability scanners. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the system architecture of a distributed vulnerability scanning system provided by an embodiment of the present invention;

[0045] Figure 2 A flowchart of an embodiment of a vulnerability scanning method provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of dividing scan requests of scan objects under different scan targets by using a double queue provided by an embodiment of the present invention;

[0047] Figure 4 A block diagram of an embodiment of a vulnerability scanning device provided by an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The following first provides an exemplary description of the distributed vulnerability scanning system provided by an embodiment of the present invention:

[0051] See also Figure 1 , is a schematic diagram of the system architecture of the distributed vulnerability scanning system provided by an embodiment of the present invention. Figure 1 The distributed vulnerability scanning system 100 shown includes: a task scheduling center 101, a main message queue 102, and a plurality of scanning nodes ( Figure 1 , shown as scanning node 1 to scanning node n).

[0052] The task scheduling center 101 is used to receive scanning tasks sent by users and send the scanning tasks to the main message queue 102. Here, the scanning tasks may include scanning tasks sent by different users, each scanning task may include at least one scanning target, and the scanning target may include at least one scanning object.

[0053] For example, the scanning task includes two scanning targets: ks***.com and www.ks***.com, wherein the scanning target ks***.com includes two scanning objects: ks***.com / login.php and ks***.com / logout.php.

[0054] It can be seen that the speed limit objects can be divided into fine-grained categories in the embodiments of the present invention.

[0055] The specific implementation of the main message queue 102 may include but is not limited to a Kafka message queue.

[0056] Each scanning node has a communication connection established with the main message queue 102, and can asynchronously read scanning tasks from the main message queue 102. Optionally, the scanning node can establish a communication connection with the main message queue 102 through an aiokafka client.

[0057] It should be noted that, since the minimum division granularity of the scanning task is the scanning object, the scanning node can read the scanning task from the main message queue 102 with the scanning object as the granularity, and each scanning object can only be consumed once.

[0058] Furthermore, in an embodiment of the present invention, each scanning node may be configured with N main coroutines, where N is a positive integer. Accordingly, an EventLoop event loop is established on the scanning node for coroutine scheduling of the N main coroutines. The N main coroutines may reuse the communication connection between the scanning node and the main message queue 102 to asynchronously read the scan object from the main message queue 102.

[0059] The asynchronous mentioned above means that N main coroutines are executed asynchronously. For example, two main coroutines are configured on the scanning node: main coroutine 1 and main coroutine 2. When main coroutine 1 is executed, main coroutine 2 is dormant; when main coroutine 1 is dormant and gives up the execution right, main coroutine 2 is executed.

[0060] Furthermore, each main coroutine includes M sub-coroutines, where M is a positive integer. The sub-coroutines are used to perform vulnerability scanning and scan rate limiting. As for how the sub-coroutines perform vulnerability scanning and scan rate limiting, it will be explained in conjunction with specific embodiments below, and will not be described in detail here.

[0061] The distributed vulnerability scanning system provided by the embodiment of the present invention implements an asynchronous scanning architecture and adopts a multi-coroutine parallel approach. Compared with ordinary multi-threaded vulnerability scanners, it can support a larger scanning concurrency and reduce the overhead of thread scheduling.

[0062] The vulnerability scanning method provided by the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0063] See also Figure 2 , is a flow chart of an embodiment of a vulnerability scanning method provided by an embodiment of the present invention. As an embodiment, the method can be applied to any scanning node in a distributed vulnerability scanning system, such as Figure 1 In the distributed vulnerability scanning system 100 shown in FIG. 1 , any one of scanning nodes 1 to scanning node n is scanned. Figure 2 As shown, the method may include the following steps:

[0064] Step 201 : Read a scan object from a main message queue through a first main coroutine, wherein the first main coroutine is any one of N main coroutines.

[0065] Step 202: Generate a scan request for instructing to perform a vulnerability scan on the read scan object, and write the scan request into a sub-message queue corresponding to a first scan target, wherein the first scan target is the scan target to which the read scan object belongs, and different scan targets correspond to different sub-message queues.

[0066] The following is a unified description of step 201 to step 202:

[0067] In an embodiment of the present invention, each main coroutine can be bound to a scanning target, and each scanning target can be bound to a speed limiter module. Here, the main coroutine is bound to the scanning target, which means that the sub-coroutine included in the main coroutine performs vulnerability scanning and scanning speed limit on the scanning object under the scanning target. The scanning target is bound to the speed limiter module, which means that the speed limiter module is responsible for scanning and speed limiting the scanning target.

[0068] It can be seen that in the embodiment of the present invention, the speed limit object can be accurately determined to the scanning target level, that is, vulnerability scanning restrictions can be imposed on a certain domain name / IP address, so that the scanning speed limits of different users, different scanning tasks, and different scanning targets will not be affected.

[0069] It should be noted that in the embodiment of the present invention, the binding relationship between the main coroutine and the scanning target, and the binding relationship between the scanning target and the speed limiter are not statically set, but dynamically set. In one embodiment, the binding relationship between the main coroutine and the scanning target, and the binding relationship between the scanning target and the speed limiter module can be dynamically set by the following process:

[0070] After the scanning node reads the scanning object from the main message queue through the first main coroutine, it can parse the scan object read to obtain the identification code of the scanning target to which the scanning object belongs. Here, the identification code of the scanning target can uniquely identify a scanning target, so different scanning objects included in the scanning target have the same identification code.

[0071] Afterwards, the first main coroutine determines whether the scanning target identified by the identification code obtained by the above analysis has been bound to other main coroutines. In an optional implementation, the first main coroutine may search for the identification code obtained by the above analysis in the identification code list recorded locally on the scanning node. The identification code list records the correspondence between the identification code and the main thread, and the correspondence indicates that the scanning target identified by the identification code has been bound to the main thread corresponding to the identification code. Therefore, if the identification code is found, it can be further determined whether the main coroutine bound to the identification code is the first main coroutine. If it is not the first main coroutine, it can be determined that the scanning target identified by the identification code has been bound to other main coroutines; conversely, if the identification code is not found, it can be determined that the identification code is not bound to other main coroutines.

[0072] When it is determined that the scanning target identified by the identification code obtained by the above analysis is not bound to other main coroutines, a new Target object can be created, the Target object is set to process the scanning target identified by the identification code, and a speed limiter module is bound to the Target object. In this way, the binding relationship between the first main coroutine and the scanning target identified by the identification code is established, and the binding relationship between the scanning target identified by the identification code and the speed limiter module is established.

[0073] Furthermore, in an embodiment of the present invention, a sub-message queue is also created for the Target object, that is, a sub-message queue is created for the scan target bound to the Target object. Based on this, the first main coroutine can write the scan request to the sub-message queue corresponding to the scan target to which the scan object belongs (for the convenience of description, referred to as the first scan target) after generating a scan request for instructing to perform a vulnerability scan on the read scan object. It can be seen that different scan targets correspond to different sub-message queues, thereby realizing the division of scan requests for scan objects under different scan targets.

[0074] In addition, when it is determined that the scan target identified by the identification code obtained by the above analysis is not bound to other main coroutines, the corresponding relationship between the identification code and the first main coroutine is added to the above identification code list.

[0075] When it is determined that the scanning target identified by the identification code obtained by the above analysis is bound to other main coroutines, the created Target object can be loaded according to the identification code obtained by the analysis, and after generating a scanning request for instructing to perform a vulnerability scan on the read scanning object, the scanning request is written into the sub-message queue corresponding to the loaded Target object, that is, written into the sub-message queue corresponding to the scanning target to which the scanning object belongs (that is, the first scanning target).

[0076] Optionally, the specific implementation of the first main thread generating a scanning request for instructing to perform a vulnerability scan on the read scanning object includes: the first main thread assembles the information of the scanning object and the payloads under the corresponding scanning scenario to obtain a scanning request for instructing to perform a vulnerability scan on the read scanning object.

[0077] For example, there are two payloads in the weak password scanning scenario:

[0078] payload1: user=root&passwd=123456;

[0079] payload2: user=admin&passwd=123456.

[0080] Then, for the scanning objects ks***.com / login.php and ks***.com / logout.php under the scanning target ks***.com, payloads are assembled respectively, and the following scanning requests can be obtained:

[0081] request1:ks***.com / login.php? user=root&passwd=123456;

[0082] request2:ks***.com / login.php? user=admin&passwd=123456;

[0083] request3:ks***.com / logout.php? user=root&passwd=123456;

[0084] request4:ks***.com / logout.php? user=admin&passwd=123456.

[0085] As can be seen from the above description, in the embodiment of the present invention, dual queues (including a main message queue and a sub-message queue) are used to implement the division of scan requests for scan objects under different scan targets in a distributed vulnerability scanning system. Figure 3 , is a schematic diagram of dividing scan requests of scan objects under different scan targets through double queues provided by an embodiment of the present invention.

[0086] Figure 3 In the example, target1 and target2 are scan targets, and request represents a scan request. After acquiring the two scan targets, scan node 1 and scan node 2 write the scan requests of the scan objects under the two scan targets into different sub-message queues.

[0087] Step 203: execute the scan request in the sub-message queue corresponding to the second scan target through the first sub-coroutine according to the set speed limit method, the first sub-coroutine is any one of the M sub-coroutines included in the first main coroutine, and the second scan target is the scan target bound to the first main coroutine.

[0088] From the above description, it can be seen that the sub-coroutine is used to perform vulnerability scanning and scan rate limiting of the scan target bound to the main thread to which it belongs. Based on this, any sub-coroutine included in the first main coroutine (for the convenience of description, referred to as the first sub-coroutine) executes the scan request in the sub-message queue corresponding to the second scan target according to the set rate limit method. Here, the second scan target is the scan target bound to the first main coroutine.

[0089] In one embodiment, the specific implementation of the first sub-coroutine executing the scan request in the sub-message queue corresponding to the second scan target according to the set speed limit method includes: the first sub-coroutine obtains the token from the token bucket in the speed limiter module bound to the second scan target, that is, the first sub-coroutine obtains the token from the token bucket corresponding to the second scan target. It can be understood that, if the token is obtained, the first sub-coroutine can read the scan request from the sub-message queue corresponding to the second scan target and execute the scan request; if the token is not obtained, the first sub-coroutine calls the preset sleep function to control the first main coroutine to sleep and start another main coroutine.

[0090] It can be seen that in the embodiment of the present invention, the scanning speed limit is respectively performed for different scanning targets, and the speed limit object is accurately set to the scanning target level, that is, the vulnerability scanning limit is performed on a certain domain name / IP address, so that the scanning speed limit between different users, different scanning tasks, and different scanning targets will not be affected. At the same time, in the embodiment of the present invention, when the scanning rate is too fast, the main coroutine can actively give up the execution right, thereby suspending the execution of the current scanning task and processing other scanning tasks, which can maximize the use of system resources.

[0091] It is understandable that in the token bucket rate limiting algorithm, the rate limiting rate can be used as the filling rate to fill tokens into the token bucket. In this regard, an embodiment of the present invention also provides a method for determining the rate limiting rate, that is, the token filling rate, in a distributed vulnerability scanning system, including the following steps:

[0092] Step a: The task scheduling center determines the number of surviving scanning nodes in the distributed vulnerability scanning system.

[0093] In the embodiment of the present invention, the task scheduling center can monitor the number of surviving scanning nodes in the distributed vulnerability scanning system in real time through service discovery.

[0094] Step b: The task scheduling center determines the maximum value of the scanning rates corresponding to the multiple scanning tasks.

[0095] Step c: The task scheduling center divides the above maximum value by the number of surviving scanning nodes in the distributed vulnerability scanning system to obtain the filling rate of the token bucket.

[0096] Assuming that the number of surviving scanning nodes in the distributed vulnerability scanning system is n, and assuming that the maximum value of the scanning rate corresponding to multiple scanning tasks is v, then when the task scheduling center issues scanning tasks, it can specify a scanning rate limit of no more than

[0097] Furthermore, the overall rate limit of the distributed vulnerability scanning system Less than or equal to v.

[0098] It can be seen that the technical solution provided by the embodiment of the present invention is different from the speed limiting method using redis as the rate synchronization in the traditional distributed speed limiting framework. A decentralized solution is proposed, which uses the task scheduling center to achieve the average speed limit rate among multiple scanning nodes, so that the overall scanning rate does not exceed the cumulative result of the scanning rates of each scanning node.

[0099] In addition, in the embodiment of the present invention, when the scanning rate is too low, the number of sub-routines may be increased to improve the scanning rate.

[0100] In one embodiment, whether the scanning rate is too low can be monitored by monitoring the number of remaining tokens in the token bucket. Specifically, after each acquisition of tokens from the token bucket corresponding to the second scanning target, the number of remaining tokens in the token bucket can be determined. If the number of remaining tokens meets the second set condition, the preset count value is increased by 1; if the number of remaining tokens does not meet the second set condition, the count value is set to 0. Afterwards, when it is determined that the count value reaches the set threshold, it is determined that the number of tokens in the token bucket corresponding to the second scanning target meets the first set condition. When it is determined that the number of tokens in the token bucket corresponding to the second scanning target meets the first set condition, it can be determined that the current scanning rate is too low.

[0101] The second setting condition may be: the number of remaining tokens is greater than half of the upper limit of the number of tokens. The setting threshold may be 100. That is, when vulnerability scans are performed on the second scan target 100 times in a row, and the number of remaining tokens in the token bucket corresponding to the second scan target is greater than half of the upper limit of the number of tokens, it can be considered that the scanning rate is too low.

[0102] In one embodiment, when increasing the number of sub-coroutines included in the first main coroutine, a step-by-step increase method may be adopted, and it is ensured that the number of sub-coroutines included in the first main coroutine does not exceed a set number threshold. Based on this, in an optional implementation method, the specific implementation of increasing the number of sub-coroutines included in the first main coroutine includes: increasing the number of sub-coroutines included in the first main coroutine according to a preset step value, that is, adding a set step value on the basis of the current number of sub-coroutines, such as 3, to determine whether the number of increased sub-coroutines exceeds the set number threshold; if so, setting the number of sub-coroutines included in the first main coroutine to the set number threshold; if not, keeping the current number of sub-coroutines unchanged.

[0103] It can be seen that in the embodiment of the present invention, by automatically increasing the number of sub-coroutines when the scanning rate is too low, it is possible to automatically increase the number of concurrent coroutines of the scanning task when the scanning task is blocked due to network, system scheduling, etc., resulting in the actual scanning rate being far less than the established limit rate, so that the scanning rate is close to the ideal state.

[0104] The technical solution provided by the embodiment of the present invention configures a main message queue in a distributed vulnerability scanning system, and the task scheduling center sends multiple scanning tasks to the main message queue, wherein the scanning task includes at least one scanning target, and the scanning target includes at least one scanning object; N main coroutines are configured on any scanning node of the distributed vulnerability scanning system, and the main coroutine includes M sub-coroutines, and any scanning node reads the scanning object from the main message queue through the first main coroutine, generates a scanning request for instructing to perform a vulnerability scan on the read scanning object, and writes the scanning request into the sub-message queue corresponding to the first scanning target to which the scanning object belongs, thereby realizing the division of the scanning object based on the scanning target granularity. Afterwards, the first sub-coroutine executes the scan request in the sub-message queue corresponding to the second scan target according to the set speed limit method, thereby realizing vulnerability scanning and scanning speed limit based on the scan target. Compared with the prior art that can only limit the scan task, this realizes a finer-grained scan speed limit, so that the scan rate can be limited more accurately. At the same time, the distributed vulnerability scanning system provided by the embodiment of the present invention realizes an asynchronous scanning architecture, adopts a multi-coroutine parallel approach, and can support a larger scanning concurrency and reduce the thread scheduling overhead compared with ordinary multi-threaded vulnerability scanners.

[0105] Corresponding to the above-mentioned embodiment of the vulnerability scanning method, the present invention also provides an embodiment of a vulnerability scanning device.

[0106] See also Figure 4 , is a block diagram of an embodiment of a vulnerability scanning device provided by an embodiment of the present invention. As an embodiment, the device can be applied to any scanning node in a distributed vulnerability scanning system. Figure 4 As shown, the device comprises:

[0107] A reading module 41, configured to read a scan object from the main message queue through a first main coroutine, wherein the first main coroutine is any one of the N main coroutines;

[0108] A request generation module 42 is used to generate a scan request for instructing to perform a vulnerability scan on the scan object read, and write the scan request into a sub-message queue corresponding to a first scan target, wherein the first scan target is the scan target to which the scan object read belongs, and different scan targets correspond to different sub-message queues;

[0109] The speed limit module 43 is used to execute the scanning request in the sub-message queue corresponding to the second scanning target through the first sub-coroutine according to the set speed limit method, the first sub-coroutine is any one of the M sub-coroutines included in the first main coroutine, and the second scanning target is the scanning target bound to the first main coroutine.

[0110] In a possible implementation, the speed limiting module 43 includes (not shown in the figure):

[0111] A token acquisition unit, used to acquire a token from a token bucket corresponding to a second scanning target through a first sub-routine;

[0112] A rate limiting unit is used to read the scan request from the sub-message queue corresponding to the second scan target through the first sub-coroutine and execute the scan request when a token is obtained.

[0113] In a possible implementation, the device further includes (not shown in the figure):

[0114] The sleep module is used to call a preset sleep function to control the first main coroutine to sleep and start another main coroutine when the token is not obtained.

[0115] In a possible implementation, the device further includes (not shown in the figure):

[0116] A parsing module, configured to parse the scan object read from the main message queue by the first main coroutine to obtain an identification code of a scan target to which the scan object belongs;

[0117] The binding module is used to establish a binding relationship between the first main coroutine and the scanning target identified by the identification code when it is determined that the scanning target identified by the identification code obtained through analysis is not bound to other main coroutines.

[0118] In a possible implementation, the filling rate of the token bucket is set by the task scheduling center in the following manner:

[0119] Determining the number of surviving scanning nodes in the distributed vulnerability scanning system;

[0120] Determine a maximum value of the scanning rates corresponding to the plurality of scanning tasks;

[0121] The maximum value is divided by the number to obtain the filling rate of the token bucket.

[0122] In a possible implementation, the device further includes (not shown in the figure):

[0123] An adding module is used to increase the number of the sub-coroutines included in the first main coroutine when it is monitored that the number of tokens in the token bucket corresponding to the second scanning target meets the first set condition.

[0124] In a possible implementation, the device further includes (not shown in the figure):

[0125] A first determination module, configured to determine the number of remaining tokens in the token bucket corresponding to the second scanning target when a token is acquired;

[0126] A counting module, configured to increase a preset count value by 1 if the number of the remaining tokens satisfies a second set condition; and to set the count value to 0 if the number of the remaining tokens does not satisfy the second set condition;

[0127] The second determination module is used to determine whether the number of tokens in the token bucket corresponding to the second scanning target meets a first set condition when it is determined that the count value reaches a set threshold.

[0128] In a possible implementation manner, the adding module is specifically used to:

[0129] Increasing the number of the sub-coroutines included in the first main coroutine according to a preset step value;

[0130] Determine whether the number of the increased sub-coroutines exceeds a set number threshold;

[0131] If yes, setting the number of the sub-coroutines included in the first main coroutine to the set number threshold;

[0132] If not, keep the current number of the sub-coroutines unchanged.

[0133] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, Figure 5 The electronic device 500 shown includes: at least one processor 501, a memory 502, at least one network interface 504 and other user interfaces 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 505 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 505 is not shown in FIG. Figure 5 Various buses are labeled as bus system 505 .

[0134] The user interface 503 may include a display, a keyboard or a pointing device (eg, a mouse, a trackball), a touch pad or a touch screen.

[0135] It can be understood that the memory 502 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0136] In some implementations, the memory 502 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 5021 and an application program 5022 .

[0137] The operating system 5021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 5022 includes various application programs, such as a media player (MediaPlayer), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 5022.

[0138] In the embodiment of the present invention, by calling the program or instruction stored in the memory 502, specifically, the program or instruction stored in the application 5022, the processor 501 is used to execute the method steps provided by each method embodiment, for example, including:

[0139] Reading a scan object from the main message queue through a first main coroutine, wherein the first main coroutine is any one of the N main coroutines;

[0140] Generate a scan request for instructing to perform a vulnerability scan on the scan object read, and write the scan request into a sub-message queue corresponding to a first scan target, wherein the first scan target is the scan target to which the scan object read belongs, and different scan targets correspond to different sub-message queues;

[0141] The scanning request in the sub-message queue corresponding to the second scanning target is executed by the first sub-coroutine according to the set speed limit method, the first sub-coroutine is any one of the M sub-coroutines included in the first main coroutine, and the second scanning target is the scanning target bound to the first main coroutine.

[0142] The method disclosed in the above embodiment of the present invention can be applied to the processor 501, or implemented by the processor 501. The processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 501 or the instruction in the form of software. The above processor 501 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 502, and the processor 501 reads the information in the memory 502 and completes the steps of the above method in combination with its hardware.

[0143] It is understood that the embodiments described herein can be implemented by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0144] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0145] The electronic device provided in this embodiment may be Figure 5 The electronic device shown in FIG. 1 may perform the following steps: Figure 2 and Figure 3 All steps of the vulnerability scanning method, thereby achieving Figure 2 and Figure 3 For details, please refer to Figure 2 and Figure 3 For the sake of brevity, the relevant description is omitted here.

[0146] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0147] When one or more programs in the storage medium can be executed by one or more processors, the vulnerability scanning method executed on the electronic device side can be implemented.

[0148] The processor is used to execute the vulnerability scanning program stored in the memory to implement the following steps of the vulnerability scanning method performed on the electronic device side:

[0149] Reading a scan object from the main message queue through a first main coroutine, wherein the first main coroutine is any one of the N main coroutines;

[0150] Generate a scan request for instructing to perform a vulnerability scan on the scan object read, and write the scan request into a sub-message queue corresponding to a first scan target, wherein the first scan target is the scan target to which the scan object read belongs, and different scan targets correspond to different sub-message queues;

[0151] The scanning request in the sub-message queue corresponding to the second scanning target is executed by the first sub-coroutine according to the set speed limit method, the first sub-coroutine is any one of the M sub-coroutines included in the first main coroutine, and the second scanning target is the scanning target bound to the first main coroutine.

[0152] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0153] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0154] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A vulnerability scanning method, characterized in that: Applied to any scanning node in a distributed vulnerability scanning system, the distributed vulnerability scanning system includes a plurality of the scanning nodes, a task scheduling center, and a main message queue, the scanning node is configured with N main coroutines, the main coroutine includes M sub-coroutines, the task scheduling center sends a plurality of scanning tasks to the main message queue, the scanning task includes at least one scanning target, the scanning target includes at least one scanning object, the method includes: Reading a scan object from the main message queue through a first main coroutine, wherein the first main coroutine is any one of the N main coroutines; Generate a scan request for instructing to perform a vulnerability scan on the scan object read, and write the scan request into a sub-message queue corresponding to a first scan target, wherein the first scan target is the scan target to which the scan object read belongs, and different scan targets correspond to different sub-message queues; The scanning request in the sub-message queue corresponding to the second scanning target is executed by the first sub-coroutine according to the set speed limit method, the first sub-coroutine is any one of the M sub-coroutines included in the first main coroutine, and the second scanning target is the scanning target bound to the first main coroutine.

2. The method according to claim 1, characterized in that The executing, by the first sub-coroutine according to the set speed limit method, the scan request in the sub-message queue corresponding to the second scan target includes: Obtain a token from the token bucket corresponding to the second scanning target through the first sub-routine; When the token is obtained, the scan request is read from the sub-message queue corresponding to the second scan target through the first sub-routine, and the scan request is executed.

3. The method according to claim 2, characterized in that The method further comprises: If the token is not obtained, a preset sleep function is called to control the first main coroutine to sleep and start another main coroutine.

4. The method according to claim 1, characterized in that After reading the scan object from the main message queue by the first main coroutine, the method further includes: Parsing the scanned object to obtain an identification code of the scanned target to which the scanned object belongs; When it is determined that the scan target identified by the identification code obtained through analysis is not bound to other main coroutines, a binding relationship is established between the first main coroutine and the scan target identified by the identification code.

5. The method according to claim 2, characterized in that: The filling rate of the token bucket is set by the task scheduling center in the following way: Determining the number of surviving scanning nodes in the distributed vulnerability scanning system; Determine a maximum value of the scanning rates corresponding to each of the plurality of scanning tasks; The maximum value is divided by the number to obtain the filling rate of the token bucket.

6. The method according to claim 2, characterized in that The method further comprises: When it is monitored that the number of tokens in the token bucket corresponding to the second scanning target meets the first set condition, the number of the sub-coroutines included in the first main coroutine is increased.

7. The method according to claim 6, characterized in that The method further comprises: When the token is acquired, determining the number of remaining tokens in the token bucket corresponding to the second scanning target; If the number of the remaining tokens meets the second setting condition, the preset count value is increased by 1; if the number of the remaining tokens does not meet the second setting condition, the count value is set to 0; When it is determined that the count value reaches the set threshold, it is determined that the number of tokens in the token bucket corresponding to the second scanning target meets a first set condition.

8. The method according to claim 6, characterized in that The increasing the number of the sub-coroutines included in the first main coroutine includes: Increasing the number of the sub-coroutines included in the first main coroutine according to a preset step value; Determine whether the number of the increased sub-coroutines exceeds a set number threshold; If yes, setting the number of the sub-coroutines included in the first main coroutine to the set number threshold; If not, keep the current number of the sub-coroutines unchanged.

9. A distributed vulnerability scanning system, characterized in that: The distributed vulnerability scanning system includes multiple scanning nodes, a task scheduling center, and a main message queue; N main coroutines are configured on the scanning node, and the main coroutine includes M sub-coroutines; The task scheduling center sends a plurality of scanning tasks to the main message queue, wherein the scanning tasks include at least one scanning target, and the scanning target includes at least one scanning object; The scanning node reads the scanning object from the main message queue through a first main coroutine, wherein the first main coroutine is any one of the N main coroutines; Generate a scan request for instructing to perform a vulnerability scan on the scan object read, and write the scan request into a sub-message queue corresponding to a first scan target, wherein the first scan target is the scan target to which the scan object read belongs, and different scan targets correspond to different sub-message queues; The scanning request in the sub-message queue corresponding to the second scanning target is speed-limited through the first sub-coroutine, where the first sub-coroutine is any one of the M sub-coroutines included in the first main coroutine, and the second scanning target is the scanning target bound to the first main coroutine.

10. A vulnerability scanning device, characterized in that: Applied to any scanning node in a distributed vulnerability scanning system, the distributed vulnerability scanning system includes a plurality of the scanning nodes, a task scheduling center, and a main message queue, the scanning node is configured with N main coroutines, the main coroutine includes M sub-coroutines, the task scheduling center sends a plurality of scanning tasks to the main message queue, the scanning task includes at least one scanning target, the scanning target includes at least one scanning object, the device includes: A reading module, configured to read a scan object from the main message queue through a first main coroutine, wherein the first main coroutine is any one of the N main coroutines; A request generation module, used to generate a scan request for instructing to perform a vulnerability scan on the scan object read, and write the scan request into a sub-message queue corresponding to a first scan target, wherein the first scan target is the scan target to which the scan object read belongs, and different scan targets correspond to different sub-message queues; A speed limiting module is used to execute a scan request in a sub-message queue corresponding to a second scan target through a first sub-coroutine according to a set speed limiting method, wherein the first sub-coroutine is any one of the M sub-coroutines included in the first main coroutine, and the second scan target is a scan target bound to the first main coroutine.

11. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a vulnerability scanning program stored in the memory to implement the vulnerability scanning method according to any one of claims 1 to 8.

12. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the vulnerability scanning method according to any one of claims 1 to 8.

Images