Task processing method, device, computer equipment and storage medium

Abstract

The present application relates to a task processing method, device, computer equipment and storage medium. The method relates to cloud computing and cloud security technology, and the method comprises: receiving vulnerability scanning tasks corresponding to a website to be scanned; storing the vulnerability scanning tasks in a first message queue in sequence; extracting the vulnerability scanning tasks from the first message queue; obtaining subtasks obtained by task decomposition of the extracted vulnerability scanning tasks; and in response to a task request from a distributed detection server, pushing the subtasks to the distributed detection server, so that the distributed detection server sends a vulnerability detection request to the website to be scanned according to the subtasks. The use of this method can improve the efficiency of vulnerability scanning.

Description

Technical Field

[0001] The present application relates to the field of cloud security technology, and in particular to a task processing method, system, device, computer equipment and storage medium. Background Art

[0002] The vulnerability scanning system is an automated vulnerability scanning tool, a general vulnerability scanning system that sends a network request with customized rules to the scanned network site, and determines whether the scanned network site has vulnerabilities based on the response content of the scanned network site.

[0003] With the rapid development of Internet technology, the number of network sites is increasing. The concurrent scanning tasks that vulnerability scanning systems have to handle are very large. This makes the above-mentioned general vulnerability scanning systems face the problems of long scanning time and low scanning efficiency. How to quickly complete vulnerability scanning is an urgent problem to be solved. Summary of the invention

[0004] Based on this, it is necessary to provide a task processing method, system, device, computer equipment and storage medium that can improve the efficiency of vulnerability scanning in response to the above technical problems.

[0005] A task processing method, executed by a distributed scheduling server, comprising:

[0006] Receive vulnerability scanning tasks corresponding to the website to be scanned;

[0007] Storing the vulnerability scanning tasks in the first message queue in sequence;

[0008] Extracting the vulnerability scanning task from the first message queue;

[0009] Obtain subtasks obtained by decomposing the extracted vulnerability scanning task;

[0010] In response to the task request of the distributed detection server, the subtask is pushed to the distributed detection server, so that the distributed detection server sends a vulnerability detection request to the website to be scanned according to the subtask.

[0011] In one embodiment, storing the vulnerability scanning tasks in the first message queue in sequence includes: parsing the vulnerability scanning tasks to obtain the URL of the website to be scanned and the scanning configuration information; storing the URL of the website to be scanned and the scanning configuration information in the first message queue accordingly.

[0012] In one embodiment, extracting the vulnerability scanning task from the first message queue includes: determining the total number of vulnerability scanning tasks currently being processed; when the total number is less than a preset processing upper limit, extracting the vulnerability scanning tasks from the first message queue in sequence at preset time intervals; when the total number is greater than or equal to the preset processing upper limit, pausing the extraction of the vulnerability scanning task from the first message queue.

[0013] In one embodiment, obtaining the subtasks obtained by decomposing the extracted vulnerability scanning task includes: sending the vulnerability scanning task extracted from the first message queue to a decomposition server; and receiving the subtasks returned by the decomposition server after decomposing the vulnerability scanning task.

[0014] In one embodiment, the decomposition server is a distributed decomposition server, and the vulnerability scanning task extracted from the first message queue is sent to the decomposition server, and the sub-tasks returned by the decomposition server after the vulnerability scanning task is decomposed include: pushing the vulnerability scan extracted from the first message queue to a task channel; after obtaining the vulnerability scanning task from the task channel, generating a decomposition request for the vulnerability scanning task; sending the decomposition request to the distributed decomposition server; and receiving the sub-tasks returned by the distributed decomposition server after the vulnerability scanning task is decomposed in response to the decomposition request.

[0015] In one embodiment, the steps of decomposing the vulnerability scanning task by the decomposition server include: traversing a plug-in list, the plug-in list including plug-ins for detecting different types of vulnerabilities; determining a designated plug-in for detecting the vulnerability scanning task from the plug-in list according to the scanning configuration information of the vulnerability scanning task; traversing a test sample list corresponding to the designated plug-in; determining a designated test sample for detecting the vulnerability scanning task from the test sample list according to the scanning configuration information of the vulnerability scanning task; and generating a subtask regarding the vulnerability scanning task according to the designated test sample.

[0016] In one embodiment, after obtaining the subtasks obtained by decomposing the extracted vulnerability scanning task, the method further includes: storing the subtasks in a second message queue in sequence; pushing the subtasks to the distributed detection server in response to the task request of the distributed detection server, including: receiving the task request of the distributed detection server; when it is queried that the subtasks exist in the second message queue, pushing the subtasks to the distributed detection server in response to the task request.

[0017] In one embodiment, the above method also includes: receiving a token request from the distributed detection server; when it is queried that a detection token for the website to be scanned currently exists, in response to the token request, returning the detection token of the website to be scanned to the distributed detection server, and the detection token is used to control the distributed detection server to send a vulnerability detection request to the website to be scanned after obtaining the detection token.

[0018] In one embodiment, the method further includes: acquiring a scanning speed of the website to be scanned from scanning configuration information of the vulnerability scanning task; and generating a detection token of the website to be scanned according to the scanning speed.

[0019] In one embodiment, the above method also includes: receiving, through the distributed detection server, a response result returned by the website to be scanned in response to the vulnerability detection request; matching the response result with the vulnerability detection rule corresponding to the subtask to obtain the vulnerability detection result of the website to be scanned.

[0020] A task processing system includes a distributed scheduling server and a distributed detection server, wherein:

[0021] The distributed scheduling server is used to receive vulnerability scanning tasks corresponding to the websites to be scanned, store the vulnerability scanning tasks in the first message queue in sequence, extract the vulnerability scanning tasks from the first message queue, and obtain subtasks obtained by decomposing the extracted vulnerability scanning tasks;

[0022] The distributed scheduling server is further used to push the subtask to the distributed detection server in response to the task request of the distributed detection server;

[0023] The distributed detection server is used to send a vulnerability detection request to the website to be scanned according to the subtask.

[0024] A task processing device, comprising:

[0025] A receiving module, used for receiving vulnerability scanning tasks corresponding to the website to be scanned;

[0026] A persistence module, used for storing the vulnerability scanning tasks in the first message queue in sequence;

[0027] An extraction module, used for extracting the vulnerability scanning task from the first message queue;

[0028] An acquisition module, used for acquiring subtasks obtained by decomposing the extracted vulnerability scanning task;

[0029] The task distribution module is used to push the subtask to the distributed detection server in response to the task request of the distributed detection server, so that the distributed detection server sends a vulnerability detection request to the website to be scanned according to the subtask.

[0030] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] Receive vulnerability scanning tasks corresponding to the website to be scanned;

[0032] Storing the vulnerability scanning tasks in the first message queue in sequence;

[0033] Extracting the vulnerability scanning task from the first message queue;

[0034] Obtain subtasks obtained by decomposing the extracted vulnerability scanning task;

[0035] In response to the task request of the distributed detection server, the subtask is pushed to the distributed detection server, so that the distributed detection server sends a vulnerability detection request to the website to be scanned according to the subtask.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0037] Receive vulnerability scanning tasks corresponding to the website to be scanned;

[0038] Storing the vulnerability scanning tasks in the first message queue in sequence;

[0039] Extracting the vulnerability scanning task from the first message queue;

[0040] Obtain subtasks obtained by decomposing the extracted vulnerability scanning task;

[0041] In response to the task request of the distributed detection server, the subtask is pushed to the distributed detection server, so that the distributed detection server sends a vulnerability detection request to the website to be scanned according to the subtask.

[0042] A computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned task processing method.

[0043] In the above-mentioned task processing method, system, device, computer equipment and storage medium, the scheduling server is distributed and can be expanded as needed. After the distributed scheduling server receives the vulnerability scanning task corresponding to the website to be scanned, the vulnerability scanning task is stored in the first message queue in turn and persisted, and then each distributed scheduling server extracts the scanning task from the first message queue in turn. Compared with the traditional method of directly processing all scanning tasks, it can not only avoid system crashes caused by too many scanning tasks, but also control the number of scanning tasks currently being processed as a whole; in addition, the detection server that sends detection requests to the website to be scanned is also distributed and can also be expanded as needed. After the distributed scheduling server obtains the task requests of each distributed detection server, it pushes each subtask obtained to different distributed detection servers respectively. The distributed detection server can send vulnerability detection requests to the website to be scanned according to the subtask. In this way, even if the concurrency of the vulnerability scanning task is large, each vulnerability scanning task can be processed quickly and orderly to improve the scanning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 An application environment diagram of a task processing method in an embodiment;

[0045] Figure 2 is a flowchart of a task processing method in one embodiment;

[0046] Figure 3 is a schematic diagram of the architecture of a task processing system in one embodiment;

[0047] Figure 4 is a schematic diagram of the architecture of a task processing system in another embodiment;

[0048] Figure 5 A flowchart of task decomposition in one embodiment;

[0049] Figure 6 is a schematic diagram of the architecture of a task processing system in yet another embodiment;

[0050] Figure 7 is a flowchart of a task processing method in another embodiment;

[0051] Figure 8 A schematic diagram of multiple distributed detection servers preempting detection tokens in one embodiment;

[0052] Fig. 9 A schematic diagram of a process for distributing detection tokens in one embodiment;

[0053] Fig.10 is a timing diagram of a task processing method in one embodiment;

[0054] Fig.11 is a flowchart of a task processing method in a specific embodiment;

[0055] Fig.12 is a structural block diagram of a task processing device in one embodiment;

[0056] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The task processing method provided in this application can be applied to Figure 1 In the application environment shown in the figure, the distributed scheduling server 102 communicates with the distributed detection server 104 through the network. The distributed scheduling server 102 receives the vulnerability scanning tasks corresponding to the website to be scanned, stores the vulnerability scanning tasks in the first message queue 106 in sequence, extracts the vulnerability scanning tasks from the first message queue 106, obtains the subtasks obtained by decomposing the extracted vulnerability scanning tasks, and pushes the subtasks to the distributed detection server 104 in response to the task request of the distributed detection server, so that the distributed detection server 104 sends a vulnerability detection request to the website to be scanned 108 according to the subtasks.

[0059] Among them, the distributed scheduling server 102 and the distributed detection server 104 can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0060] In one embodiment, referring to Figure 1 , the distributed scheduling server 102 can receive the vulnerability scanning task sent by the terminal 110, and the terminal 110 communicates with the distributed scheduling server 102 through the network. A vulnerability scanning client can be run on the terminal 110, and the vulnerability scanning service of the vulnerability scanning client can be provided by the vulnerability scanning server 100, and the vulnerability scanning server 100 can include the distributed scheduling server 102 and the distributed detection server 104. The user can submit the vulnerability scanning task through the vulnerability scanning client, and the vulnerability scanning server 100 processes the vulnerability scanning task submitted by the user.

[0061] Among them, the terminal 110 can be but is not limited to various personal computers, smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, etc., but is not limited to these.

[0062] The task processing method provided in the embodiment of the present application may be executed by the task processing device provided in the embodiment of the present application, or a computer device integrated with the task processing device, wherein the task processing device may be implemented in hardware or software. The computer device may be the above-mentioned distributed scheduling server 102.

[0063] The task processing method provided in the embodiment of the present application relates to cloud computing technology. Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0064] For example, the task processing method provided in the embodiment of the present application distributes the vulnerability scanning task on a large number of distributed scheduling servers. After the distributed scheduling servers store the vulnerability scanning tasks in the first message queue in sequence, each distributed scheduling server extracts the vulnerability scanning task from the first message queue in sequence. The subtasks obtained by decomposing the task can also be distributed on a large number of distributed detection servers. Each distributed detection server sends a vulnerability detection request to the website to be scanned according to the subtask. When the concurrency of the vulnerability scanning task is large, each vulnerability scanning task can be processed quickly and orderly, thereby improving the scanning efficiency.

[0065] The task processing method provided in the embodiment of the present application also involves cloud security technology. Cloud security refers to the general term for security software, hardware, users, organizations, and security cloud platforms based on cloud computing business model applications. Cloud security integrates emerging technologies and concepts such as parallel processing, grid computing, and unknown virus behavior judgment. Through a large number of meshed clients, it monitors the abnormal behavior of software in the network, obtains the latest information of Trojans and malicious programs on the Internet, and sends it to the server for automatic analysis and processing, and then distributes virus and Trojan solutions to each client.

[0066] The main research directions of cloud security include: 1. Cloud computing security, which mainly studies how to ensure the security of the cloud itself and various applications on the cloud, including cloud computer system security, secure storage and isolation of user data, user access authentication, information transmission security, network attack protection, compliance auditing, etc.; 2. Cloudification of security infrastructure, which mainly studies how to use cloud computing to build and integrate security infrastructure resources and optimize security protection mechanisms, including building a large-scale security event, information collection and processing platform through cloud computing technology, realizing the collection and correlation analysis of massive information, and improving the control ability of security events and risk control capabilities of the entire network; 3. Cloud security services, which mainly studies various security services provided to users based on cloud computing platforms, such as antivirus services.

[0067] For example, the task processing method provided in the embodiment of the present application can also send a vulnerability detection request to the website to be scanned according to the decomposed sub-tasks through a distributed detection server serving as a cloud server, and after receiving the response result returned by the website to be scanned, match the response result with the vulnerability detection rule corresponding to the sub-task to obtain the vulnerability detection result of the website to be scanned.

[0068] In one embodiment, Figure 2 As shown, a task processing method is provided, which is applied to Figure 1 Taking the distributed scheduling server 102 in the example as an example, the following steps are included:

[0069] Step 202, receiving a vulnerability scanning task corresponding to the website to be scanned.

[0070] Among them, the vulnerability scanning task is a task to be performed for vulnerability scanning, and the website to be scanned for vulnerability can be determined according to the vulnerability scanning task. The vulnerability scanning task includes the URL corresponding to the website to be scanned, which can be the URL of the top layer of the website to be scanned, such as the URL of the homepage of the website to be scanned https: / / www.xxx.com / , and the URL can also be the URL of a specific functional webpage of the website to be scanned, such as the user login page, payment page or user information display page, etc.

[0071] Specifically, the vulnerability scanning task may be submitted by a user through a terminal to a distributed scheduling server. There are multiple distributed scheduling servers, and each distributed scheduling server may receive a vulnerability scanning task corresponding to a website to be scanned.

[0072] Optionally, the vulnerability scanning task received by the distributed scheduling server may further include scanning configuration information, and the distributed scheduling server may process the vulnerability scanning task according to the scanning configuration information. The scanning configuration information may include at least one of scanning intensity, scanning speed, scanning plug-in type, and scanning test sample identification.

[0073] Scanning intensity refers to the degree of vulnerability scanning of the website to be scanned. Scanning intensity can be divided into deep scanning and fast scanning according to different scanning degrees. Obviously, the scanning time consumed by deep scanning may be longer than the scanning time consumed by fast scanning. Deep scanning refers to a relatively in-depth scanning of the website to be scanned. When the distributed scheduling server obtains the vulnerability scanning task, and the scanning configuration information of the vulnerability scanning task indicates that the corresponding website to be scanned needs to be deeply scanned, the distributed scheduling server needs to perform a deep scan on the URL of the vulnerability scanning task. For example, when the vulnerability scanning task includes the website of the homepage of the website to be scanned, the distributed scheduling server can further crawl the web pages associated with the URL and scan them in sequence to perform a comprehensive scan of the web pages related to the website to be scanned; for another example, use multiple types of scanning plug-ins to scan the website to be scanned to perform a comprehensive scan of the vulnerabilities that may exist in the website to be scanned; for another example, for some types of scanning plug-ins, use different test samples to scan the website to be scanned to perform multiple and comprehensive scans of some types of vulnerabilities that may exist in the website to be scanned.

[0074] Correspondingly, a quick scan refers to a relatively simple scan of the website to be scanned. When the distributed scheduling server obtains the vulnerability scanning task, and the scanning configuration information of the vulnerability scanning task indicates that the corresponding website to be scanned needs to be quickly scanned, the distributed scheduling server needs to quickly scan the URL of the vulnerability scanning task. For example, the distributed scheduling server can only scan the URLs included in the vulnerability scanning task; for another example, only some scanning plug-ins with prone vulnerabilities can be used to scan the website to be scanned; for another example, for some easy-to-detect vulnerabilities, only one or fewer test samples can be used for scanning.

[0075] Scanning speed refers to the number of probe requests sent to the website to be scanned per unit time, for example, 50 probe requests per second. During vulnerability scanning, it is necessary to send a probe request with a customized rule (payload) to the website to be scanned, and determine whether the website to be scanned has a vulnerability based on the response content of the scanned website. Therefore, the website to be scanned will continuously receive probe requests. Obviously, the more probe requests sent per unit time, the faster the scanning speed will be. If too many probe requests are sent per unit time, it will cause a large load on the website to be scanned, and in severe cases, it may cause the website to be scanned to crash. Therefore, the vulnerability scanning task can also include the scanning speed, so that the stability of the website to be scanned can be guaranteed while achieving the fastest scanning possible.

[0076] The scan plug-in type is the type of plug-in used to detect vulnerabilities. Different types of scan plug-ins are used to detect different vulnerabilities. Therefore, the vulnerability scanning task can also include the scan plug-in type, so that the corresponding scan plug-in can be used to detect the corresponding vulnerability according to actual needs. The scan test sample identifier is used to uniquely identify different scan test samples. A scan plug-in corresponds to a certain vulnerability. For a certain type of scan plug-in, it can correspond to multiple scan test samples. These multiple scan test samples use different detection methods to detect the vulnerability of this type.

[0077] In one embodiment, when a user submits a vulnerability scanning task to a distributed scheduling server through a terminal, the user can specify at least one of the scanning intensity, scanning speed, scanning plug-in type used for scanning, and test sample identification of the website to be scanned. In this way, the distributed scheduling server can implement customized vulnerability scanning of the website to be scanned according to the user's needs.

[0078] Optionally, if the user does not specify the scanning configuration information, the distributed scheduling server can process the vulnerability scanning task according to the default scanning configuration information after receiving the vulnerability scanning task. For example, the default scanning configuration information is that the scanning intensity is deep scanning, the scanning speed is 50 detection requests per second, all scanning plug-ins are used for scanning, and all test samples are used to send detection requests.

[0079] Step 204, storing the vulnerability scanning tasks in the first message queue in sequence.

[0080] Among them, the first message queue can be a queue implemented based on Redis, RabbitMQ, etc., and the distributed scheduling server stores the received vulnerability scanning tasks in the first message queue. Specifically, after the distributed scheduling server receives the vulnerability scanning tasks corresponding to the website to be scanned, it stores the vulnerability scanning tasks in the first message queue in turn and persists them, and then each distributed scheduling server extracts the scanning tasks from the first message queue in turn. Compared with the traditional method of directly processing all scanning tasks, the received vulnerability scanning tasks are first stored in the message queue, and the message queue is used as a buffer for a large number of concurrent vulnerability scanning tasks. This can not only avoid the crash of the scanning task processing system due to too many scanning tasks, but also control the number of scanning tasks currently processed by the scanning task processing system as a whole, thereby improving scanning efficiency.

[0081] In one embodiment, the vulnerability scanning tasks are sequentially stored in the first message queue, including: parsing the vulnerability scanning tasks to obtain the URL and scanning configuration information of the website to be scanned; and storing the URL and scanning configuration information of the website to be scanned in the first message queue accordingly.

[0082] As mentioned earlier, the vulnerability scanning task includes the URL corresponding to the website to be scanned, and may also include scanning configuration information. Therefore, after the distributed scheduling server receives the vulnerability scanning task, it parses the vulnerability scanning task, extracts the website to be scanned and the corresponding scanning configuration information, and stores the website to be scanned and the scanning configuration information in the first message queue accordingly.

[0083] In one embodiment, the distributed scheduling servers may share a first message queue for storing vulnerability scanning tasks, that is, all distributed scheduling servers may store the vulnerability scanning tasks they receive in the same first message queue. The distributed scheduling servers may also be divided into groups according to actual needs, and the vulnerability scanning tasks received by the distributed scheduling servers in the same group may be stored in the same message queue, and the vulnerability scanning tasks received by the distributed scheduling servers in different groups may be stored in different message queues, so as to avoid excessive clustering of tasks in the message queue.

[0084] In addition, the distributed scheduling server is distributed and can dynamically expand or shrink according to actual needs, that is, increase or decrease the number of distributed scheduling servers. After the distributed scheduling server persists the received vulnerability scanning tasks in the first message queue, even if a single point failure occurs, other normally operating distributed scheduling servers can still extract vulnerability scanning tasks from the first message queue to ensure the normal operation of the scanning task processing system.

[0085] Step 206: extract the vulnerability scanning task from the first message queue.

[0086] Specifically, the distributed scheduling server may extract the vulnerability scanning task from the first message queue.

[0087] In one embodiment, the distributed scheduling server can set up a timed polling component, through which vulnerability scanning tasks are extracted from the first message queue in sequence in a timed polling manner, thereby controlling the number and efficiency of tasks currently processed by the entire scanning task processing system, and ensuring the stability of the scanning task processing system and the website to be scanned.

[0088] In one embodiment, extracting vulnerability scanning tasks from a first message queue includes: determining the total number of vulnerability scanning tasks currently being processed; when the total number is less than a preset processing upper limit, extracting vulnerability scanning tasks from the first message queue in sequence at preset time intervals; when the total number is greater than or equal to the preset processing upper limit, pausing the extraction of vulnerability scanning tasks from the first message queue.

[0089] In this embodiment, the polling component in the distributed scheduling server can control whether to extract vulnerability scanning tasks from the message queue based on the total number of vulnerability scanning tasks currently being processed, thereby avoiding the crash of the scanning task processing system due to too many scanning tasks, and can also control the number of scanning tasks currently being processed as a whole.

[0090] Step 208: Obtain subtasks obtained by decomposing the extracted vulnerability scanning task.

[0091] In order to achieve a comprehensive vulnerability scan of the website to be scanned, it is necessary to decompose the vulnerability scanning task. Task decomposition can divide the vulnerability scanning task into multiple subtasks. Task decomposition is actually to parse the URL in the vulnerability scanning task to obtain the parameters in the URL. For each parameter, the corresponding parameter is replaced with the test sample corresponding to the scanning plug-in to obtain the subtask corresponding to the vulnerability scanning task. The parameters in the URL include but are not limited to HOST parameters, User Agent parameters, Referer parameters, GET parameters, POST parameters, Cookie parameters and Path parameters.

[0092] In one embodiment, the distributed scheduling server can also decompose the extracted vulnerability scanning tasks to obtain subtasks. In one embodiment, the distributed scheduling server is mainly responsible for the scheduling of vulnerability scanning tasks. The distributed scheduling server can send the vulnerability scanning tasks to the decomposition server for task decomposition, and receive the subtasks returned by the decomposition server. Through this distributed scheduling, the stability of the scanning task processing system and the availability of the website to be scanned are improved when the concurrent vulnerability scanning tasks are large.

[0093] In one embodiment, obtaining subtasks obtained by decomposing the extracted vulnerability scanning task includes: sending the vulnerability scanning task extracted from the first message queue to a decomposition server; and receiving subtasks returned by the decomposition server after decomposing the vulnerability scanning task.

[0094] like Figure 3 FIG. 1 is a schematic diagram of the architecture of a task processing system in an embodiment. Figure 3 The distributed scheduling server sends the vulnerability scanning task extracted from the first message queue to the decomposition server, requests the decomposition server to decompose the vulnerability scanning task, and then the decomposition server returns the subtask to the distributed scheduling server.

[0095] In one embodiment, the decomposition server is a distributed decomposition server, which sends the vulnerability scanning task extracted from the first message queue to the decomposition server, and receives the sub-tasks returned by the decomposition server after decomposing the vulnerability scanning task, including: pushing the vulnerability scan extracted from the first message queue to the task channel; after obtaining the vulnerability scanning task from the task channel, generating a decomposition request for the vulnerability scanning task; sending the decomposition request to the distributed decomposition server; receiving the sub-tasks returned by the distributed decomposition server after decomposing the vulnerability scanning task in response to the decomposition request.

[0096] Specifically, the polling component of the distributed scheduling server can push the vulnerability scan extracted from the first message queue to the task channel. Once there is a vulnerability scanning task in the task channel, it will be immediately taken out and sent to the distributed decomposition server, and the decomposition server will be requested to decompose the complete vulnerability scanning task into multiple subtasks.

[0097] In this embodiment, the scanning task processing system further includes distributed decomposition servers. By having these decomposition servers respond to the decomposition requests of the distributed scheduling server in parallel, the overall vulnerability scanning efficiency can be greatly improved.

[0098] In one embodiment, the distributed scheduling server may include distributed decomposition nodes. After each decomposition node takes out a vulnerability scanning task from a task channel, it generates a decomposition request for the vulnerability scanning task, and then sends the decomposition request to the distributed decomposition server, which decomposes the vulnerability scanning task through the distributed decomposition server.

[0099] like Figure 4 FIG. 1 is a schematic diagram of the architecture of a task processing system in an embodiment. Figure 4 The user submits the vulnerability scanning task to the distributed scheduling server through the terminal. The distributed scheduling server stores the received vulnerability scanning tasks in the first message queue in sequence, extracts the vulnerability scanning tasks from the first message queue in sequence through the polling component, pushes them to the task channel, and then takes out the vulnerability scanning task from the task channel through the distributed decomposition node, requests the distributed decomposition server to decompose the task, and receives the subtasks returned by the distributed decomposition server.

[0100] In one embodiment, the steps of decomposing a vulnerability scanning task by a decomposition server include: traversing a plug-in list, the plug-in list includes plug-ins for detecting different types of vulnerabilities; determining a specified plug-in for detecting the vulnerability scanning task from the plug-in list according to the scanning configuration information of the vulnerability scanning task; traversing a test sample list corresponding to the specified plug-in; determining a specified test sample for detecting the vulnerability scanning task from the test sample list according to the scanning configuration information of the vulnerability scanning task; and generating a subtask regarding the vulnerability scanning task according to the specified test sample.

[0101] Among them, the plug-in refers to a vulnerability scanning plug-in, which is a program written according to certain specifications for a specific type of network vulnerability. The test sample corresponds to the plug-in. The plug-in for a certain vulnerability can correspond to a variety of different test samples, and the different test samples correspond to different detection methods for the vulnerability. For example, for vulnerability A, we can use test samples 00101, 00102, and 00103 corresponding to plug-in 001 to send different vulnerability detection requests to the website to be scanned. Specifically, when the scanning configuration information of the vulnerability scanning task specifies the plug-in type, the decomposition server determines the specified plug-in for detecting the vulnerability scanning task from the plug-in list during the process of traversing the plug-in list. When the scanning configuration information of the vulnerability scanning task also specifies the test samples used by certain plug-ins, the decomposition server determines the specified test sample for detecting the vulnerability scanning task from the test sample list during the process of traversing the test sample list of these plug-ins, and uses the specified test sample to replace the parameters in the URL of the website to be scanned to obtain the subtask.

[0102] In one embodiment, the decomposition server may traverse each plug-in in sequence according to the serial number of each plug-in in the plug-in list. Similarly, it may also traverse each test sample in sequence according to the serial number of each test sample in the test sample list corresponding to each plug-in, and generate subtasks using the traversed test samples.

[0103] like Figure 5 FIG. 1 is a flowchart of task decomposition in an embodiment. Figure 5 After the vulnerability scanning task decomposition begins, loop through the plug-in list to determine whether the current plug-in is greater than the plug-in serial number of the last decomposition; if it is less, traverse the next plug-in in the plug-in list; if it is greater, continue to determine whether the current plug-in belongs to the plug-in type specified by the scanning configuration information of the vulnerability scanning task; if not, traverse the next plug-in in the plug-in list; if so, traverse the test sample list corresponding to the plug-in; determine whether the current test sample serial number is greater than the test sample serial number of the last decomposition; if it is less, traverse the next test sample in the test sample list; if it is greater, continue to determine whether the current test sample belongs to the test sample specified by the scanning configuration information of the vulnerability scanning task; if so, further generate a subtask based on the test sample.

[0104] Specifically, when generating a subtask, when the parameter to be tested by the test sample is HOST, and the vulnerability scanning task specifies that HOST is to be scanned, the test sample is used to replace the HOST parameter in the URL of the website to be scanned, and the subtask is obtained. When the parameter to be tested by the test sample is User Agent, and the vulnerability scanning task specifies that UserAgent (UA for short) is to be scanned, the test sample is used to replace the User Agent parameter in the URL of the website to be scanned, and the subtask is obtained. When the parameter to be tested by the test sample is GET, and the vulnerability scanning task specifies that GET is to be scanned, the test sample is used to replace the GET parameter in the URL of the website to be scanned, and the subtask is obtained. When the parameter to be tested by the test sample is POST, and the vulnerability scanning task specifies that POST is to be scanned, the test sample is used to replace the POST parameter in the URL of the website to be scanned, and the subtask is obtained. When the parameter to be tested by the test sample is Cookie, and the vulnerability scanning task specifies that Cookie is to be scanned, the test sample is used to replace the Cookie parameter in the URL of the website to be scanned, and the subtask is obtained. When the parameter to be tested by the test sample is PATH, and the vulnerability scanning task specifies to scan PATH, the test sample is used to replace the PATH parameter in the URL of the website to be scanned to obtain a subtask. Finally, in order to avoid unlimited decomposition of the parameters of some malicious websites to be scanned, resulting in too many subtasks and occupying the resources of the scanning task processing system, it is also necessary to determine whether the subtasks corresponding to the website to be scanned are greater than the preset number. If so, the decomposition of the URL of the website to be scanned is terminated. If not, the plug-in list is continued to be looped through and the URL of the website to be scanned is continued to be decomposed.

[0105] In one embodiment, after obtaining the subtasks obtained by decomposing the extracted vulnerability scanning task, the method further includes: storing the subtasks in sequence in the second message queue.

[0106] In this embodiment, the generated subtasks are stored in the second message queue, and the message queue is used as a buffer for a large number of concurrent subtasks. This can prevent the scanning task processing system from crashing due to too many subtasks. In addition, even if a single point failure occurs in the distributed detection server, other normally operating distributed detection servers can still obtain subtasks from the second message queue to ensure the normal operation of the scanning task processing system.

[0107] Step 210, in response to the task request of the distributed detection server, push the subtask to the distributed detection server, so that the distributed detection server sends a vulnerability detection request to the website to be scanned according to the subtask.

[0108] The distributed detection server is set up in a distributed manner. The distributed detection server is a server that directly interacts with the website to be scanned. The distributed detection server can send a task request to the distributed scheduling server. The distributed scheduling server responds to the task request and pushes the subtask to the distributed detection server. After receiving the subtask, the distributed detection server generates a vulnerability detection request according to the subtask and sends the vulnerability detection request to the website to be scanned, for example, sending the http request generated according to the subtask to the website to be scanned. The website to be scanned responds to the vulnerability detection request, and the returned response content can be used as a basis for determining whether the website to be scanned has the vulnerability tested by the subtask.

[0109] In one embodiment, after obtaining the subtasks obtained by decomposing the extracted vulnerability scanning task, the above method also includes: storing the subtasks in a second message queue in sequence; pushing the subtasks to the distributed detection server in response to a task request from the distributed detection server, including: receiving the task request from the distributed detection server; when it is queried that the subtask exists in the second message queue, pushing the subtask to the distributed detection server in response to the task request.

[0110] In this embodiment, the distributed detection server actively obtains subtasks from the distributed scheduling server when there are sufficient resources, rather than passively receiving a large number of subtasks assigned by the distributed scheduling server, which can ensure the effective operation of the distributed detection server.

[0111] like Figure 6 FIG. 1 is a schematic diagram of the architecture of a task processing system in an embodiment. Figure 6 , the user submits the vulnerability scanning task to the distributed scheduling server through the terminal. The distributed scheduling server stores the received vulnerability scanning tasks in the first message queue in sequence. After extracting the vulnerability scanning tasks from the first message queue in sequence through the polling component, it is pushed to the task channel. Then the vulnerability scanning task is taken out from the task channel through the distributed decomposition node, and the distributed decomposition server is requested to decompose the task. The distributed decomposition node receives the subtask returned by the distributed decomposition server and stores it in the second message queue. The distributed detection server requests the subtask from the second message queue and sends a vulnerability detection request to the website to be scanned according to the subtask.

[0112] In one embodiment, the above method also includes: receiving a response result returned by the website to be scanned in response to a vulnerability detection request through a distributed detection server; matching the response result with a vulnerability detection rule corresponding to the subtask to obtain a vulnerability detection result of the website to be scanned.

[0113] Since website vulnerabilities mainly refer to security vulnerabilities existing in web program codes developed in various languages, after receiving the response result returned by the website to be scanned, the response result can be compared with the response result expected to be returned when the vulnerability tested by the subtask exists. If they are the same, it means that the website to be scanned has the vulnerability tested by the subtask.

[0114] In one embodiment, the above method also includes: persistently storing the vulnerability scanning task, the corresponding subtask and the vulnerability detection results of the subtask, so that even if a single point failure occurs during the processing of the vulnerability scanning task or subtask and the distributed scheduling server needs to be restarted, the vulnerability scanning task will not be lost, especially when the concurrency is large, and a large number of scanning tasks will not be lost.

[0115] In the above task processing method, the scheduling server is distributed and can be expanded as needed. After the distributed scheduling server receives the vulnerability scanning task corresponding to the website to be scanned, the vulnerability scanning task is persisted and stored in the first message queue in turn, and then each distributed scheduling server extracts the scanning task from the first message queue in turn. Compared with the traditional method of directly processing all scanning tasks, it can not only avoid system crashes caused by too many scanning tasks, but also control the number of scanning tasks currently being processed as a whole; in addition, the detection server that sends detection requests to the website to be scanned is also distributed and can be expanded as needed. After the distributed scheduling server obtains the task requests of each distributed detection server, it pushes each subtask obtained to different distributed detection servers respectively. The distributed detection server can send vulnerability detection requests to the website to be scanned according to the subtask. In this way, even if the concurrency of the vulnerability scanning task is large, each vulnerability scanning task can be processed quickly and orderly to improve the scanning efficiency.

[0116] like Figure 7 FIG. 1 is a flowchart of a task processing method in an embodiment. Figure 7 , the above method further comprises:

[0117] Step 702, receiving a vulnerability scanning task corresponding to the website to be scanned.

[0118] Step 704, storing the vulnerability scanning tasks in the first message queue in sequence.

[0119] Step 706: extract the vulnerability scanning task from the first message queue.

[0120] Step 708: Obtain subtasks obtained by decomposing the extracted vulnerability scanning task.

[0121] Step 710: Generate a detection token for the website to be scanned according to the scanning speed corresponding to the vulnerability scanning task.

[0122] Among them, the scanning speed is used to achieve distributed speed limiting. Distributed speed limiting means that multiple distributed detection servers simultaneously process subtasks of the same website to be scanned. Speed ​​limiting is needed to prevent these multiple distributed detection servers from making access requests to the website to be scanned at the same time, resulting in a high concurrency of access to the website to be scanned at this time. In order to prevent the website from being unable to operate normally due to the excessive concurrency of access to the website to be scanned, the scanning rate and the normal operation of the website are dynamically guaranteed by distributing detection tokens to the distributed detection servers. The distributed scheduling server generates detection tokens according to the scanning speed specified in the vulnerability scanning task. Therefore, the detection tokens are generated dynamically. Before the detection server needs to send a vulnerability detection request, it needs to obtain the detection token of the website to be scanned. This avoids the situation where the website to be scanned cannot operate normally due to excessive concurrency.

[0123] The scanning speed may be determined according to the concurrency processing capability of the website to be scanned, may be specified by the user when submitting the vulnerability scanning task, or may be a default setting.

[0124] In one embodiment, the method further includes: acquiring a scanning speed of the website to be scanned from scanning configuration information of the vulnerability scanning task; and generating a detection token of the website to be scanned according to the scanning speed.

[0125] like Figure 8 FIG. 1 is a schematic diagram of a plurality of distributed detection servers preempting detection tokens in one embodiment. Figure 8 , multiple distributed detection servers simultaneously seize the detection token generated for the same website to be scanned. Only when the detection token is seized can a vulnerability detection request be sent to the website to be scanned. If it is not seized, a vulnerability detection request cannot be sent to the website to be scanned.

[0126] In one embodiment, the distributed scheduling server can also dynamically adjust the scanning speed according to the concurrent processing capability of the website to be scanned, so as to realize personalized scanning speed for different websites to be scanned. For example, if the website to be scanned is a relatively large website, its availability is very high, and the website has good concurrent processing capability, then the scanning speed can be set to a higher value, such as 100, otherwise, the corresponding scanning speed is set to a smaller value, such as 50.

[0127] Step 712, in response to the task request of the distributed detection server, the subtask is pushed to the distributed detection server, so that the distributed detection server sends a vulnerability detection request to the website to be scanned according to the subtask after obtaining the detection token.

[0128] In one embodiment, the above method also includes: receiving a token request from a distributed detection server; when a detection token for the website to be scanned is found to exist, in response to the token request, returning the detection token of the website to be scanned to the distributed detection server, and the detection token is used to control the distributed detection server to send a vulnerability detection request to the website to be scanned after obtaining the detection token.

[0129] Specifically, after the distributed scheduling server pushes the subtask to the distributed detection server, the distributed detection server needs to request the detection token of the website to be scanned from the distributed scheduling server before sending a vulnerability detection request based on the subtask. Since the detection token is generated according to the scanning speed, for example, if the scanning speed is 50, there are only 50 detection tokens per unit time. No matter how many distributed detection servers request detection tokens per unit time, only 50 tokens can be taken away in total. In this way, when multiple distributed detection servers scan the website to be scanned, it can ensure that the rate at which the website is scanned is stable within the scanning speed set by the user.

[0130] In one embodiment, when it is found that there is no detection token for the website to be scanned, the distributed detection server needs to wait until the distributed scheduling server generates a new detection token and obtains the detection token before sending a vulnerability detection request to the website to be scanned.

[0131] like Fig. 9 FIG. 1 is a flow chart of distributing detection tokens in one embodiment. Fig. 9 After the distributed scheduling server generates the detection token according to the scanning speed, it determines whether the token bucket is full. If so, the generated detection token is discarded. If not, the generated detection token is stored in the token bucket. When the distributed scheduling server receives the token request for the website to be scanned from the distributed detection server, it determines whether the detection token of the website to be scanned still exists in the current token bucket. If so, the token is taken out from the token bucket and issued. The distributed detection server can send a vulnerability detection request to the website to be scanned. If not, the distributed detection server needs to wait and cannot send a vulnerability detection request temporarily until the detection token is obtained. Then, the vulnerability detection request can be sent to the website to be scanned.

[0132] like Fig.10 FIG. 1 is a timing diagram of a task processing method in an embodiment. Fig.10, submit the vulnerability scanning task to the distributed scheduling server through the application program interface, the distributed scheduling server stores the obtained vulnerability scanning task in the first message queue, takes out the vulnerability scanning task from the first message queue, sends a decomposition request to the decomposition server, the decomposition server responds to the decomposition request, returns the decomposed subtask to the distributed scheduling server, the distributed scheduling server stores the subtask in the second message queue, the distributed detection server sends a task request to the distributed scheduling server, the distributed scheduling server checks whether there is a subtask in the second message queue, and if so, pushes it to the distributed detection server. After the distributed detection server gets the subtask, it requests the distributed scheduling server whether the website to be scanned corresponding to the subtask has enough detection tokens. If so, it sends a vulnerability detection request to the website to be scanned according to the subtask, and obtains the vulnerability detection result according to the response content of the website to be scanned, and returns the vulnerability detection result to the distributed scheduling server. If there is no detection token, it waits in a loop.

[0133] like Fig.11 As shown, a flowchart of a task processing method in a specific embodiment is shown. Fig.11 , the method comprises the following steps:

[0134] Step 1102, receiving a vulnerability scanning task corresponding to the website to be scanned.

[0135] Step 1104, parse the vulnerability scanning task to obtain the URL of the website to be scanned and the scanning configuration information.

[0136] Step 1106: store the URL of the website to be scanned and the scanning configuration information in the first message queue accordingly.

[0137] Step 1108, determining the total number of vulnerability scanning tasks currently being processed.

[0138] Step 1110, when the total number is less than a preset processing upper limit, vulnerability scanning tasks are sequentially extracted from the first message queue at preset time intervals.

[0139] Step 1112: When the total number is greater than or equal to a preset processing upper limit, suspending the extraction of vulnerability scanning tasks from the first message queue.

[0140] Step 1114: Push the vulnerability scan extracted from the first message queue to the task channel.

[0141] Step 1116, after obtaining the vulnerability scanning task from the task channel, a decomposition request for the vulnerability scanning task is generated.

[0142] Step 1118, sending a decomposition request to the distributed decomposition server.

[0143] Step 1120: receiving subtasks returned by the distributed decomposition server after decomposing the vulnerability scanning task in response to the decomposition request.

[0144] Step 1122, storing the subtasks in the second message queue in sequence.

[0145] Step 1124, receiving a task request from a distributed detection server.

[0146] Step 1126: When it is found that the second message queue contains a subtask, the subtask is pushed to the distributed detection server in response to the task request.

[0147] Step 1128, obtaining the scanning speed of the website to be scanned from the scanning configuration information of the vulnerability scanning task.

[0148] Step 1130, generating a detection token for the website to be scanned according to the scanning speed.

[0149] Step 1132, receiving a token request from the distributed detection server.

[0150] Step 1134: when it is found that a detection token of the website to be scanned currently exists, the detection token of the website to be scanned is returned to the distributed detection server in response to the token request.

[0151] Step 1136: After obtaining the detection token, the distributed detection server sends a vulnerability detection request to the website to be scanned.

[0152] Step 1138: Receive, through the distributed detection server, a response result returned by the website to be scanned in response to the vulnerability detection request.

[0153] Step 1140, matching the response result with the vulnerability detection rule corresponding to the subtask to obtain the vulnerability detection result of the website to be scanned.

[0154] In the above-mentioned task processing method, the scheduling server is distributed and can be expanded as needed. After the distributed scheduling server receives the vulnerability scanning task corresponding to the website to be scanned, the vulnerability scanning task is stored in the first message queue in turn and persisted, and then each distributed scheduling server extracts the scanning task from the first message queue in turn. Compared with the traditional method of directly processing all scanning tasks, it can not only avoid system crashes caused by too many scanning tasks, but also control the number of scanning tasks currently being processed as a whole; in addition, the detection server that sends detection requests to the website to be scanned is also distributed and can be expanded as needed. After the distributed scheduling server obtains the task requests of each distributed detection server, it pushes each obtained subtask to different distributed detection servers respectively, and the distributed detection server can send vulnerability detection requests to the website to be scanned according to the subtask. In this way, even if the concurrency of the vulnerability scanning task is large, each vulnerability scanning task can be processed quickly and orderly to improve the scanning efficiency.

[0155] It should be understood that although Fig.11 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Fig.11 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0156] In one embodiment, a task processing system is provided, such as Figure 1 As shown, the task processing system includes a distributed scheduling server and a distributed detection server, wherein:

[0157] The distributed scheduling server is used to receive vulnerability scanning tasks corresponding to the websites to be scanned, store the vulnerability scanning tasks in the first message queue in sequence, extract the vulnerability scanning tasks from the first message queue, and obtain subtasks obtained by decomposing the extracted vulnerability scanning tasks;

[0158] The distributed scheduling server is also used to respond to the task request of the distributed detection server and push the subtask to the distributed detection server;

[0159] The distributed detection server is used to send vulnerability detection requests to the website to be scanned according to the subtask.

[0160] In one embodiment, the distributed scheduling server is also used to parse the vulnerability scanning task, obtain the URL of the website to be scanned and the scanning configuration information; store the URL of the website to be scanned and the scanning configuration information in the first message queue accordingly.

[0161] In one embodiment, the distributed scheduling server is also used to determine the total number of vulnerability scanning tasks currently being processed; when the total number is less than a preset processing upper limit, the vulnerability scanning tasks are extracted from the first message queue in sequence at preset time intervals; when the total number is greater than or equal to the preset processing upper limit, the extraction of vulnerability scanning tasks from the first message queue is suspended.

[0162] In one embodiment, the task processing system also includes a decomposition server, and the distributed scheduling server is also used to send the vulnerability scanning task extracted from the first message queue to the decomposition server; the decomposition server is used to decompose the vulnerability scanning task and return the subtasks obtained by the task decomposition to the distributed scheduling server; the distributed scheduling server is also used to receive the subtasks returned by the decomposition server.

[0163] In one embodiment, the decomposition server is a distributed decomposition server, and the distributed scheduling server is also used to push the vulnerability scan extracted from the first message queue to the task channel; after obtaining the vulnerability scanning task from the task channel, generate a decomposition request for the vulnerability scanning task; send the decomposition request to the distributed decomposition server; receive the subtasks returned by the distributed decomposition server after decomposing the vulnerability scanning task in response to the decomposition request.

[0164] In one embodiment, the decomposition server is further used to traverse a plug-in list, the plug-in list includes plug-ins for detecting different types of vulnerabilities; determine a specified plug-in for detecting the vulnerability scanning task from the plug-in list according to the scanning configuration information of the vulnerability scanning task; traverse a test sample list corresponding to the specified plug-in; determine a specified test sample for detecting the vulnerability scanning task from the test sample list according to the scanning configuration information of the vulnerability scanning task; generate a subtask for the vulnerability scanning task according to the specified test sample

[0165] In one embodiment, the distributed scheduling server is also used to store subtasks in the second message queue in sequence; receive task requests from the distributed detection server; and when it is found that there are subtasks in the second message queue, push the subtasks to the distributed detection server in response to the task request.

[0166] In one embodiment, the distributed scheduling server is also used to receive a token request from a distributed detection server; when a detection token for a website to be scanned is found to exist, the detection token for the website to be scanned is returned to the distributed detection server in response to the token request, and the detection token is used to control the distributed detection server to send a vulnerability detection request to the website to be scanned after obtaining the detection token.

[0167] In one embodiment, the distributed scheduling server is further used to obtain the scanning speed of the website to be scanned from the scanning configuration information of the vulnerability scanning task; and generate a detection token of the website to be scanned according to the scanning speed.

[0168] In one embodiment, the distributed detection server is also used to match the response result returned by the website to be scanned in response to the vulnerability detection request with the vulnerability detection rule corresponding to the subtask to obtain the vulnerability detection result of the website to be scanned.

[0169] In the above-mentioned task processing system, the scheduling server is distributed and can be expanded as needed. After the distributed scheduling server receives the vulnerability scanning task corresponding to the website to be scanned, the vulnerability scanning task is stored in the first message queue in turn and persisted, and then each distributed scheduling server extracts the scanning task from the first message queue in turn. Compared with the traditional method of directly processing all scanning tasks, it can not only avoid system crashes caused by too many scanning tasks, but also control the number of scanning tasks currently being processed as a whole; in addition, the detection server that sends detection requests to the website to be scanned is also distributed and can be expanded as needed. After the distributed scheduling server obtains the task request of each distributed detection server, it pushes each subtask obtained to different distributed detection servers respectively. The distributed detection server can send vulnerability detection requests to the website to be scanned according to the subtask. In this way, even if the concurrency of the vulnerability scanning task is large, each vulnerability scanning task can be processed quickly and orderly to improve the scanning efficiency.

[0170] In one embodiment, Fig.12 As shown, a task processing device 1200 is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: a receiving module 1202, a persistence module 1204, an extraction module 1206, an acquisition module 1208 and a task distribution module 1210, wherein:

[0171] A receiving module 1202 is used to receive a vulnerability scanning task corresponding to a website to be scanned;

[0172] The persistence module 1204 is used to store the vulnerability scanning tasks in the first message queue in sequence;

[0173] An extraction module 1206 is used to extract vulnerability scanning tasks from the first message queue;

[0174] An acquisition module 1208 is used to acquire subtasks obtained by decomposing the extracted vulnerability scanning task;

[0175] The task distribution module 1210 is used to push the subtask to the distributed detection server in response to the task request of the distributed detection server, so that the distributed detection server sends a vulnerability detection request to the website to be scanned according to the subtask.

[0176] In one embodiment, the persistence module 1204 is further used to parse the vulnerability scanning task, obtain the URL of the website to be scanned and the scanning configuration information; and store the URL of the website to be scanned and the scanning configuration information in the first message queue accordingly.

[0177] In one embodiment, the extraction module 1206 is also used to determine the total number of vulnerability scanning tasks currently being processed; when the total number is less than a preset processing upper limit, the vulnerability scanning tasks are extracted from the first message queue in sequence at preset time intervals; when the total number is greater than or equal to the preset processing upper limit, the extraction of vulnerability scanning tasks from the first message queue is suspended.

[0178] In one embodiment, the acquisition module 1208 is further used to send the vulnerability scanning task extracted from the first message queue to the decomposition server; and receive the subtasks returned by the decomposition server after decomposing the vulnerability scanning task.

[0179] In one embodiment, the decomposition server is a distributed decomposition server, and the acquisition module 1208 is also used to push the vulnerability scan extracted from the first message queue to the task channel; after obtaining the vulnerability scanning task from the task channel, generate a decomposition request for the vulnerability scanning task; send the decomposition request to the distributed decomposition server; receive the distributed decomposition server in response to the decomposition request, and return the subtask after decomposing the vulnerability scanning task.

[0180] In one embodiment, the persistence module 1204 is also used to store the subtasks in the second message queue in sequence, and the task distribution module 1210 is also used to receive task requests from the distributed detection server; when it is found that there are subtasks in the second message queue, the subtasks are pushed to the distributed detection server in response to the task request.

[0181] In one embodiment, the above-mentioned device also includes a token request module, which is used to receive a token request from a distributed detection server; when it is queried that a detection token for the website to be scanned currently exists, the detection token of the website to be scanned is returned to the distributed detection server in response to the token request, and the detection token is used to control the distributed detection server to send a vulnerability detection request to the website to be scanned after obtaining the detection token.

[0182] In one embodiment, the above-mentioned device also includes a token generation module, which is used to obtain the scanning speed of the website to be scanned from the scanning configuration information of the vulnerability scanning task; and generate a detection token for the website to be scanned according to the scanning speed.

[0183] In one embodiment, the above-mentioned device also includes a detection result acquisition module, which is used to receive the response result returned by the website to be scanned in response to the vulnerability detection request through the distributed detection server; match the response result with the vulnerability detection rule corresponding to the subtask to obtain the vulnerability detection result of the website to be scanned.

[0184] In the above-mentioned task processing device, the scheduling server is distributed and can be expanded as needed. After the distributed scheduling server receives the vulnerability scanning task corresponding to the website to be scanned, the vulnerability scanning task is stored in the first message queue in turn and persisted, and then each distributed scheduling server extracts the scanning task from the first message queue in turn. Compared with the traditional method of directly processing all scanning tasks, it can not only avoid system crashes caused by too many scanning tasks, but also control the number of scanning tasks currently being processed as a whole; in addition, the detection server that sends detection requests to the website to be scanned is also distributed and can be expanded as needed. After the distributed scheduling server obtains the task request of each distributed detection server, it pushes each subtask obtained to different distributed detection servers respectively. The distributed detection server can send vulnerability detection requests to the website to be scanned according to the subtask. In this way, even if the concurrency of the vulnerability scanning task is large, each vulnerability scanning task can be processed quickly and orderly to improve the scanning efficiency.

[0185] For the specific definition of the task processing device 1200, please refer to the definition of the task processing method above, which will not be repeated here. Each module in the above-mentioned task processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0186] In one embodiment, a computer device is provided. The computer device may be a distributed scheduling server, and its internal structure diagram may be as follows: Fig.13As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a task processing method is implemented.

[0187] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0188] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0189] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0190] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0191] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0192] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A task processing method, the method comprising: The distributed scheduling server receives vulnerability scanning tasks corresponding to the website to be scanned; The distributed scheduling server stores the vulnerability scanning tasks in a first message queue in sequence. The first message queue is shared by the distributed scheduling servers assigned to the same group. The vulnerability scanning tasks received by the distributed scheduling servers assigned to the same group are stored in the same message queue, and the vulnerability scanning tasks received by the distributed scheduling servers assigned to different groups are stored in different message queues. When the total number of vulnerability scanning tasks currently being processed is less than a preset processing upper limit, the distributed scheduling server extracts vulnerability scanning tasks from the first message queue and pushes the extracted vulnerability scanning tasks to the task channel; When there is a vulnerability scanning task in the task channel, the distributed scheduling server takes out the vulnerability scanning task from the task channel and sends it to the distributed decomposition server, requesting the distributed decomposition server to perform task decomposition; The distributed scheduling server receives subtasks obtained and returned by the distributed decomposition server after decomposing the vulnerability scanning task, and stores the subtasks in the second message queue in sequence; The distributed scheduling server receives a task request from a distributed detection server, and when it is found that there is a subtask in the second message queue, the distributed scheduling server pushes the subtask to the distributed detection server in response to the task request, and the distributed detection server is a server that directly interacts with the website to be scanned; The distributed scheduling server obtains the scanning speed of the website to be scanned from the scanning configuration information of the vulnerability scanning task, generates a detection token of the website to be scanned according to the scanning speed, and determines whether the token bucket is full. If so, the generated detection token is discarded; if not, the generated detection token is stored in the token bucket; The distributed scheduling server receives the token request from the distributed detection server, determines whether there is a detection token for the website to be scanned in the token bucket, and when it is found that there is a detection token for the website to be scanned, takes out the detection token for the website to be scanned from the token bucket, and returns the detection token for the website to be scanned to the distributed detection server in response to the token request. The detection token is used to control the distributed detection server to send a vulnerability detection request to the website to be scanned according to the subtask after obtaining the detection token for the website to be scanned. When it is found that the detection token of the website to be scanned does not exist currently, the distributed detection server is notified to wait.

2. The method according to claim 1, characterized in that The step of sequentially storing the vulnerability scanning tasks in the first message queue includes: Parsing the vulnerability scanning task to obtain the URL and scanning configuration information of the website to be scanned; The URL of the website to be scanned and the scanning configuration information are correspondingly stored in the first message queue.

3. The method according to claim 1, characterized in that The method further comprises: Determine the total number of vulnerability scan tasks currently being processed; When the total quantity is greater than or equal to the preset processing upper limit, The extraction of the vulnerability scanning task from the first message queue is suspended.

4. The method according to claim 1, characterized in that The taking out the vulnerability scanning task from the task channel and sending it to the distributed decomposition server, and requesting the distributed decomposition server to perform task decomposition, comprises: After taking out the vulnerability scanning task from the task channel, generating a decomposition request for the vulnerability scanning task; The decomposition request is sent to the distributed decomposition server.

5. The method according to claim 1, characterized in that The step of the distributed decomposition server decomposing the vulnerability scanning task comprises: Traversing a plug-in list, the plug-in list including plug-ins for detecting different types of vulnerabilities; Determining, from the plug-in list, a designated plug-in for detecting the vulnerability scanning task according to the scanning configuration information of the vulnerability scanning task; Traverse the test sample list corresponding to the specified plug-in; Determining, from the test sample list, a designated test sample for detecting the vulnerability scanning task according to the scanning configuration information of the vulnerability scanning task; A subtask related to the vulnerability scanning task is generated according to the specified test sample.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Receiving, by the distributed detection server, a response result returned by the website to be scanned in response to the vulnerability detection request; The response result is matched with the vulnerability detection rule corresponding to the subtask to obtain the vulnerability detection result of the website to be scanned.

7. A task processing system, comprising a distributed scheduling server, a distributed decomposition server and a distributed detection server, wherein: The distributed scheduling server is used to receive vulnerability scanning tasks corresponding to the websites to be scanned, and store the vulnerability scanning tasks in a first message queue in sequence. The first message queue is shared by the distributed scheduling servers assigned to the same group. The vulnerability scanning tasks received by the distributed scheduling servers assigned to the same group are stored in the same message queue, and the vulnerability scanning tasks received by the distributed scheduling servers assigned to different groups are stored in different message queues. When the total number of vulnerability scanning tasks currently being processed is less than a preset processing upper limit, the vulnerability scanning tasks are extracted from the first message queue, and the extracted vulnerability scanning tasks are pushed to the task channel. When there are vulnerability scanning tasks in the task channel, the vulnerability scanning tasks are taken out from the task channel and sent to the distributed decomposition server, requesting the distributed decomposition server to perform task decomposition. The distributed decomposition server is used to decompose the vulnerability scanning task to obtain subtasks, and return the subtasks to the distributed scheduling server; The distributed scheduling server is used to receive subtasks obtained and returned by the distributed decomposition server after decomposing the vulnerability scanning task, and store the subtasks in the second message queue in sequence; The distributed detection server is used to send a task request to the distributed scheduling server, and the distributed detection server is a server that directly interacts with the website to be scanned; The distributed scheduling server is used to receive the task request of the distributed detection server, and when it is found that there is a subtask in the second message queue, the distributed scheduling server pushes the subtask to the distributed detection server in response to the task request; The distributed scheduling server is also used to obtain the scanning speed of the website to be scanned from the scanning configuration information of the vulnerability scanning task, generate a detection token of the website to be scanned according to the scanning speed, and determine whether the token bucket is full. If so, discard the generated detection token; if not, store the generated detection token in the token bucket; The distributed detection server is used to send a token request to the distributed scheduling server; The distributed scheduling server is also used to receive a token request from the distributed detection server, determine whether there is a detection token for the website to be scanned in the token bucket, and when it is found that there is a detection token for the website to be scanned, take out the detection token for the website to be scanned from the token bucket, and return the detection token for the website to be scanned to the distributed detection server in response to the token request; when it is found that there is no detection token for the website to be scanned, notify the distributed detection server to wait; The distributed detection server is used to send a vulnerability detection request to the website to be scanned according to the subtask after acquiring the detection token of the website to be scanned.

8. A task processing device, comprising: A receiving module, used for receiving vulnerability scanning tasks corresponding to the website to be scanned; A persistence module, used for storing the vulnerability scanning tasks in a first message queue in sequence, the first message queue is shared by the distributed scheduling servers assigned to the same group, the vulnerability scanning tasks received by the distributed scheduling servers assigned to the same group are stored in the same message queue, and the vulnerability scanning tasks received by the distributed scheduling servers assigned to different groups are stored in different message queues; An extraction module is used to extract vulnerability scanning tasks from the first message queue when the total number of vulnerability scanning tasks currently being processed is less than a preset processing upper limit value, push the extracted vulnerability scanning tasks to the task channel, and when there are vulnerability scanning tasks in the task channel, take out the vulnerability scanning tasks from the task channel and send them to the distributed decomposition server, requesting the distributed decomposition server to perform task decomposition; An acquisition module, used for receiving subtasks obtained and returned by the distributed decomposition server after decomposing the vulnerability scanning task; The persistence module is further used to store the subtasks in the second message queue in sequence; A task distribution module, configured to receive a task request from a distributed detection server, and when a subtask is found in the second message queue, push the subtask to the distributed detection server in response to the task request, wherein the distributed detection server is a server that directly interacts with the website to be scanned; A token generation module, used to obtain the scanning speed of the website to be scanned from the scanning configuration information of the vulnerability scanning task, generate a detection token of the website to be scanned according to the scanning speed, and determine whether the token bucket is full. If so, discard the generated detection token; if not, store the generated detection token in the token bucket; A token request module, used for receiving a token request from the distributed detection server, determining whether there is a detection token for the website to be scanned in the token bucket, and when it is found that there is a detection token for the website to be scanned, taking out the detection token for the website to be scanned from the token bucket, and returning the detection token for the website to be scanned to the distributed detection server in response to the token request, wherein the detection token is used to control the distributed detection server to send a vulnerability detection request to the website to be scanned according to the subtask after obtaining the detection token for the website to be scanned; When it is found that the detection token of the website to be scanned does not exist currently, the distributed detection server is notified to wait.

9. The device according to claim 8, characterized in that The persistence module is also used to parse the vulnerability scanning task to obtain the URL of the website to be scanned and the scanning configuration information; and store the URL of the website to be scanned and the scanning configuration information in the first message queue accordingly.

10. The device according to claim 8, characterized in that The extraction module is further used to determine the total number of vulnerability scanning tasks currently being processed; when the total number is greater than or equal to a preset processing upper limit, the extraction of the vulnerability scanning tasks from the first message queue is suspended.

11. The device according to claim 8, characterized in that The acquisition module is further used to send the vulnerability scanning task extracted from the first message queue to the decomposition server; and receive the subtasks returned by the decomposition server after decomposing the vulnerability scanning task.

12. The device according to claim 8, characterized in that The acquisition module is further used to generate a decomposition request for the vulnerability scanning task after taking out the vulnerability scanning task from the task channel; and send the decomposition request to the distributed decomposition server.

13. The device according to any one of claims 8 to 12, characterized in that The device also includes: The detection result acquisition module is used to receive the response result returned by the website to be scanned in response to the vulnerability detection request through the distributed detection server; match the response result with the vulnerability detection rule corresponding to the subtask to obtain the vulnerability detection result of the website to be scanned.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images