A dynamically allocated website monitoring and scheduling method

By dynamically allocating website monitoring scheduling methods, the execution order and parallelism of monitoring tasks are adjusted according to task type and priority, thus solving the task skew problem and achieving balanced acquisition of monitoring tasks and reasonable triggering of firewall policies.

CN113946422BActive Publication Date: 2024-11-05BEIJING TOPSEC NETWORK SECURITY TECH +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111210276.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-18
Publication Date
2024-11-05
Estimated Expiration
2041-10-18

AI Technical Summary

Technical Problem

In traditional website monitoring, the order in which tasks are created can lead to task skew, which can easily trigger customer firewall policies and cause unnecessary trouble.

Method used

A dynamic website monitoring scheduling method is adopted, which dynamically adjusts the execution order and parallelism of monitoring tasks through task type, priority and parallel limit strategy to ensure balanced acquisition of monitoring tasks.

Benefits of technology

This reduces task skew, improves the balanced acquisition of monitoring tasks, lowers the risk of triggering firewall policies, and enhances the flexibility and rationality of monitoring task allocation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113946422B_ABST
    Figure CN113946422B_ABST
Patent Text Reader

Abstract

The present application relates to a dynamically allocated website monitoring scheduling method, comprising the following steps: obtaining a monitoring target and a monitoring task corresponding to the monitoring target, and assigning a priority to the monitoring task; storing the monitoring task in a first queue to be executed according to the task type; traversing the first queue to be executed, obtaining a first type of monitoring task, and taking out executable first type of monitoring tasks from the first queue to be executed based on the priority order; determining whether it is necessary to limit the parallel first type of monitoring tasks according to the monitoring target dimension; if not, directly executing the monitoring task operation on the first type of monitoring target based on the priority order; if so, executing the monitoring task operation on the first type of monitoring target based on the priority order and the preset monitoring target dimension task parallel restriction scheme; executing all types of monitoring task operations in the first queue to be executed. The present application has the effect of reducing task inclination and obtaining monitoring tasks in a balanced manner during the website monitoring process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of website monitoring, and in particular to a dynamically allocated website monitoring scheduling method. Background Art

[0002] Website monitoring specifically refers to collecting data information from the pages of customer-authorized websites, analyzing based on strategies whether the website has been maliciously modified, has Trojans implanted, has vulnerabilities, and other risks, and notifying customers through alarms and reports when the analysis shows that there is a risk.

[0003] The traditional collection process is to obtain the collection tasks in the order of task creation time. Therefore, when multiple website tasks are created at a similar time, the collection is concentrated on the subtasks of a website created first, while the subtask collection of other websites is ignored.

[0004] With respect to the above-mentioned related technologies, the inventors believe that centralized collection of website subtasks is prone to task bias. Since customer-authorized websites generally have unknown firewall policies, operations for centralized collection are very likely to trigger their firewall policies, causing unnecessary trouble. Summary of the invention

[0005] In order to reduce task inclination and acquire monitoring tasks in a balanced manner during website monitoring, the present application provides a website monitoring scheduling method with dynamic allocation.

[0006] The present application provides a method for dynamically allocating website scheduling, which adopts the following technical solutions:

[0007] A method for dynamically allocating website scheduling, a method for dynamically allocating website monitoring scheduling, comprising the following steps:

[0008] Obtaining a monitoring target and a monitoring task corresponding to the monitoring target, and assigning a priority to the monitoring task, wherein the monitoring task includes a parent task and a child task;

[0009] Storing the monitoring task into a first queue to be executed according to the task type;

[0010] Traversing the first queue to be executed, obtaining monitoring tasks of the first type, and taking out executable monitoring tasks of the first type from the first queue to be executed based on the priority order;

[0011] Determine whether it is necessary to limit the parallel first type of monitoring tasks according to the monitoring target dimension;

[0012] If not, directly performing the monitoring task operation on the first type of monitoring target based on the priority order;

[0013] If so, performing a monitoring task operation on the monitoring target of the first type based on the priority order and a preset monitoring target dimension task parallel restriction scheme;

[0014] Execute all types of monitoring task operations in the first to-be-executed queue.

[0015] By adopting the above technical scheme, the monitoring tasks are sorted according to the task type and stored in the first queue to be executed. The executable part of the first type of monitoring tasks is taken out in order of priority. If it is necessary to limit the parallel first type of monitoring tasks according to the monitoring target dimension, the monitoring task operation is performed on the first type of monitoring target based on the priority order and the preset monitoring target dimension task parallel restriction scheme. If it is not necessary to limit the parallel first type of monitoring tasks according to the monitoring target dimension, the monitoring task operation is directly performed on the first type of monitoring target based on the priority order. All types of monitoring tasks in the first queue to be executed are processed according to the allocation method of the first type of monitoring tasks. This method assigns priorities to all monitoring tasks, and gives priority to executing monitoring tasks with higher priorities in the process of executing monitoring tasks, rather than executing them in the chronological order of the creation of monitoring tasks, which reduces task skewness and makes the acquisition of monitoring tasks more balanced.

[0016] Optionally, taking out executable monitoring tasks of the first type from the first to-be-executed queue in the priority order comprises the following steps:

[0017] Traversing the first queue to be executed to obtain a first type of monitoring task;

[0018] Query the number of parallel tasks and the parallel restriction strategy of the first type of monitoring tasks according to a preset time cycle, wherein the number of parallel tasks is the total number of the first type of monitoring tasks being executed, and the parallel restriction strategy is the maximum total number of the first type of monitoring tasks that can be executed simultaneously;

[0019] The number of parallel tasks and the parallel restriction strategy are analyzed to obtain a first analysis result, and executable monitoring tasks of the first type are retrieved based on the priority order and the first analysis result.

[0020] By adopting the above technical solution, it can be known whether new monitoring tasks can be added according to the number of parallel tasks and the parallel restriction strategy, which is convenient for allocation.

[0021] Optionally, the analyzing the number of parallel tasks and the parallel restriction strategy to obtain a first analysis result, and extracting executable monitoring tasks of the first type based on the priority order and the first analysis result comprises the following steps:

[0022] Determine whether the number of parallel tasks is less than the parallel limit strategy;

[0023] If the number of parallel tasks is greater than or equal to the parallel restriction strategy, then re-obtain the number of parallel tasks and the parallel restriction strategy of the first type of monitoring task after a preset time, and analyze the number of parallel tasks and the parallel restriction strategy again;

[0024] If the number of parallel tasks is less than the parallel limitation strategy, the executable monitoring tasks are taken out from the first queue to be executed in order of priority, and the total number of executable tasks of the first type of monitoring tasks is obtained, and the total number of executable tasks is the difference between the parallel limitation strategy and the number of parallel tasks.

[0025] By adopting the above technical solution, when the number of parallel tasks is greater than or equal to the parallel restriction strategy, it means that the workload of the current first type of monitoring probe has been saturated, and the first type of monitoring tasks can no longer be added at this time; when the number of parallel tasks of the first type is less than the parallel restriction strategy, the monitoring tasks with higher priorities are taken out in order of priority, and the total number of executable monitoring tasks is the difference between the parallel restriction strategy and the number of parallel tasks, indicating that monitoring tasks can continue to be added, which can reduce the blindness of monitoring task allocation and make the acquisition of monitoring tasks more balanced.

[0026] Optionally, the performing of the monitoring task operation based on the priority of the monitoring task and a preset monitoring target dimension task parallel restriction scheme includes the following steps:

[0027] Storing the first type of monitoring tasks in a second queue to be executed;

[0028] Classifying the monitoring tasks in the second to-be-executed queue to form a first set, wherein the first set includes a plurality of the monitoring targets and a parent task and a child task corresponding to each of the monitoring targets;

[0029] Traversing the first set, obtaining a first monitoring target with a higher priority from the first set in order of priority, obtaining the number of subtask parallelism and the subtask limitation policy of the first monitoring target, analyzing the number of subtask parallelism and the subtask limitation policy to obtain a second analysis result, wherein the number of subtask parallelism is the number of monitoring tasks of the first type being executed by the first monitoring target, and the subtask limitation policy is the maximum number of monitoring tasks that the first monitoring target can support to be executed simultaneously;

[0030] Performing a monitoring task operation on the first monitoring target based on the priority order and the second analysis result;

[0031] All monitoring task operations in the second to-be-executed queue are executed based on the priority order.

[0032] By adopting the above technical solution, the monitoring task operation is performed on the first type of monitoring target based on the priority order and the preset monitoring target dimension task parallel restriction scheme, which can further improve the rationality of monitoring task allocation.

[0033] Optionally, the analyzing the subtask parallel number and the subtask restriction strategy to obtain a second analysis result comprises the following steps:

[0034] Determine whether the number of subtask parallelism of the first monitoring target is less than the subtask limitation strategy;

[0035] If the number of subtask parallelism is greater than or equal to the subtask restriction strategy, reacquire the number of subtask parallelism and the subtask restriction strategy of the first monitoring target after a preset time, and analyze the number of subtask parallelism and the subtask restriction strategy again;

[0036] If the number of subtask parallelism is less than the subtask limitation strategy, the parent task and subtask of the first monitoring target are obtained to form a second set, and the number of executable subtasks of the first monitoring target is obtained, and the number of executable subtasks is equal to the difference between the subtask limitation strategy and the number of subtask parallelism.

[0037] By adopting the above technical solution, if the number of subtasks in parallel is greater than or equal to the subtask limit policy, it means that the number of monitoring tasks executed simultaneously has been saturated, and the firewall policy of the website will be triggered, and no more monitoring tasks can be added; if the number of subtasks in parallel is less than the subtask limit policy, it means that monitoring tasks to be executed can still be added at this time, which can reduce the blindness of monitoring task allocation and make monitoring task allocation more reasonable.

[0038] Optionally, the performing the monitoring task operation on the first monitoring target based on the priority order and the second analysis result includes the following steps:

[0039] Traversing the second set, and taking out monitoring tasks with higher priorities according to the priority order;

[0040] Add the taken monitoring task to the third queue to be executed, and reduce the number of executable subtasks and the total number of executable tasks by 1;

[0041] The total number of executable tasks is analyzed to obtain a third analysis result, and the monitoring task operations in the third queue to be executed are executed based on the third analysis result.

[0042] By adopting the above technical solution, it can be known whether all monitoring tasks of the first type are taken out according to the total number of executable tasks, which is convenient for allocation.

[0043] Optionally, analyzing the total number of executable tasks to obtain a third analysis result, and executing the monitoring task operation in the third queue to be executed based on the third analysis result comprises the following steps:

[0044] Determine whether the total number of executable tasks of the first type is less than or equal to 0;

[0045] If the total number of executable tasks is less than or equal to 0, directly execute all subtasks in the third queue to be executed;

[0046] If the total number of executable tasks is greater than 0, the number of executable subtasks is analyzed to obtain a fourth analysis result, and the monitoring task operation in the third queue to be executed is executed based on the fourth analysis result.

[0047] By adopting the above technical solution, it is determined whether the total number of executable tasks of the first type is less than or equal to 0, so as to know whether all monitoring tasks of the first type are taken out.

[0048] Optionally, analyzing the number of executable subtasks to obtain a fourth analysis result, and executing the monitoring task operation in the third queue to be executed based on the fourth analysis result comprises the following steps:

[0049] Determine whether the number of executable subtasks is less than or equal to 0;

[0050] If the number of executable subtasks is greater than 0, continue to traverse the second set, and take out monitoring tasks with higher priorities in order of priority and add them to the third queue to be executed;

[0051] If the number of executable subtasks is less than or equal to 0, the first set is analyzed to obtain a fifth analysis result, and the monitoring task operation in the third queue to be executed is executed based on the fifth analysis result.

[0052] By adopting the above technical solution, it is determined whether the number of executable subtasks of the first monitoring target is less than or equal to 0, so as to know whether all the monitoring tasks in the second set are taken out.

[0053] Optionally, analyzing the first set and obtaining a fifth analysis result comprises the following steps:

[0054] Determine whether the traversal of the first set is completed;

[0055] If the traversal of the first set has not been completed, continue to traverse the first set, obtain the second monitoring target with a higher priority from the first set in order of priority, obtain the subtask parallel number and subtask limitation strategy of the second monitoring target again, and analyze the subtask parallel number and subtask limitation strategy of the second monitoring target again;

[0056] If the traversal of the first set is completed, all monitoring tasks in the third queue to be executed are executed.

[0057] By adopting the above technical solution, it is determined whether the traversal of the first set is completed, and whether all types of monitoring tasks are completely taken out.

[0058] Optionally, the determining whether the traversal of the first set is completed comprises the following steps:

[0059] Obtaining a total target quantity of all monitored targets, and determining the number of operations based on the total target quantity;

[0060] When a monitoring target is taken out from the first set, the number of operations is reduced by 1;

[0061] Whether the traversal of the first set is completed is determined according to whether the number of operations is greater than 0. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the overall process structure of a dynamically allocated website monitoring and scheduling method according to an embodiment of the present application.

[0063] Figure 2 It is a flow structure diagram of steps S201-S203 in a dynamically allocated website monitoring and scheduling method according to an embodiment of the present application.

[0064] Figure 3 It is a flow structure diagram of steps S301-S303 in a dynamically allocated website monitoring and scheduling method according to an embodiment of the present application.

[0065] Figure 4 It is a flow structure diagram of steps S401-S405 in a dynamically allocated website monitoring and scheduling method according to an embodiment of the present application.

[0066] Figure 5 It is a flow structure diagram of steps S501-S503 in a dynamically allocated website monitoring and scheduling method according to an embodiment of the present application.

[0067] Figure 6 It is a flow structure diagram of steps S601-S603 in a dynamically allocated website monitoring and scheduling method according to an embodiment of the present application.

[0068] Figure 7 It is a flow structure diagram of steps S701-S703 in a dynamically allocated website monitoring and scheduling method according to an embodiment of the present application.

[0069] Figure 8 It is a flow structure diagram of steps S801-S803 in a dynamically allocated website monitoring and scheduling method according to an embodiment of the present application.

[0070] Fig. 9 It is a flow structure diagram of steps S901-S903 in a dynamically allocated website monitoring and scheduling method according to an embodiment of the present application.

[0071] Fig.10 It is a flow structure diagram of steps S901a-S903c in a dynamically allocated website monitoring scheduling method according to an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following is combined with Figure 1-Figure 10 This application is described in further detail.

[0073] The present application embodiment discloses a method for dynamically allocating website monitoring and scheduling, referring to Figure 1 , including the following steps:

[0074] S101, obtaining a monitoring target and a monitoring task corresponding to the monitoring target, and assigning a priority to the monitoring task, wherein the monitoring task includes a parent task and a child task;

[0075] S102, storing the monitoring task into a first queue to be executed according to the task type;

[0076] S103, traversing the first queue to be executed, obtaining the monitoring tasks of the first type, and taking out executable monitoring tasks of the first type from the first queue to be executed based on the priority order;

[0077] S104, determining whether it is necessary to limit parallel monitoring tasks according to the monitoring target dimension;

[0078] S105: If it is necessary to limit the parallel monitoring tasks according to the monitoring target dimension, the monitoring task operation is performed on the monitoring target based on the priority order and the preset monitoring target dimension task parallel limitation scheme;

[0079] S106: If there is no need to limit the parallel monitoring tasks according to the monitoring target dimension, directly perform the monitoring task operation on the monitoring target;

[0080] S107: Execute all types of monitoring task operations in the first queue to be executed.

[0081] Among them, in step S101, firstly, the monitoring target and the monitoring task corresponding to the monitoring target are obtained. The monitoring target is a website. In this embodiment, the monitoring task includes a parent task and a child task, and includes multiple types. Generally speaking, the parent task is to monitor the homepage of the website, and the child task is to monitor the sub-links of the website. When obtaining the monitoring target and the monitoring task of the monitoring target, priority values ​​are assigned to all monitoring tasks according to different needs, so that more important monitoring tasks are executed first.

[0082] Specifically, taking the Baidu website as an example, the Baidu website is the monitoring target, and the monitoring tasks include the parent task of monitoring the Baidu main website and the sub-task of monitoring the Baidu sub-links. The monitoring tasks can be divided into website availability ping, availability http, availability socket, domain name monitoring, web page sensitive word monitoring, web page tampering monitoring, web page dark link monitoring, website Trojan horse monitoring and website vulnerability scanning and other types. In the actual monitoring process, the above-mentioned monitoring tasks are all applicable to the parent task, that is, all the above-mentioned types of tasks can be performed on the Baidu main website. The availability ping, availability http, availability socket, domain name monitoring, and website Trojan horse monitoring types are not applicable to sub-tasks, that is, the sub-links of the Baidu website are not applicable to the execution of availability ping, availability http, availability socket, domain name monitoring, and website Trojan horse monitoring tasks, while the web page sensitive word monitoring, web page tampering monitoring, web page dark link monitoring and website vulnerability scanning are applicable to sub-tasks, that is, the sub-links of the Baidu website are applicable to web page sensitive word monitoring, web page tampering monitoring, web page dark link monitoring and website vulnerability scanning monitoring tasks.

[0083] Specifically, taking Baidu website and Sina website as examples, the monitoring tasks of Baidu website and Sina website are prioritized. For example, the monitoring task of Baidu website is identified as "1", indicating an expedited processing priority, and the monitoring task of Sina website is identified as "0", indicating an ordinary processing priority. In this embodiment, a monitoring task with a priority of "ordinary processing" is defined, and its parent task and child task are assigned priorities according to 0-1000. A monitoring task with a priority of "expedited processing" is defined, and its parent task and child task are assigned priorities according to 1001-2000. It can be seen from the above that the priority of the parent task and child task of monitoring Baidu website is assigned a random number between 1001-2000, and the priority of the parent task and child task of monitoring Sina website is assigned a random number between 0-1000.

[0084] Among them, in step S102, the monitoring tasks are stored in the first queue to be executed according to the task type, and all monitoring tasks are classified and sorted according to multiple types such as website availability ping, availability http, availability socket, domain name monitoring, webpage sensitive word monitoring, webpage tampering monitoring, webpage dark link monitoring, website Trojan horse monitoring and website vulnerability scanning, and the sorted monitoring tasks are stored in the first queue to be executed. For example:

[0085] 1. Availability ping task type: Baidu main website (parent task 1), Sina main website (parent task 2)...;

[0086] 2. Website vulnerability scanning task types: Baidu main website (parent task 1), Baidu sub-link 1 (sub-task 11), Baidu sub-link 2 (sub-task 12), Sina main website (parent task 2), Sina sub-link 1 (parent task 21), Sina sub-link 2 (parent task 22)...;

[0087] …

[0088] Among them, in step S103, the first queue to be executed is traversed to obtain monitoring tasks of the same type, and some monitoring tasks that can be executed currently are taken out according to the priority order. Figure 2 Steps S201 to S203 in:

[0089] S201, traverse the first queue to be executed to obtain a first type of monitoring task;

[0090] S202, obtaining the number of parallel tasks and the parallel restriction strategy of the first type of monitoring tasks according to a preset time cycle, wherein the number of parallel tasks is the total number of the first type of monitoring tasks being executed, and the parallel restriction strategy is the maximum total number of the first type of monitoring tasks that can be executed simultaneously;

[0091] S203: Analyze the number of parallel tasks and the parallel restriction strategy to obtain a first analysis result, and extract executable monitoring tasks of the first type based on the priority order and the first analysis result.

[0092] In step S201, the first queue to be executed is traversed to take out the first type of monitoring tasks, for example, the website vulnerability scanning monitoring task is taken as the first type of monitoring task.

[0093] In step S202, the number of parallel tasks of the first type and the parallel restriction strategy of the first type are obtained cyclically, the number of parallel tasks represents the total number of tasks of the first type currently being executed, and the parallel restriction strategy represents the maximum total number of tasks that can be supported to be executed simultaneously. The preset time can be adjusted according to actual needs. In this embodiment, the preset time is set to 20s, that is, the number of parallel tasks of the first type and the parallel restriction strategy are obtained once every 20s.

[0094] Specifically, monitoring tasks are executed through monitoring probes, which can be regarded as carriers for executing monitoring tasks. The parallel limitation strategy is proportional to the monitoring probes that execute the task, and the maximum number of tasks that each monitoring probe can execute is fixed. It can be seen that the more monitoring probes there are, the more monitoring tasks can be executed simultaneously. For example, if the number of monitoring probes responsible for executing website vulnerability scanning tasks is fixed at 10, and the maximum number of tasks that each monitoring probe can execute is 20, then the number of website vulnerability scanning tasks that can be executed simultaneously is 200, that is, the parallel limitation strategy for the website vulnerability scanning monitoring task is 200.

[0095] In step S203, the number of parallel tasks and the parallel monitoring strategy are analyzed to obtain a first analysis result, thereby determining whether a monitoring task can be added for execution according to the first analysis result. Figure 3 Steps S301 to S303 in:

[0096] S301, determining whether the number of parallel tasks is less than the parallel limit policy;

[0097] S302: if the number of parallel tasks is greater than or equal to the parallel restriction strategy, reacquire the number of parallel tasks and the parallel restriction strategy of the first type of monitoring task after a preset time, and analyze the number of parallel tasks and the parallel restriction strategy again;

[0098] S303. If the number of parallel tasks is less than the parallel restriction strategy, the executable monitoring tasks are taken out from the first queue to be executed in order of priority, and the total number of executable tasks of the first type of monitoring tasks is obtained. The total number of executable tasks is the difference between the parallel restriction strategy and the number of parallel tasks.

[0099] Specifically, when the number of parallel tasks of the first type is greater than or equal to the parallel restriction strategy, it means that the workload of the current first type of monitoring probe has been saturated, and the first type of monitoring tasks can no longer be added. After waiting for a preset time, the number of parallel tasks and the parallel restriction strategy of the first type of monitoring task are obtained again, and the analysis is performed again until the number of parallel tasks is less than the parallel restriction strategy.

[0100] When the number of parallel tasks of the first type is less than the parallel limit policy, the monitoring tasks with higher priority are taken out in order of priority, and the total number of monitoring tasks that can be executed is the difference between the parallel limit policy and the number of parallel tasks. For example, if the parallel limit policy of the website vulnerability scanning monitoring task is 200 and the number of parallel tasks is 180, the total number of executable tasks is 200-180=20, which means that vulnerability scanning monitoring tasks for 20 more websites can be received.

[0101] Among them, after taking out the executable monitoring tasks of the same type, it is determined whether the parent task needs to limit the parallel monitoring tasks according to the monitoring target dimension, which can be understood as whether the subtasks of the parent task need to be dynamically allocated. Taking the Baidu website as an example, the availability ping, availability http, availability socket, domain name monitoring, and website Trojan horse monitoring types are not applicable to the sublinks of the Baidu website, and there is no need to dynamically allocate these subtasks; web page sensitive word monitoring, web page tampering monitoring, web page dark link monitoring, and website vulnerability scanning are applicable subtasks, and these subtasks need to be dynamically allocated.

[0102] Reference Figure 1In step S104 to step S106, the first type of monitoring task is judged, and if it is not necessary to limit the parallel monitoring tasks according to the monitoring target dimension, the monitoring task operation is directly performed according to the priority order of the monitoring tasks;

[0103] If it is necessary to limit parallel monitoring tasks according to the monitoring target dimension, the monitoring task operation is performed on the first type of monitoring target based on the priority order and the preset monitoring target dimension task parallel limitation scheme. For details, refer to Figure 4 Steps S401-S405 in .

[0104] S401, storing the first type of monitoring task into a second queue to be executed;

[0105] S402: classify the monitoring tasks in the second to-be-executed queue to form a first set, where the first set includes a number of monitoring targets and a parent task and a child task corresponding to each monitoring target;

[0106] S403, traverse the first set, obtain a first monitoring target with a higher priority from the first set in order of priority, obtain the number of subtask parallelism and the subtask restriction policy of the first monitoring target, analyze the number of subtask parallelism and the subtask restriction policy, and obtain a second analysis result, where the number of subtask parallelism is the number of monitoring tasks of the first type being executed by the first monitoring target, and the subtask restriction policy is the maximum number of monitoring tasks that the first monitoring target can support to be executed simultaneously;

[0107] S404, performing a monitoring task operation on the first monitoring target based on the priority order and the second analysis result;

[0108] S405: Execute all monitoring task operations in the second queue to be executed based on the priority order.

[0109] Specifically, in step S401, the first type of monitoring tasks are stored in the second queue to be executed, for example, the website vulnerability scanning type tasks are stored in the second queue to be executed, and in step S402, the first type of monitoring tasks are classified to form a first set. The website vulnerability scanning monitoring tasks include multiple parent tasks and subtasks corresponding to the parent tasks. For example, the parent tasks and subtasks of the Baidu website and the Sina website both need to perform the website vulnerability scanning monitoring tasks, then the parent tasks and subtasks of the Baidu website are classified into one category, and the parent tasks and subtasks of the Sina website are classified into another category. Therefore, the first set is represented as follows in combination with the priority:

[0110] 1. Baidu website, priority: expedited processing, marked "1":

[0111] Main website: Parent task 1, priority value "2000";

[0112] Sublink 1: Subtask 11, priority assigned to "1800";

[0113] Sublink 2: Subtask 12, priority assigned to "1500"...

[0114] 2. Sina website, priority: normal processing, mark "0":

[0115] Main website: Parent task 2, priority value "1000"

[0116] Sublink 1: Subtask 21, priority assigned to "800";

[0117] Sublink 2: Subtask 22, priority assigned to "500"...

[0118] …

[0119] Specifically, in step S403, the first set is traversed, and the first monitoring target with a higher priority is obtained from the first set in order of priority, and the number of subtask parallelism and subtask restriction policy of the first monitoring target are obtained. It can be seen from the above first set that the Baidu website with a priority mark of "1" is taken out as the first monitoring target, and the number of subtask parallelism and subtask restriction policy of the Baidu website are obtained, wherein the number of subtask parallelism indicates the number of monitoring tasks of the first type being executed by the first monitoring target, and the subtask restriction policy indicates the maximum number of monitoring tasks that the first monitoring target can support to be executed simultaneously. The subtask restriction policy is obtained based on the firewall policy of each monitoring target website. The firewall policy of each website is different, so the subtask restriction policy of each monitoring target is different. Taking the Baidu website as an example, assuming that the firewall policy of the Baidu website allows up to 20 monitoring tasks to be executed simultaneously, it is determined that the subtask restriction policy of the Baidu website is 20.

[0120] Specifically, after obtaining the number of subtask parallelism and the subtask restriction strategy, analyze the number of subtask parallelism and the subtask restriction strategy to determine whether to continue adding subtasks to be executed. For details, refer to Figure 5 Steps S501-S503 in .

[0121] S501, determining whether the number of subtask parallelism of the first monitoring target is less than the subtask limit strategy;

[0122] S502: if the number of subtask parallelism is greater than or equal to the subtask restriction strategy, reacquire the number of subtask parallelism and the subtask restriction strategy of the first monitoring target after a preset time, and analyze the number of subtask parallelism and the subtask restriction strategy again;

[0123] S503. If the number of subtask parallelism is less than the subtask limitation strategy, obtain the parent task and subtask of the first monitoring target to form a second set, and obtain the number of executable subtasks of the first monitoring target. The number of executable subtasks is equal to the difference between the subtask limitation strategy and the number of subtask parallelism.

[0124] Specifically, in step S501 to step S503, it is determined whether the number of subtask parallelism of the first monitoring target is less than the subtask restriction strategy, so as to know whether monitoring tasks of the first monitoring target can continue to be added.

[0125] If the number of subtask parallelism is greater than or equal to the subtask limit policy, it means that the number of monitoring tasks executed simultaneously has been saturated, and the firewall policy of the website will be triggered, and no more monitoring tasks can be added. At this time, you need to wait for the preset time to pass, obtain the number of subtask parallelism and subtask limit policy of the first monitoring target again, and analyze again until the number of subtask parallelism is less than the subtask limit policy.

[0126] If the number of subtasks in parallel is less than the subtask limit policy, it means that you can continue to add monitoring tasks to be executed. At this time, all the first-type subtasks to be executed for the first monitoring target are obtained to form a second set, and the number of executable subtasks is determined. The number of executable subtasks is the number of monitoring tasks to be executed, which is numerically equal to the difference between the subtask limit policy and the number of subtasks in parallel. Still taking Baidu website as an example, Baidu website has a subtask limit policy of 20 for website vulnerability scanning monitoring tasks. At this time, the number of subtasks being executed in parallel is 15, so the number of executable subtasks is equal to 20-15=5.

[0127] Specifically, in step S404, after analyzing the number of subtask parallelism and the subtask restriction strategy, the subtask operations in the second set are executed according to the priority order of the monitoring tasks and the second analysis result. Figure 6 Steps S601 to S603 in:

[0128] S601, traverse the second set and take out the monitoring tasks with higher priority in order of priority;

[0129] S602, adding the taken monitoring task to the third queue to be executed, and reducing the number of executable subtasks and the total number of executable tasks by 1;

[0130] S603: Analyze the total number of executable tasks to obtain a third analysis result, and execute monitoring task operations in a third queue to be executed based on the third analysis result.

[0131] Specifically, in step S601, the second set is traversed, and the monitoring tasks with higher priority assignments are taken out according to the priority assignment order of all monitoring tasks. For example, among the parent tasks and child tasks of the above-mentioned Baidu website, the parent task 1 has the highest priority assignment, so the parent task 1 needs to be taken out first, and the child task 11 has the second highest priority assignment, so the child task 11 needs to be taken out next.

[0132] In step S602, the retrieved subtasks are stored in the third queue to be executed in the order in which they are retrieved, and the values ​​of the number of executable subtasks and the total number of executable tasks are both reduced by 1. For example, after the parent task 1 of the Baidu website is retrieved, the number of executable subtasks of the Baidu website is 5-1=4, and the total number of executable tasks of the website vulnerability scanning and monitoring task is 20-1=19.

[0133] In step S603, the total number of executable tasks is analyzed to obtain a third analysis result, determine whether all monitoring tasks of the first type are taken out, and execute the monitoring task operations in the third queue to be executed based on the third analysis result. For specific actions, refer to Figure 7 Steps S701 to S703 in:

[0134] S701, determining whether the total number of executable tasks of the first type is less than or equal to 0;

[0135] S702: If the total number of executable tasks is less than or equal to 0, directly execute all subtasks in the third queue to be executed;

[0136] S703: If the total number of executable tasks is greater than 0, analyze the number of executable subtasks to obtain a fourth analysis result, and execute the monitoring task operation in the third queue to be executed based on the fourth analysis result.

[0137] Specifically, in step S701 to step 703, it is determined whether the total number of executable tasks of the first type is less than or equal to 0, so as to know whether all monitoring tasks of the first type are taken out.

[0138] If the total number of executable tasks is less than or equal to 0, it is determined that all monitoring tasks in the first set have been taken out and stored in the third queue to be executed. At this time, all monitoring tasks in the third queue to be executed can be directly executed.

[0139] If the total number of executable tasks is greater than 0, it is determined that the first type of monitoring tasks have not been completely taken out. At this time, the number of executable subtasks is analyzed to obtain a fourth analysis result to determine whether all monitoring tasks in the second set are taken out. For details, refer to Figure 8 Steps S801 to S803 in:

[0140] S801, determine whether the number of executable subtasks is less than or equal to 0;

[0141] S802: If the number of executable subtasks is greater than 0, continue to traverse the second set, and take out monitoring tasks with higher priority in order of priority and add them to the third queue to be executed;

[0142] S803: If the number of executable subtasks is less than or equal to 0, analyze the first set and obtain a fifth analysis result, and execute the monitoring task operation in the third queue to be executed based on the fifth analysis result.

[0143] Specifically, in step S801 to step S803, it is determined whether the number of executable subtasks of the first monitoring target is less than or equal to 0, so as to know whether all monitoring tasks in the second set are taken out.

[0144] If the number of executable subtasks is greater than 0, it is determined that all monitoring tasks in the second set have not been taken out. At this time, the second set is continued to be traversed, and monitoring tasks with higher priorities are taken out in order of priority and added to the third queue to be executed.

[0145] If the number of executable subtasks is less than or equal to 0, it is determined that all monitoring tasks in the second set have been taken out, that is, all monitoring tasks of the first monitoring target have been taken out. At this time, the first set is analyzed and the fifth analysis result is obtained. Based on the fifth analysis result, the monitoring task operation in the third queue to be executed is executed. For details, refer to Fig. 9 Steps S901 to S903 in:

[0146] S901, determine whether the traversal of the first set is completed;

[0147] S902. If the traversal of the first set is not completed, continue to traverse the first set, obtain the second monitoring target with a higher priority from the first set in order of priority, obtain the subtask parallel number and subtask restriction strategy of the second monitoring target again, and analyze the subtask parallel number and subtask restriction strategy of the second monitoring target again;

[0148] S903: If the traversal of the first set is completed, execute all monitoring tasks in the third queue to be executed.

[0149] Specifically, in step S901 to step S903, it is determined whether the traversal of the first set is completed, and it can be known whether all monitoring tasks of the first type of all monitoring targets have been taken out.

[0150] If the first set is not traversed to completion, the first set is traversed continuously, and the second monitoring target is selected in the order of the parent task priority values ​​corresponding to the monitoring target, and steps S403 to S405 are performed on the second monitoring target. For example, if the priority value of the parent task of Baidu website is 2000, the priority value of the parent task of Sina website is 1000, and the priority value of the parent task of Ali website is 500, then Sina website is selected as the second monitoring target, and Ali website is selected as the third monitoring target.

[0151] If the first set has been traversed to completion, it is determined that all monitoring tasks of the first type of all monitoring targets have been taken out and stored in the third queue to be executed. At this time, all monitoring tasks in the third queue to be executed are directly executed.

[0152] More specifically, refer to Fig.10 , step S901 further includes the following steps:

[0153] S901a, obtaining the total number of all monitored targets, and determining the number of operations based on the total number of targets;

[0154] S901b, taking out a monitoring target from the first set, and the number of operations is reduced by 1;

[0155] S901c. Determine whether the traversal of the first set is complete based on whether the number of operations is greater than 0.

[0156] In step S901a-step S901c, the total number of monitoring targets is obtained. For example, if the Baidu website, Sina website and Alibaba website need to be monitored, the total number of targets is determined to be 3. The number of operations is determined according to the total number of targets. If the total number of targets is 3, the number of operations is equal to 3, that is, 3 monitoring targets need to be taken out from the first set.

[0157] When one monitoring target is taken out from the first set, the number of operations is reduced by 1, that is, when the first monitoring target is taken out, the number of operations is equal to 3-1=2.

[0158] When the number of operations is greater than 0, it is determined that the traversal of the first set has not been completed; when the number of operations is equal to 0, it is determined that the traversal of the first set has been completed.

[0159] Specifically, in step S405, the second type of monitoring tasks are selected according to the priority order, and are executed according to the above-mentioned processing method for the first type of monitoring tasks until all types of monitoring tasks in the second queue to be processed are completely executed.

[0160] Specifically, in step S107, all types of monitoring task operations in the first queue to be executed are executed, the first set is traversed, and all types of monitoring tasks are processed according to the processing method of the first type of monitoring tasks described above until all monitoring task operations in the first queue to be executed are completed.

[0161] The implementation principle of a dynamically allocated website monitoring scheduling method in an embodiment of the present application is as follows: all monitoring tasks are stored in a first queue to be executed according to the task type, the first queue to be executed is traversed, the monitoring tasks of the first type are taken out, and the executable monitoring tasks of the first type are taken out based on the relationship between the total number of parallel tasks and the parallel restriction strategy and the priority order of the monitoring tasks. If the taken-out monitoring tasks need to limit the parallel monitoring tasks according to the monitoring target dimension, the executable monitoring tasks of the first type are stored in a second queue to be executed and sorted out to form a first set according to the monitoring target dimension, the monitoring tasks of the first monitoring target are taken out based on the relationship between the parallel number of subtasks of the first monitoring target and the subtask restriction strategy to form a second set, and the monitoring tasks with higher priorities are taken out in order of priority and added to the third queue to be executed until all monitoring tasks in the second set are taken out, and all monitoring task operations in the third queue to be executed are executed according to whether the total number of executable tasks is 0, whether the number of executable subtasks is 0, and whether the first set has been traversed. Similarly, the monitoring task operations of the second monitoring target in the second queue to be executed are executed, and similarly, all types of monitoring task operations in the first queue to be executed are executed. The setting of this method reduces the possibility of allocating tasks according to time, can reduce task inclination during website monitoring, and obtain monitoring tasks evenly; if there are special tasks, the priority assignment of the task can be adjusted so that it is executed first, which is highly flexible, adaptable and practical.

[0162] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A dynamically allocated website monitoring and scheduling method, It is characterized in that The following steps are involved: Obtaining a monitoring target and a monitoring task corresponding to the monitoring target, and assigning a priority to the monitoring task, wherein the monitoring task includes a parent task and a child task; Storing the monitoring task into a first queue to be executed according to the task type; Traversing the first queue to be executed, obtaining the first type of monitoring tasks, and taking out the executable first type of monitoring tasks from the first queue to be executed based on the priority order includes the following steps: traversing the first queue to be executed, obtaining the first type of monitoring tasks; querying the number of parallel tasks and the parallel limitation strategy of the first type of monitoring tasks according to a preset time loop, the number of parallel tasks being the total number of the first type of monitoring tasks being executed, and the parallel limitation strategy being the maximum total number of the first type of monitoring tasks that can be executed simultaneously; analyzing the number of parallel tasks and the parallel limitation strategy to obtain a first analysis result, and taking out the executable first type of monitoring tasks based on the priority order and the first analysis result; Determine whether it is necessary to limit the parallel first type of monitoring tasks according to the monitoring target dimension; If not, directly performing the monitoring task operation on the first type of monitoring target based on the priority order; If so, performing a monitoring task operation on the monitoring target of the first type based on the priority order and a preset monitoring target dimension task parallel restriction scheme; Execute all types of monitoring task operations in the first to-be-executed queue.

2. A method for dynamically allocating website monitoring and scheduling according to claim 1, It is characterized in that The analyzing the number of parallel tasks and the parallel restriction strategy to obtain a first analysis result, and extracting executable monitoring tasks of the first type based on the priority order and the first analysis result comprises the following steps: Determine whether the number of parallel tasks is less than the parallel limit strategy; If the number of parallel tasks is greater than or equal to the parallel restriction strategy, then re-obtain the number of parallel tasks and the parallel restriction strategy of the first type of monitoring task after a preset time, and analyze the number of parallel tasks and the parallel restriction strategy again; If the number of parallel tasks is less than the parallel limitation strategy, the executable monitoring tasks are taken out from the first queue to be executed in order of priority, and the total number of executable tasks of the first type of monitoring tasks is obtained, and the total number of executable tasks is the difference between the parallel limitation strategy and the number of parallel tasks.

3. A method for dynamically allocating website monitoring and scheduling according to claim 2, It is characterized in that The operation of performing the monitoring task based on the priority of the monitoring task and the preset monitoring target dimension task parallel restriction scheme includes the following steps: Storing the first type of monitoring tasks in a second queue to be executed; Classifying the monitoring tasks in the second to-be-executed queue to form a first set, wherein the first set includes a plurality of the monitoring targets and a parent task and a child task corresponding to each of the monitoring targets; Traversing the first set, obtaining a first monitoring target with a higher priority from the first set in order of priority, obtaining the number of subtask parallelism and the subtask limitation policy of the first monitoring target, analyzing the number of subtask parallelism and the subtask limitation policy to obtain a second analysis result, wherein the number of subtask parallelism is the number of monitoring tasks of the first type being executed by the first monitoring target, and the subtask limitation policy is the maximum number of monitoring tasks that the first monitoring target can support to be executed simultaneously; Performing a monitoring task operation on the first monitoring target based on the priority order and the second analysis result; All monitoring task operations in the second to-be-executed queue are executed based on the priority order.

4. A method for dynamically allocating website monitoring and scheduling according to claim 3, It is characterized in that The analyzing the subtask parallel number and the subtask restriction strategy to obtain a second analysis result comprises the following steps: Determine whether the number of subtask parallelism of the first monitoring target is less than the subtask limitation strategy; If the number of subtask parallelism is greater than or equal to the subtask restriction strategy, reacquire the number of subtask parallelism and the subtask restriction strategy of the first monitoring target after a preset time, and analyze the number of subtask parallelism and the subtask restriction strategy again; If the number of subtask parallelism is less than the subtask limitation strategy, the parent task and subtask of the first monitoring target are obtained to form a second set, and the number of executable subtasks of the first monitoring target is obtained, and the number of executable subtasks is equal to the difference between the subtask limitation strategy and the number of subtask parallelism.

5. A method for dynamically allocating website monitoring and scheduling according to claim 4, It is characterized in that The performing of the monitoring task operation on the first monitoring target based on the priority order and the second analysis result comprises the following steps: Traversing the second set, and taking out monitoring tasks with higher priorities according to the priority order; Add the taken monitoring task to the third queue to be executed, and reduce the number of executable subtasks and the total number of executable tasks by 1; The total number of executable tasks is analyzed to obtain a third analysis result, and the monitoring task operations in the third queue to be executed are executed based on the third analysis result.

6. A method for dynamically allocating website monitoring and scheduling according to claim 5, It is characterized in that The analyzing the total number of executable tasks to obtain a third analysis result, and executing the monitoring task operation in the third queue to be executed based on the third analysis result comprises the following steps: Determine whether the total number of executable tasks of the first type is less than or equal to 0; If the total number of executable tasks is less than or equal to 0, directly execute all subtasks in the third queue to be executed; If the total number of executable tasks is greater than 0, the number of executable subtasks is analyzed to obtain a fourth analysis result, and the monitoring task operation in the third queue to be executed is executed based on the fourth analysis result.

7. A method for dynamically allocating website monitoring and scheduling according to claim 6, It is characterized in that The analyzing the number of executable subtasks to obtain a fourth analysis result, and executing the monitoring task operation in the third queue to be executed based on the fourth analysis result comprises the following steps: Determine whether the number of executable subtasks is less than or equal to 0; If the number of executable subtasks is greater than 0, continue to traverse the second set, and take out monitoring tasks with higher priorities in order of priority and add them to the third queue to be executed; If the number of executable subtasks is less than or equal to 0, the first set is analyzed to obtain a fifth analysis result, and the monitoring task operation in the third queue to be executed is executed based on the fifth analysis result.

8. A method for dynamically allocating website monitoring and scheduling according to claim 7, It is characterized in that The step of analyzing the first set and obtaining a fifth analysis result comprises the following steps: Determine whether the traversal of the first set is completed; If the traversal of the first set has not been completed, continue to traverse the first set, obtain the second monitoring target with a higher priority from the first set in order of priority, obtain the subtask parallel number and subtask limitation strategy of the second monitoring target again, and analyze the subtask parallel number and subtask limitation strategy of the second monitoring target again; If the traversal of the first set is completed, all monitoring tasks in the third queue to be executed are executed.

9. A method for dynamically allocating website monitoring and scheduling according to claim 8, It is characterized in that Determining whether the traversal of the first set is completed comprises the following steps: Obtaining a total target quantity of all monitored targets, and determining the number of operations based on the total target quantity; When a monitoring target is taken out from the first set, the number of operations is reduced by 1; Whether the traversal of the first set is completed is determined according to whether the number of operations is greater than 0.