Analysis task scheduling method and device and electronic equipment

Abstract

The invention relates to an analysis task scheduling method and device and electronic equipment, and relates to the technical field of data processing, and the method comprises the steps: determining task parameter information and application resource information of an analysis task, and carrying out verification processing on the analysis task based on the task parameter information; under the condition that the analysis task passes the verification, screening a target computing node matched with the analysis task from a plurality of computing nodes of which the operation state is a normal operation state based on the application resource information; and scheduling the target computing node to execute the analysis task. By applying the scheme of the invention, the plurality of computing nodes in the normal operating state are selected, so that the disabled, abnormal and high-task-load computing nodes are eliminated, and the target computing node matched with the analysis task is screened from the plurality of computing nodes in the normal operating state, so that the task load of the target computing node is improved. The purpose of improving the resource utilization rate of the computing nodes is achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to an analysis task scheduling method, device, and electronic device. Background Art

[0002] At present, when scheduling analysis tasks, the distributed asynchronous message task queue Celery is mainly used to integrate the message middleware for scheduling analysis tasks. The distributed asynchronous message task queue is used for real-time processing and scheduling tasks, which is simple, flexible and reliable. When an operation triggered by a user takes a long time to complete, the distributed asynchronous message task queue will execute the operation asynchronously as a task and return the result to the user after execution.

[0003] However, when scheduling analysis tasks based on the distributed asynchronous message task queue integrated with the message middleware, this method cannot reasonably select computing nodes suitable for the analysis tasks, resulting in reduced resource utilization of the computing nodes and poor analysis task scheduling results. Summary of the invention

[0004] In view of this, the present disclosure provides an analysis task scheduling method, device, and electronic device, the main purpose of which is to solve the technical problem that when scheduling analysis tasks based on distributed asynchronous message task queues integrated with message middleware, it is impossible to reasonably select computing nodes suitable for the analysis tasks, resulting in reduced resource utilization of the computing nodes and poor analysis task scheduling effects.

[0005] According to a first aspect of the present disclosure, a method for scheduling an analysis task is provided, the method comprising:

[0006] Determine task parameter information and application resource information of the analysis task, and perform verification processing on the analysis task based on the task parameter information;

[0007] In the case where the analysis task passes the verification, based on the application resource information, selecting a target computing node matching the analysis task from among a plurality of computing nodes in a normal operation state;

[0008] The target computing node is scheduled to execute the analysis task.

[0009] According to a second aspect of the present disclosure, there is provided an analysis task scheduling device, the device comprising:

[0010] A verification module, used to determine task parameter information and application resource information of the analysis task, and to perform verification processing on the analysis task based on the task parameter information;

[0011] A screening module, configured to screen a target computing node matching the analysis task from among a plurality of computing nodes in a normal operating state based on the application resource information if the analysis task passes verification;

[0012] A scheduling module is used to schedule the target computing node to execute the analysis task.

[0013] According to the third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the aforementioned first aspect.

[0014] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method of the aforementioned first aspect.

[0015] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the method described in the first aspect above.

[0016] The analysis task scheduling method, device, and electronic device provided by the present disclosure, compared with the prior art, the present disclosure determines the task parameter information and the application resource information of the analysis task, and verifies the analysis task based on the task parameter information; when the analysis task passes the verification, based on the application resource information, a target computing node matching the analysis task is screened from multiple computing nodes in a normal operating state; and the target computing node is scheduled to perform the analysis task. By applying the scheme of the present disclosure, multiple computing nodes in a normal operating state are selected, thereby excluding the computing nodes that are disabled, abnormal, and have a high task load, and by screening the target computing node matching the analysis task from multiple computing nodes in a normal operating state, the purpose of improving the resource utilization of the computing nodes is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 A flowchart of an analysis task scheduling method provided by an embodiment of the present disclosure;

[0020] Figure 2 A flowchart of another analysis task scheduling method provided by an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of the principle of selecting a target computing node provided by an embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram of the structure of an analysis task scheduling device provided by an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of the structure of another analysis task scheduling device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, in the absence of conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0025] The following describes an analysis task scheduling method, device, and electronic device according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0026] The present disclosure provides an analysis task scheduling method, device, and electronic device, which achieves the purpose of improving the resource utilization of computing nodes by screening target computing nodes matching the analysis task from multiple computing nodes in normal operating state.

[0027] like Figure 1 As shown, an embodiment of the present disclosure provides an analysis task scheduling method, including:

[0028] Step 101: Determine task parameter information and resource application information of the analysis task, and perform verification processing on the analysis task based on the task parameter information.

[0029] Among them, the analysis task can be a task created in advance when performing bioinformatics analysis. Bioinformatics analysis refers to a field that uses bioinformatics tools, methods and technologies to analyze and study life science data. It is mainly used in the field of molecular biology and is an important means to effectively process, mine and analyze large amounts of bioinformatics data.

[0030] An analysis task may include task parameter information and resource application information. Task parameter information may be the parameters required for bioinformatics analysis, such as the reference species for comparison, library construction method, etc.; resource application information may be the resources applied for the analysis task, such as the number of CPU cores applied for the analysis task, memory size, etc.

[0031] For the embodiments of the present disclosure, the legality of the analysis task can be verified based on the task parameter information, and abnormal and illegal analysis tasks can be terminated in advance to prevent useless analysis of abnormal and illegal analysis tasks and waste of resources. Verifying the legality of the analysis task is to verify whether the task parameter information of the analysis task complies with regulations. If the task parameter information complies with regulations, the analysis task is determined to be legal. If the task parameter information does not comply with regulations, the analysis task is determined to be illegal.

[0032] For the embodiments of the present disclosure, the execution subject may be an analysis task scheduling device. When the analysis task passes verification, the analysis task scheduling device may screen a target computing node that matches the analysis task from multiple computing nodes in a normal operating state based on the application resource information, and schedule the target computing node to execute the analysis task, thereby achieving the purpose of improving the resource utilization of the computing nodes. The specific implementation process may refer to steps 102 and 103 of the embodiment.

[0033] Step 102: When the analysis task passes verification, based on the application resource information, select a target computing node that matches the analysis task from among multiple computing nodes in a normal operation state.

[0034] The operation status may include a suspended operation status, a completed operation status, an abnormal operation status, a stopped operation status, a normal operation status, etc. The normal operation status is when the node operation status of the computing node meets the normal operation conditions, and the computing node is determined to be in a normal operation status. The node operation status may be the status of the computing node during the operation of the analysis task.

[0035] The computing node can be a working node under the open source cloud computing platform, which is mainly responsible for providing computing, storage and network resources for virtual machines. The maximum number of parallel tasks of the computing node can be horizontally expanded according to the actual application situation, which can improve the computing power and availability of the computing environment of the cloud computing platform. The target computing node can be the computing node with the highest degree of matching with the analysis task among multiple computing nodes.

[0036] As a possible implementation method for the disclosed embodiment, a computing node having execution authority for the analysis task and application resource information that meets the requirements of the analysis task can be selected from multiple computing nodes in normal operation as the target computing node.

[0037] Step 103: Schedule the target computing node to execute the analysis task.

[0038] Among them, task scheduling refers to the process of the system executing the analysis task at a specific agreed time in order to automatically complete the analysis task. Task scheduling can save more human resource costs and use the system to automatically execute the analysis task.

[0039] As a possible implementation method, the task management engine Cromwell under the analysis task scheduling device can be used to distribute the analysis tasks to the task scheduler. Then the task scheduler will schedule or delay the target computing node to execute the analysis task according to the scheduling time of the analysis task or the time interval of the scheduling task, which can improve the resource utilization of the computing node.

[0040] In summary, according to the analysis task scheduling method provided by the present disclosure, compared with the prior art, the present disclosure determines the task parameter information and the application resource information of the analysis task, and verifies the analysis task based on the task parameter information; when the analysis task passes the verification, based on the application resource information, selects the target computing node that matches the analysis task from multiple computing nodes whose running status is a normal running status; and schedules the target computing node to execute the analysis task. By applying the scheme of the present disclosure, multiple computing nodes whose running status is a normal running status are selected, thereby excluding the computing nodes that are disabled, abnormal, and have a high task load, and by selecting the target computing node that matches the analysis task from multiple computing nodes whose running status is a normal running status, the purpose of improving the resource utilization of the computing nodes is achieved.

[0041] Further, as a refinement and extension of the above embodiment, in order to fully illustrate the specific implementation process of the method of this embodiment, this embodiment provides the following Figure 2 The specific method shown includes:

[0042] Step 201: Determine sample information and analysis process, and create an analysis task using the sample information and analysis process.

[0043] Among them, the sample information can be biological sample information for bioinformatics analysis, and the sample information can be human sample information, animal sample information, etc.; the analysis process refers to the task workflow for performing different bioinformatics analyses, and the analysis process can be the human whole genome resequencing standard information analysis process, the human whole exome resequencing standard information analysis process, etc.

[0044] For the embodiments of the present disclosure, multiple analysis processes can be constructed based on a workflow language for fields such as bioinformatics / genomics. According to the bioinformatics analysis to be performed by the user, a corresponding analysis process is selected from multiple analysis processes and sample information is entered. According to the default task priority at the analysis process level, resource application information and other information, a corresponding analysis task is created. The user can adjust the priority of the analysis task, task parameter information and resource application information through visual page management, etc. Among them, the workflow language may include but is not limited to the WDL language. The WDL language is a bioinformatics process development language developed by the Broad Institute for defining and organizing tasks and workflows. Compared with the traditional shell process, it has the advantages of good reusability, support for multiple cluster architectures, and convenience for multi-task parallelism.

[0045] Step 202: Determine the task parameter information and resource application information of the analysis task, and perform verification processing on the analysis task based on the task parameter information.

[0046] For the disclosed embodiment, after the created analysis task is submitted and saved, in order to terminate the abnormal and illegal analysis task in advance and prevent useless analysis of the abnormal and illegal analysis task, resulting in a waste of resources, the system will perform preliminary legality verification on the task parameter information of the analysis task according to the selected analysis process to determine whether the analysis task is an abnormal and illegal analysis task. If the analysis task is determined to be an abnormal and illegal analysis task, the relevant abnormal information can be fed back to the user through email, message middleware, etc. for abnormal warning reminder, reducing manual inspection work. If it is determined that the analysis task passes the verification, the verified analysis task is added to the task pool for task preparation, and the task pool is used for asynchronous processing of analysis tasks. Among them, the verification task parameter information can be verified according to the verification rules of the analysis process when defining the required parameters. For example, for whole genome sequencing data, a deep learning model can be selected according to different library construction methods (the library construction method may include an amplicon (PCR) library construction method and a library-free amplification (PCR-free) library construction method), which can be used to verify whether the library construction method of the task parameter information is a PCR library construction method or a PCR-free library construction method supported by the process.

[0047] Step 203: When the analysis task passes verification, based on the application resource information, select a target computing node that matches the analysis task from among multiple computing nodes in a normal operation state.

[0048] For the disclosed embodiments, after the created analysis task is submitted and saved, in order to avoid the analysis task being unable to run normally for a long time in the task pool queue due to operating condition restrictions, resulting in waste of resources, the system will verify the rationality of the computing node based on the selected analysis process to see if it meets the application resource information of the analysis task. If so, and there are computing nodes that meet the operating conditions of the analysis task, the analysis task will be run. Otherwise, the relevant abnormal information can be fed back to the user through email, message middleware, etc. for abnormal warning reminders, thereby reducing manual inspection work. Among them, checking whether the rationality of the computing node meets the application resource information of the analysis task can verify whether there are computing nodes that meet the running conditions of the analysis task. Among them, the hardware configurations of the computing nodes may be different. The running conditions for meeting the analysis task can be that the hardware configuration of the computing node meets the resources required for the analysis task; if the running conditions are met, it is determined that the rationality of the computing node meets the application resource information of the analysis task; otherwise, it is determined that the rationality of the computing node does not meet the application resource information of the analysis task. For example, the hardware configuration of the computing nodes in the current cluster is that the number of CPU cores is 40 and the memory is 70G, while the number of CPU cores required to run the analysis task is 50 and the memory requires 100G. In this case, the analysis task cannot be run, and the abnormal information needs to be fed back to the user through email, message middleware, etc. to prevent task blocking.

[0049] For the disclosed embodiments, multiple initial computing nodes that can be used to execute analysis tasks can be determined based on the file agent deployed on the computing nodes, wherein the initial computing nodes are computing nodes that have the execution authority for the analysis tasks and the application resource information that meets the analysis tasks. When it is detected that the task pool has a ready analysis task, the node operation status of each of the multiple initial computing nodes will be periodically queried through the task management engine Cromwell through cron expressions based on the scheduled task scheduling framework Quartz. The node operation status may include resource usage status, health status, and task load status, etc. The servers of the computing nodes in the cluster are running normally, various hardware statuses are normal, the task load is less than the normal load of the preset task load, and the resource usage status is that there are remaining resources that are not used and there is sufficient resource storage space. The initial computing nodes are determined as target computing nodes that match the analysis tasks. Figure 3 As shown, exemplarily, when the analysis task is on storage node 3, the analysis task will be run on one of computing nodes 2, computing node 3 and computing node 4, and will be optimized to be selected when the node operating status of computing node 3 is a normal operating status.

[0050] Step 204: sort the comprehensive priorities of the analysis tasks in the task pool from high to low according to the priorities corresponding to the analysis tasks and the resource application information, prioritize the target computing nodes of the analysis tasks with the highest comprehensive priorities, and use the task management engine to schedule the target computing nodes to execute the analysis tasks with the highest comprehensive priorities.

[0051] For the disclosed embodiment, when delivering analysis tasks based on the task management engine Cromwell, comprehensive sorting can be performed based on the priority of the analysis tasks and the resource application information, and the maximum number of parallel tasks for each computing node can be set separately to avoid resource runs. For long analysis tasks and short analysis tasks that have a large difference in resource consumption and running time, scheduling is performed by monitoring the resource conditions of the computing nodes. Among them, long analysis tasks can be tasks that consume more resources (such as: CPU, memory) and have a longer running time; short analysis tasks can be tasks that consume fewer resources (such as: CPU, memory) and have a shorter running time.

[0052] When the priority of a long analysis task is high, and the target computing node that matches the target computing node of the analysis task has a running task that does not meet the delivery condition temporarily, where the delivery condition is that the hardware configuration of the target computing node meets the condition of the application resource information for running the analysis task, the target computing node will be reserved. If the target computing node meets the delivery condition, the target computing node will be scheduled to execute the analysis task with the highest comprehensive priority; if there are remaining computing nodes, the analysis task will continue to be delivered, and the remaining computing nodes will be used to execute the next analysis task with the highest comprehensive priority. The remaining analysis tasks will continue to wait in the task pool to avoid resource runs when there are many analysis tasks, and runs between analysis tasks that consume more resources and analysis tasks that consume fewer resources, until all analysis tasks in the task pool are executed, to avoid high-priority long analysis tasks blocking the task queue and causing resource waste, so that the resources of the computing nodes are fully utilized, where the remaining computing nodes are computing nodes other than the target computing node among multiple computing nodes.

[0053] When the analysis task scheduling device detects that an analysis task is running, it will periodically monitor the task execution status of the analysis task through the task management engine Cromwell based on the scheduled task scheduling framework Quartz through cron expressions. Users can edit cron expressions through a visual page to adjust the query cycle interval, providing real-time monitoring. When the task execution status is abnormal, the operation log will be collected, and the relevant abnormal information will be fed back to the user through emails, message middleware, etc. for abnormal warning reminders, reducing manual inspection work. When the analysis task scheduling device detects that the analysis task is completed, the task management engine Cromwell is used to organize the result files, collect operation log information, and so on. Among them, the operation log is the log generated during the operation of the analysis task; the result file is the result generated when the analysis task is completed; different analysis processes generate different result files, for example, generating a test report, etc.

[0054] For the disclosed embodiment, the task scheduling solution based on the scheduled task scheduling framework Quart and the task management engine Cromwell enhances the adaptability of the system and improves resource utilization, and provides early warning and timely information feedback functions based on the message middleware.

[0055] Step 205: Based on the task management engine, access the workflow language of the local cluster through a preset task scheduling engine.

[0056] For the disclosed embodiment, since the preset task scheduling engine cannot directly access the workflow language of the local cluster, and the task management engine Cromwell supports the current mainstream cluster task scheduler to access the local cluster and other environments, such as cluster management tools (Sun Grid Engine, SGE), distributed cluster management systems (Load Sharing Facility, LSF), etc., the workflow language (such as WDL) accessing the local cluster does not need to consider the computing environment problem, which simplifies the construction of the analysis environment and has portability, strong adaptability to different distributed task management engines, and can quickly adapt to different user environments. Among them, the preset task scheduling engine may include but is not limited to a local machine, a cloud platform, and a task scheduler.

[0057] Correspondingly, the steps of the implementation example may specifically include: based on the task management engine, accessing the workflow language of the local cluster through the local machine; based on the task management engine, accessing the workflow language of the local cluster through the cloud platform; based on the task management engine, accessing the workflow language of the local cluster through the task scheduler.

[0058] In summary, according to the analysis task scheduling method provided by the present disclosure, compared with the prior art, the present disclosure determines the task parameter information and the application resource information of the analysis task, and verifies the analysis task based on the task parameter information; when the analysis task passes the verification, based on the application resource information, selects the target computing node that matches the analysis task from multiple computing nodes whose running status is a normal running status; and schedules the target computing node to execute the analysis task. By applying the scheme of the present disclosure, multiple computing nodes whose running status is a normal running status are selected, thereby excluding the computing nodes that are disabled, abnormal, and have a high task load, and by selecting the target computing node that matches the analysis task from multiple computing nodes whose running status is a normal running status, the purpose of improving the resource utilization of the computing nodes is achieved.

[0059] Based on the above Figure 1 and Figure 2 The specific implementation of the method shown in this embodiment provides an analysis task scheduling device, such as Figure 4 As shown, the device includes: a verification module 31, a screening module 32, and a scheduling module 33;

[0060] Verification module 31, used to determine task parameter information and application resource information of the analysis task, and perform verification processing on the analysis task based on the task parameter information;

[0061] A screening module 32 is used to screen a target computing node matching the analysis task from among multiple computing nodes in a normal operation state based on the application resource information when the analysis task passes the verification;

[0062] The scheduling module 33 is used to schedule the target computing node to execute the analysis task.

[0063] In specific application scenarios, such as Figure 5 As shown, the device also includes: a determination module 34, a creation module 35;

[0064] A determination module 34, used to determine sample information and analysis process;

[0065] The creation module 35 is used to create an analysis task using sample information and analysis process.

[0066] In a specific application scenario, the verification module 31 is used to determine whether the analysis task is an abnormal and illegal analysis task by analyzing the task parameter information; if so, it is determined that the analysis task has not passed the verification, and an abnormal warning reminder is issued for the analysis task; if not, it is determined that the analysis task has passed the verification.

[0067] In a specific application scenario, the screening module 32 is used to determine multiple initial computing nodes that can be used to execute the analysis task based on the file agent program deployed on the computing node and the application resource information, the initial computing node being a computing node that has the execution authority of the analysis task and the application resource information that meets the analysis task; query the node operation status of each node in the multiple initial computing nodes, the node operation status includes resource usage status, health status and task load status; determine the initial computing nodes among the multiple initial computing nodes whose corresponding node operation status meets the normal operating conditions as the target computing nodes that match the analysis task; the normal operating conditions include: the health status is a normal state; and, the resource usage status is that there are remaining resources that are not used; and, the task load status is that the task load is less than the preset task load.

[0068] In a specific application scenario, the screening module 32 is used to query the node operation status of each node in the multiple initial computing nodes periodically through the cron expression based on the scheduled task scheduling framework.

[0069] In specific application scenarios, such as Figure 5 As shown, the device further includes: an adding module 36;

[0070] The adding module 36 is used to add the verified analysis task to the task pool, and the task pool is used to asynchronously process the analysis task.

[0071] In a specific application scenario, the scheduling module 33 can be used to sort the comprehensive priorities of the analysis tasks in the task pool from high to low according to the priorities corresponding to the analysis tasks and the resource application information; give priority to matching the target computing nodes of the analysis tasks with the highest comprehensive priority, and use the task management engine to schedule the target computing nodes to execute the analysis tasks with the highest comprehensive priority.

[0072] In a specific application scenario, the scheduling module 33 can be used to determine whether the target computing node meets the delivery conditions, where the delivery conditions are that the hardware configuration of the target computing node meets the conditions for applying for resource information for running the analysis task; if the target computing node meets the delivery conditions, the target computing node is scheduled to execute the analysis task with the highest comprehensive priority; if the target computing node does not meet the delivery conditions, the target computing node is reserved, and when the target computing node meets the delivery conditions, the task management engine is used to schedule the target computing node to execute the analysis task with the highest comprehensive priority; the remaining computing nodes are used to execute the next analysis task with the highest comprehensive priority, until all analysis tasks in the task pool are executed, and the remaining computing nodes are computing nodes other than the target computing node among multiple computing nodes.

[0073] In specific application scenarios, such as Figure 5 As shown, the device further includes: an early warning module 37;

[0074] The early warning module 37 is used to monitor the task execution status of the analysis task, and when the task execution status is abnormal, issue an abnormal early warning reminder for the analysis task.

[0075] In a specific application scenario, the early warning module 37 can be used to monitor and analyze the task execution status of the task periodically through the cron expression based on the scheduled task scheduling framework.

[0076] In specific application scenarios, such as Figure 5 As shown, the device further includes: an access module 38;

[0077] The access module 38 is used to access the workflow language of the local cluster through a preset task scheduling engine based on the task management engine.

[0078] In a specific application scenario, the access module 38 can be used to access the workflow language of the local cluster through the local machine based on the task management engine; access the workflow language of the local cluster through the cloud platform based on the task management engine; access the workflow language of the local cluster through the task scheduler based on the task management engine.

[0079] It should be noted that for other corresponding descriptions of the functional units involved in the analysis task scheduling device provided in this embodiment, reference can be made to Figure 1 and Figure 2 The corresponding description in will not be repeated here.

[0080] Based on the above Figure 1 and Figure 2 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figure 1 and Figure 2 The method shown.

[0081] Based on this understanding, the technical solution of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various implementation scenarios of the present disclosure.

[0082] Based on the above Figure 1 and Figure 2 The method shown, and Figure 4 and Figure 5In order to achieve the above-mentioned purpose, the embodiment of the present disclosure also provides an electronic device that can be configured on the vehicle (such as an electric vehicle) side, and the device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 2 The method shown.

[0083] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0084] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0085] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present disclosure can be implemented by means of software plus necessary general hardware platforms, or by hardware. Compared with the prior art, the technical solution in the present disclosure determines the task parameter information and application resource information of the analysis task, and verifies the analysis task based on the task parameter information; when the analysis task passes the verification, based on the application resource information, the target computing node matching the analysis task is screened from multiple computing nodes whose running state is a normal operating state; and the target computing node is scheduled to perform the analysis task. By applying the solution of the present disclosure, multiple computing nodes whose running state is a normal operating state are selected, thereby excluding disabled, abnormal, and high-task-load computing nodes, and by screening the target computing nodes matching the analysis task from multiple computing nodes whose running state is a normal operating state, the purpose of improving the resource utilization of the computing nodes is achieved.

[0087] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0088] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. A method for scheduling analysis tasks, characterized in that: The method comprises: Determine task parameter information and application resource information of the analysis task, and perform verification processing on the analysis task based on the task parameter information; In the case where the analysis task passes the verification, based on the application resource information, selecting a target computing node matching the analysis task from among a plurality of computing nodes in a normal operation state; The target computing node is scheduled to execute the analysis task.

2. The method according to claim 1, characterized in that The method further comprises: Determine sample information and analysis procedures; The analysis task is created using the sample information and the analysis process.

3. The method according to claim 1, characterized in that The verifying process of the analysis task based on the task parameter information includes: By analyzing the task parameter information, determining whether the analysis task is an abnormal and illegal analysis task; If yes, it is determined that the analysis task has not passed the verification, and an abnormal warning reminder is issued for the analysis task; If not, it is determined that the analysis task has passed the verification.

4. The method according to claim 1, characterized in that: When the analysis task passes the verification, based on the application resource information, selecting a target computing node matching the analysis task from a plurality of computing nodes in a normal operation state includes: Determine, according to the file agent program deployed on the computing node and the resource application information, a plurality of initial computing nodes that can be used to execute the analysis task, wherein the initial computing nodes are computing nodes that have execution authority for the analysis task and meet the resource application information for the analysis task; Querying a node operation status of each of the multiple initial computing nodes, wherein the node operation status includes a resource usage status, a health status, and a task load status; Determine, among the multiple initial computing nodes, an initial computing node corresponding to the node running state that meets the normal operating condition as a target computing node matching the analysis task; The normal operating conditions include: the health state is a normal state; and, The resource usage status is that there are remaining resources that are not used; and, The task load state is that the task load is less than a preset task load.

5. The method according to claim 4, characterized in that The querying of the node operation status of each node in the plurality of initial computing nodes includes: Based on the scheduled task scheduling framework, the task management engine periodically queries the node running status of each node in multiple initial computing nodes through cron expressions.

6. The method according to claim 3, characterized in that In the case where the analysis task passes the verification, the method further comprises: Adding the verified analysis task to a task pool, wherein the task pool is used to asynchronously process the analysis task; The scheduling the target computing node to execute the analysis task includes: According to the priorities corresponding to the analysis tasks and the resource application information, the comprehensive priorities of the analysis tasks in the task pool are sorted in descending order; The target computing node of the analysis task with the highest comprehensive priority is matched first, and the target computing node is scheduled to execute the analysis task with the highest comprehensive priority by using the task management engine.

7. The method according to claim 6, characterized in that The using the task management engine to schedule the target computing node to execute the analysis task with the highest comprehensive priority includes: Determine whether the target computing node meets the delivery condition, where the delivery condition is that the hardware configuration of the target computing node meets the condition of applying for resource information for running the analysis task; If the target computing node meets the delivery condition, scheduling the target computing node to execute the analysis task with the highest comprehensive priority; If the target computing node does not meet the delivery condition, the target computing node is reserved, and when the target computing node meets the delivery condition, the target computing node is scheduled by the task management engine to execute the analysis task with the highest comprehensive priority; The remaining computing nodes are used to execute the next analysis task with the highest comprehensive priority until all analysis tasks in the task pool are executed. The remaining computing nodes are computing nodes among the multiple computing nodes except the target computing node.

8. The method according to claim 6, characterized in that After scheduling the target computing node to execute the analysis task, the method further includes: The task execution status of the analysis task is monitored, and when the task execution status is abnormal, an abnormal warning reminder is issued to the analysis task.

9. The method according to claim 8, characterized in that The monitoring of the task execution status of the analysis task includes: Based on the scheduled task scheduling framework, the task management engine periodically monitors the task execution status of the analysis task through cron expressions.

10. The method according to any one of claims 5, 6, 7 or 9, characterized in that: The method further comprises: Based on the task management engine, a workflow language of a local cluster is accessed through a preset task scheduling engine.

11. The method according to claim 8, characterized in that The preset task scheduling engine includes a local machine, and the workflow language based on the task management engine and accessing the local cluster through the preset task scheduling engine includes: Based on the task management engine, a workflow language of a local cluster is accessed through the local machine.

12. The method according to claim 8, characterized in that The preset task scheduling engine includes a cloud platform, and the workflow language based on the task management engine and accessing the local cluster through the preset task scheduling engine includes: Based on the task management engine, a workflow language of a local cluster is accessed through the cloud platform.

13. The method according to claim 8, characterized in that The preset task scheduling engine includes a task scheduler, and the workflow language of the local cluster accessed through the preset task scheduling engine based on the task management engine includes: Based on the task management engine, the workflow language of the local cluster is accessed through the task scheduler.

14. An analysis task scheduling device, characterized in that: include: A verification module, used to determine task parameter information and application resource information of the analysis task, and to perform verification processing on the analysis task based on the task parameter information; A screening module, configured to screen a target computing node matching the analysis task from among a plurality of computing nodes in a normal operating state based on the application resource information if the analysis task passes verification; A scheduling module is used to schedule the target computing node to execute the analysis task.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

17. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images