Process task analysis method and device, equipment and medium

Abstract

The invention discloses a process task analysis method and device, equipment and a medium, and relates to the technical field of raw message analysis, and the method comprises the steps: obtaining a process execution request sent by a preset process script, and generating a corresponding target process task based on a target process file in the process execution request and a corresponding target execution parameter; splitting the target process task into a plurality of target jobs based on a preset splitting rule; the target resource quantity required by the target job is recorded in the target job; a successive dependency relationship exists between the target jobs; and sending the plurality of target jobs to a target cluster according to the successive dependency relationship, so that the target cluster respectively allocates corresponding computer resources to the plurality of target jobs based on the target resource quantity, and executes parallel analysis operation on the target jobs by using the computer resources. In this way, the time consumed for carrying out the raw signal analysis can be shortened.

Description

Technical Field

[0001] The present invention relates to the field of bioinformatics analysis technology, and in particular to a process task analysis method, device, equipment and medium. Background Art

[0002] Bioinformatics analysis is the process of processing and interpreting biological data, which usually involves large-scale data processing, statistical analysis and the application of bioinformatics methods. With the increasing demand for molecular diagnostic services in hospitals, the demand for bioinformatics analysis is also increasing day by day. At the same time, higher requirements are placed on the accuracy and timeliness of bioinformatics analysis.

[0003] At present, the Bioinformatics Analysis Institute mainly uses python (a computer programming language) to call sequencing alignment, variant detection, transcriptome analysis and other toolkits, and uses CPU resources to process and calculate biological data. In addition, the execution order between each step is set by using Shell. For example, it takes 4 hours to perform bioinformatics analysis on a single sample of WES 5G (Whole-Exome Sequencing 5G, that is, whole exome sequencing with a size of 5GB), and it takes 9 hours to perform bioinformatics analysis on a single sample of WGS 10G (Whole-Genome Sequencing 10G, that is, whole genome sequencing with a size of 10GB). These data are very long for both experimenters and patients.

[0004] From the above, it can be seen that how to reduce the time spent on bioinformatics analysis is an urgent problem to be solved. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a process task analysis method, device, equipment and medium, which can split the target process task, allocate corresponding computer resources to each target job obtained after the split, and use the corresponding computer resources to analyze each target job. Thereby, the time spent on process task analysis is reduced during the process of process task analysis, the computing efficiency is improved, and the user experience is enhanced. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a process task analysis method, which is applied to a preset scheduling framework, including:

[0007] Obtain a process execution request sent by a preset process script, and generate a corresponding target process task based on a target process file and corresponding target execution parameters in the process execution request;

[0008] The target process task is split into several target jobs based on a preset splitting rule; the target job records the target resource quantity required by the target job; and there is a sequential dependency relationship between the target jobs;

[0009] The target jobs are sent to a target cluster according to the sequential dependency relationship, so that the target cluster allocates corresponding computer resources to the target jobs based on the target resource amount, and uses the computer resources to perform parallel analysis operations on the target jobs.

[0010] Optionally, before sending the plurality of target jobs to the target cluster according to the sequential dependency relationship, the method further includes:

[0011] Deploy a plurality of target nodes corresponding to the target cluster into a plurality of target containers respectively by using a preset containerization technology;

[0012] A plurality of the target containers are started in parallel, so that the plurality of the target containers use a plurality of the target nodes to analyze a plurality of the target jobs.

[0013] Optionally, obtaining a process execution request sent by a preset process script, and generating a corresponding target process task based on a target process file and corresponding target execution parameters in the process execution request, includes:

[0014] Obtaining a process execution request sent by a preset process script through a preset request interface, and generating a corresponding target process task based on a target process file and corresponding target execution parameters in the process execution request;

[0015] Among them, the target process file in the process execution request is a process file generated by a preset file generation script based on all steps corresponding to the bioinformatics analysis process.

[0016] Optionally, before splitting the target process task into a plurality of target jobs based on a preset splitting rule, the method further includes:

[0017] Counting the number of files of the target process files in the process execution request obtained at the same time, and determining whether the number of files is greater than one;

[0018] If the number of the files is greater than one, the total amount of hardware resources is split to obtain a number of split resources, and the number of split resources are respectively allocated to the corresponding target process files.

[0019] Optionally, after counting the number of files of the target process files in the process execution request obtained at the same time, the method further includes:

[0020] Determine whether the number of files is greater than a preset file number threshold;

[0021] If the number of files is not greater than the preset file number threshold, allocating the idle resources in the total hardware resources to a number of the target process files;

[0022] If the number of files is greater than the preset file number threshold, a target number of target process files in all the target process files are submitted to the tail of a preset waiting queue, so that when there are idle resources, the target process files are extracted from the head of the preset waiting queue for analysis; wherein, the file acquisition time of the target number of target process files is later than that of other target process files in all the target process files.

[0023] Optionally, after performing the parallel analysis operation on the target job using the computer resources, the method further includes:

[0024] Counting the time required for analyzing each target job to obtain the analysis time corresponding to each target job;

[0025] It is determined whether the analysis duration corresponding to each target job is greater than a preset duration threshold. If so, the execution status corresponding to the target job is determined as execution failure, and the target job is recycled.

[0026] Optionally, after performing the parallel analysis operation on the target job using the computer resources, the method further includes:

[0027] Monitoring the execution status of each target job to obtain execution results corresponding to each target job;

[0028] If the execution result is successful, executing the target job that has a dependency relationship with the target job corresponding to the execution result and has not been executed;

[0029] If the execution result is execution failure, the target job corresponding to the execution result is recycled.

[0030] In a second aspect, the present application provides a process task analysis device, which is applied to a preset scheduling framework, including:

[0031] A request acquisition module, used to acquire a process execution request sent by a preset process script, and generate a corresponding target process task based on a target process file and corresponding target execution parameters in the process execution request;

[0032] A task splitting module is used to split the target process task into several target jobs based on preset splitting rules; the target job records the target resource quantity required by itself; and there is a sequential dependency relationship between the target jobs;

[0033] The job sending module is used to send the target jobs to the target cluster according to the sequential dependency relationship, so that the target cluster allocates corresponding computer resources to the target jobs based on the target resource amount, and uses the computer resources to perform parallel analysis operations on the target jobs.

[0034] In a third aspect, the present application provides an electronic device, including:

[0035] Memory, used to store computer programs;

[0036] The processor is used to execute the computer program to implement the aforementioned process task analysis method.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned process task analysis method when executed by a processor.

[0038] As can be seen from the above, before the process task analysis is performed, the present application needs to obtain the process execution request sent by the preset process script, and generate the corresponding target process task based on the target process file in the process execution request and the corresponding target execution parameter; split the target process task into several target jobs based on the preset splitting rule; the target job includes the target resource amount; there is a sequential dependency relationship between each of the target jobs; according to the sequential dependency relationship, several of the target jobs are sent to the target cluster, so that the target cluster allocates corresponding computer resources to several of the target jobs based on the target resource amount, and uses the computer resources to perform parallel analysis operations on the target jobs. It can be seen that the present application splits the target process task, allocates corresponding computer resources to each target job obtained after the split, and uses the corresponding computer resources to analyze each target job. In this way, the time spent on process task analysis can be reduced during the process task analysis process, which improves the computing efficiency and the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0040] Figure 1 A flow chart of a process task analysis method disclosed in this application;

[0041] Figure 2 This is a schematic diagram of the types of genetic samples;

[0042] Figure 3 A schematic diagram of a mirror type;

[0043] Figure 4 A schematic diagram of the process of splitting and merging target process tasks in the bioinformatics process based on chromosomes;

[0044] Figure 5 This is a structural flow chart of a process task analysis device disclosed in this application;

[0045] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Bioinformatics analysis is the process of processing and interpreting biological data, usually involving large-scale data processing, statistical analysis, and the application of bioinformatics methods. With the increasing demand for molecular diagnostic services in the clinical end of hospitals, the demand for bioinformatics analysis is also increasing day by day. At the same time, higher requirements are placed on the accuracy and timeliness of bioinformatics analysis. To this end, the present application provides a process task analysis method that can reduce the time spent on process task analysis during the analysis of process tasks, improve computing efficiency, and enhance user experience.

[0048] See also Figure 1 As shown, an embodiment of the present invention discloses a process task analysis method, which is applied to a preset scheduling framework, including:

[0049] Step S11: Obtain a process execution request sent by a preset process script, and generate a corresponding target process task based on a target process file and corresponding target execution parameters in the process execution request.

[0050] In this embodiment, after obtaining the process execution request sent by the preset process script, it is necessary to generate the corresponding target process task based on the target process file and the corresponding target execution parameters included in the process execution request. Among them, the format of the target process file can be a wdl file (i.e., a bioinformatics file format); it is worth mentioning that the format of the target process file can be set based on user needs, and it is not specifically limited here. Specifically, the process execution request sent by the preset process script is obtained, and the corresponding target process task is generated based on the target process file and the corresponding target execution parameters in the process execution request, which may include: obtaining the process execution request sent by the preset process script through the preset request interface, and generating the corresponding target process task based on the target process file and the corresponding target execution parameters in the process execution request; wherein, the target process file in the process execution request is a process file generated by the preset file generation script based on all the steps corresponding to the bioinformatics analysis process, and each step in the bioinformatics analysis process can be defined based on user needs.

[0051] In a specific implementation, the preset request interface is an HTTP (Hypertext Transfer Protocol) interface. It is worth mentioning that any interface that can be integrated in a preset scheduling framework and receive a process execution request sent by a preset process script is applicable to this embodiment and is not specifically limited here.

[0052] It is worth mentioning that the current bioinformatics analysis process takes a long time to calculate and cannot meet the customer's demand for efficient output of results. For this reason, in one embodiment, the current bioinformatics analysis process attempts to use commercial paid software and GPU (Graphics Processing Unit, i.e., graphics processor) hardware acceleration, but while increasing the speed of bioinformatics analysis, it increases costs. In another specific embodiment, in the bioinformatics analysis process, various bioinformatics analysis tools are scripted through shell (i.e., a programming language), but parallel analysis of each module cannot be achieved, resulting in low analysis efficiency and failure to ensure that server resources are fully utilized. Therefore, if big data technology is introduced to perform parallel calculations on files during bioinformatics analysis, the calculation process can be accelerated, so that customers can get the report obtained after analysis as soon as possible. As life science solutions based on Spark (a computing engine) continue to mature and improve, more and more tools and workflows can run on the Spark computing platform. Because the Spark computing platform has fault tolerance, its scalability is also constantly improving. In addition, parallel life science applications will shorten the sequence analysis time from the current more than ten hours to less than one hour. At the same time, by utilizing distributed job scheduling and low-cost clusters, the analysis cost is greatly reduced.

[0053] In a specific implementation, the preset scheduling framework is a bioinformatics process scheduling framework based on Cromwell (i.e., an open source workflow management system), and the bioinformatics process scheduling framework built on Cromwell is responsible for parsing the target process file, splitting and scheduling the target process tasks, controlling computer resources, and managing data, and supports parallel processing of multiple gene samples and multiple gene samples; that is, the bioinformatics process scheduling framework built on Cromwell can improve the speed of processing the target process file by rationally utilizing server resources. Among them, the types of gene samples can be as follows: Figure 2 As shown in the figure, the types of bioinformatics analysis data are WES (Whole-Exome Sequencing, i.e. whole exome sequencing), WGS (Whole Genome Sequencing, i.e. whole genome sequencing), 69 genes, 9 genes, and 620 genes.

[0054] Step S12: split the target process task into a number of target jobs based on a preset splitting rule; the target job records the target resource quantity required by the target job; and there is a sequential dependency relationship between the target jobs.

[0055] In a specific implementation, the preset splitting rule is to split the target process task based on chromosomes to obtain several target jobs, and each target job corresponds to a thread, and one thread processes the data corresponding to a chromosome, thereby realizing multiple threads to execute the calculation process of each chromosome data in parallel, so as to reduce the amount of data processed by each thread in the bioinformatics process, and realize the data splitting and parallel accelerated calculation.

[0056] In this embodiment, the embodiment of the present application realizes parallel accelerated calculation of multiple samples by submitting multiple tasks at the same time. When more than one bioinformatics process file needs to be analyzed at the same time, the preset scheduling framework will allocate computing resources to the target process task according to the hardware resource situation. Specifically, before splitting the target process task into several target jobs based on the preset splitting rules, it can also include: counting the number of files of several target process files in the process execution request obtained at the same time, and judging whether the number of files is greater than one; if the number of files is greater than one, the total amount of hardware resources is split to obtain several split resources, and the several split resources are respectively allocated to the corresponding target process files.

[0057] It is worth mentioning that when the number of target process files that need to be analyzed is less than the upper limit, the available resources in the system can be allocated to the target process tasks corresponding to the new target process files. When the number of target process files that need to be analyzed reaches or exceeds the upper limit, the process tasks corresponding to the new target process files are controlled to enter the preset waiting queue. Specifically, after counting the number of files of several target process files in the process execution request obtained at the same time, it can also include: judging whether the number of files is greater than the preset file number threshold; if the number of files is not greater than the preset file number threshold, the idle resources in the total hardware resources are allocated to several target process files; if the number of files is greater than the preset file number threshold, the target number of target process files in all the target process files are submitted to the tail of the preset waiting queue, so that when there are idle resources, the target process files are extracted from the head of the preset waiting queue for analysis; wherein, the file acquisition time of the target number of target process files is later than that of other target process files in all the target process files. It is worth mentioning that the target number is the difference between the number of all target process files and the preset file number threshold.

[0058] Step S13: sending the target jobs to a target cluster according to the sequential dependency relationship, so that the target cluster allocates corresponding computer resources to the target jobs based on the target resource amount, and uses the computer resources to perform parallel analysis operations on the target jobs.

[0059] In this embodiment, the embodiment of the present application realizes parallel computing of data by building a spark cluster and executing the variant detection and genotyping steps in the analysis process on the spark cluster. It is worth mentioning that the entire spark cluster runs in multiple docker (i.e., an open source application container engine) containers. Specifically, before sending the target jobs to the target cluster according to the sequential dependency, it can also include: using the preset containerization technology to deploy the target nodes corresponding to the target cluster to the target containers respectively; starting the target containers in parallel so that the target containers can use the target nodes to analyze the target jobs in parallel.

[0060] It is worth mentioning that after generating the target process task, the preset scheduling framework converts the bioinformatics process file into a target job that can be used for batch calculation. Each target job is a logical unit for independent analysis, and each target job contains the docker image address corresponding to the execution task. Among them, the embodiment of the present application can realize the intercommunication and sharing of multiple containers of the local machine through the path mapping of the docker container.

[0061] The embodiment of the present application builds corresponding tool image packages based on different bioinformatics analysis tools, and dynamically configures and uses computer resources to maximize the analysis performance of the target job under limited server resources. Figure 3 As shown in the figure, the current verification images can be divided into 6 categories, namely fastq (a text format that stores biological sequences and corresponding quality evaluations), BWA (a bioinformatics tool), GATK (Genome Analysis Toolkit), SAMtools (a bioinformatics tool for processing SAM / BAM files), ANNOVAR (a gene annotation software) and other (other defined Docker containers). Containers can be started when in use, and automatically release the occupied computer resources when not in use, thereby making full use of limited computer resources. In addition, when computer resources are sufficient, multiple containers can be started in parallel for parallel computing of target jobs, that is, the computer resources are divided into several reusable small blocks. When the computer resources meet the requirements, the corresponding resource blocks can be started and participate in the operation of the target job.

[0062] In this embodiment, when a new target process task starts, the preset scheduling framework will submit the target job to the spark cluster according to the sequential dependency relationship between each target job, so as to use the computer resources in the spark cluster for parallel analysis and monitor the running status of each target job. Among them, the running status of the target job includes execution success and execution failure; when the execution is successful, the downstream task of the target job will be executed; when the analysis time of the job exceeds the preset time upper limit, it is considered that the target job has failed to execute, the target job is judged as dead, and the dead target job is recycled. Specifically, after using the computer resources to perform parallel analysis operations on the target job, it can also include: monitoring the execution status of each target job to obtain the execution results corresponding to each target job; if the execution result is successful, the target job corresponding to the execution result has a dependency relationship and has not been executed and is executed; if the execution result is failed, the target job corresponding to the execution result is recycled.

[0063] It is worth mentioning that the basis for judging whether the execution of the target job fails is whether the time required for analyzing the target job is greater than a preset time threshold; specifically, after using the computer resources to perform parallel analysis operations on the target job, it may also include: counting the time required for analyzing each of the target jobs to obtain the analysis time corresponding to each of the target jobs; judging whether the analysis time corresponding to each of the target jobs is greater than a preset time threshold, if so, determining the execution status corresponding to the target job as execution failure, and recycling the target job.

[0064] As can be seen from the above, before the process task analysis is performed, the present application needs to obtain the process execution request sent by the preset process script, and generate the corresponding target process task based on the target process file in the process execution request and the corresponding target execution parameter; split the target process task into several target jobs based on the preset splitting rule; the target job includes the target resource amount; there is a sequential dependency relationship between each of the target jobs; according to the sequential dependency relationship, several of the target jobs are sent to the target cluster, so that the target cluster allocates corresponding computer resources to several of the target jobs based on the target resource amount, and uses the computer resources to perform parallel analysis operations on the target jobs. It can be seen that the present application splits the target process task, allocates corresponding computer resources to each target job obtained after the split, and uses the corresponding computer resources to analyze each target job. In this way, the time spent on process task analysis can be reduced during the process task analysis process, which improves the computing efficiency and the user experience.

[0065] For further information, see Figure 4 As shown, an embodiment of the present invention discloses a process task analysis method, including:

[0066] In this embodiment, the process of splitting and merging the target process tasks in the bioinformatics process based on chromosomes is as follows: Figure 4As shown. In a specific embodiment, by distributing the key steps in the analysis process, that is, by splitting the target process file in the format of the bioinformatics process based on chromosomes, multiple chromosome sequences are obtained, and the multiple chromosome sequences are sorted from small to large by serial number, so as to create an index file based on the sorted chromosome sequence. Then, the index file is further split to obtain the product of the number of quality control file splits and the number of chromosomes, files; wherein the number of quality control file splits is a number customized based on user needs, and the number of chromosomes is 23 pairs. Subsequently, the files obtained after further splitting are respectively assigned to multiple processors or computers for simultaneous execution, which can significantly reduce the cost required for analysis and shorten the running time on the basis of ensuring that the accuracy of the analysis results is consistent with GATK (Genome Analysis Toolkit, a widely used genomic data analysis tool). In a specific embodiment, for double-end 5G samples, the processing performance of the whole exome process can be improved by about 5 times by splitting the chromosomes.

[0067] It is worth mentioning that although the split files can be aligned and sequenced normally, in order to integrate the genetic analysis into the reference genome, these split files need to be merged. That is, after analyzing the files based on chromosome splitting, all files need to be merged; in a specific embodiment, several files are merged in units of 24, wherein each merged file corresponds to a chromosome. Subsequently, each merged file is PCR (Polymerase Chain Reaction), and then a comparison index file is created based on the marked file to perform parallel analysis of the comparison index file. Among them, the file merging process is a complex process involving the use of multiple steps and tools. For example, the splitting may limit the length of the file, thereby causing some problems, and the purpose of merging is to solve the above problems and ensure the accuracy of subsequent genetic analysis.

[0068] In this embodiment, since the whole genome data contains the information of the entire genome, the amount of data is usually large. Therefore, when processing the whole genome data, the fastq file corresponding to the whole genome data can be split by line, and the bam file corresponding to the whole genome data (i.e., a file storing the sequencing data after comparison) can be split by chromosome grouping. Doing so can reduce the size of a single file and facilitate processing. However, the whole exome data only contains the exon region in the genome, and the amount of data is relatively small. Therefore, when processing the whole exome data, the method of splitting the fastq file corresponding to the whole exome data by line, but not splitting the bam file corresponding to the whole exome data can be adopted. It can be understood that according to which preset splitting rule the target process task is split, it depends on the data amount of the target process file in the target process task.

[0069] As can be seen from the above, before the process task analysis is performed, the present application needs to obtain the process execution request sent by the preset process script, and generate the corresponding target process task based on the target process file in the process execution request and the corresponding target execution parameter; split the target process task into several target jobs based on the preset splitting rule; the target job includes the target resource amount; there is a sequential dependency relationship between each of the target jobs; according to the sequential dependency relationship, several of the target jobs are sent to the target cluster, so that the target cluster allocates corresponding computer resources to several of the target jobs based on the target resource amount, and uses the computer resources to perform parallel analysis operations on the target jobs. It can be seen that the present application splits the target process task, allocates corresponding computer resources to each target job obtained after the split, and uses the corresponding computer resources to analyze each target job. In this way, the problem of slow calculation of bioinformatics analysis can be solved, and the participation of GPU is not required, and commercial software does not need to be purchased, so as to achieve the purpose of reducing analysis costs, and then solve the problems of low execution efficiency and slow analysis speed of traditional methods, and large-scale bioinformatics process analysis can be realized.

[0070] Accordingly, see Figure 5 As shown, the present application also provides a process task analysis device, including:

[0071] The request acquisition module 11 is used to acquire the process execution request sent by the preset process script, and generate the corresponding target process task based on the target process file and the corresponding target execution parameters in the process execution request;

[0072] The task splitting module 12 is used to split the target process task into a plurality of target jobs based on a preset splitting rule; the target job records the target resource quantity required by itself; and there is a sequential dependency relationship between the target jobs;

[0073] The job sending module 13 is used to send the target jobs to the target cluster according to the sequential dependency relationship, so that the target cluster allocates corresponding computer resources to the target jobs based on the target resource amount, and uses the computer resources to perform parallel analysis operations on the target jobs.

[0074] As can be seen from the above, before the process task analysis is performed, the present application needs to obtain the process execution request sent by the preset process script, and generate the corresponding target process task based on the target process file in the process execution request and the corresponding target execution parameter; split the target process task into several target jobs based on the preset splitting rule; the target job includes the target resource amount; there is a sequential dependency relationship between each of the target jobs; according to the sequential dependency relationship, several of the target jobs are sent to the target cluster, so that the target cluster allocates corresponding computer resources to several of the target jobs based on the target resource amount, and uses the computer resources to perform parallel analysis operations on the target jobs. It can be seen that the present application splits the target process task, allocates corresponding computer resources to each target job obtained after the split, and uses the corresponding computer resources to analyze each target job. In this way, the time spent on process task analysis can be reduced during the process task analysis process, which improves the computing efficiency and the user experience.

[0075] In some specific embodiments, the process task analysis device further includes:

[0076] A node deployment unit, used to deploy a plurality of target nodes corresponding to the target cluster into a plurality of target containers respectively by using a preset containerization technology;

[0077] The container starting unit is used to start the plurality of target containers in parallel, so that the plurality of target containers use the plurality of target nodes to analyze the plurality of target jobs.

[0078] In some specific implementations, the request acquisition module 11 may specifically include:

[0079] A request acquisition unit is used to obtain a process execution request sent by a preset process script through a preset request interface, and generate a corresponding target process task based on a target process file in the process execution request and corresponding target execution parameters; wherein the target process file in the process execution request is a process file generated by a preset file generation script based on all steps corresponding to the bioinformatics analysis process.

[0080] In some specific embodiments, the process task analysis device further includes:

[0081] a quantity counting unit, used to count the number of files of the target process files in the process execution request obtained at the same time, and determine whether the number of files is greater than one;

[0082] The resource splitting unit is used to split the total hardware resource amount to obtain a number of split resources if the number of the files is greater than one, and respectively allocate the number of split resources to the corresponding target process files.

[0083] Accordingly, in some specific embodiments, the process task analysis device further includes:

[0084] A quantity determination unit, used to determine whether the number of files is greater than a preset file quantity threshold;

[0085] A resource allocation unit, configured to allocate idle resources in the total hardware resources to a number of the target process files if the number of the files is not greater than the preset file number threshold;

[0086] A file submission unit is used to submit a target number of target process files among all the target process files to the tail of a preset waiting queue if the number of files is greater than the preset file number threshold, so as to extract the target process files from the head of the preset waiting queue for analysis when there are idle resources; wherein the file acquisition time of the target number of target process files is later than that of other target process files among all the target process files.

[0087] In some specific embodiments, the process task analysis device further includes:

[0088] A duration statistics unit, used to count the duration required for analyzing each target job, so as to obtain the analysis duration corresponding to each target job;

[0089] The duration judgment unit is used to judge whether the analysis duration corresponding to each of the target jobs is greater than a preset duration threshold. If so, the execution status corresponding to the target job is determined as execution failure, and the target job is recycled.

[0090] In some specific embodiments, the process task analysis device further includes:

[0091] A status monitoring unit, used to monitor the execution status of each target job to obtain the execution results corresponding to each target job;

[0092] a job execution unit, configured to execute the target job that has a dependency relationship with the target job corresponding to the execution result and has not been executed if the execution result is successful;

[0093] The job recovery unit is used to recover the target job corresponding to the execution result if the execution result is execution failure.

[0094] Furthermore, the present application also discloses an electronic device. Figure 6 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input and output interface 25 and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the process task analysis method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment can specifically be an electronic computer.

[0095] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0096] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0097] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the process task analysis method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0098] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed process task analysis method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0099] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0100] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0101] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0102] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0103] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A process task analysis method, characterized in that: Applied to preset scheduling framework, including: Obtain a process execution request sent by a preset process script, and generate a corresponding target process task based on a target process file and corresponding target execution parameters in the process execution request; The target process task is split into several target jobs based on a preset splitting rule; the target job records the target resource quantity required by the target job; and there is a sequential dependency relationship between the target jobs; The target jobs are sent to a target cluster according to the sequential dependency relationship, so that the target cluster allocates corresponding computer resources to the target jobs based on the target resource amount, and uses the computer resources to perform parallel analysis operations on the target jobs.

2. The process task analysis method according to claim 1, characterized in that: Before sending the plurality of target jobs to the target cluster according to the sequential dependency relationship, the method further includes: Deploy a plurality of target nodes corresponding to the target cluster into a plurality of target containers respectively by using a preset containerization technology; A plurality of the target containers are started in parallel, so that the plurality of the target containers use a plurality of the target nodes to analyze a plurality of the target jobs.

3. The process task analysis method according to claim 1, characterized in that: The obtaining of a process execution request sent by a preset process script, and generating a corresponding target process task based on a target process file and corresponding target execution parameters in the process execution request, includes: Obtaining a process execution request sent by a preset process script through a preset request interface, and generating a corresponding target process task based on a target process file and corresponding target execution parameters in the process execution request; Among them, the target process file in the process execution request is a process file generated by a preset file generation script based on all steps corresponding to the bioinformatics analysis process.

4. The process task analysis method according to claim 1, characterized in that: Before splitting the target process task into a plurality of target jobs based on a preset splitting rule, the method further includes: Counting the number of files of the target process files in the process execution request obtained at the same time, and determining whether the number of files is greater than one; If the number of the files is greater than one, the total amount of hardware resources is split to obtain a number of split resources, and the number of split resources are respectively allocated to the corresponding target process files.

5. The process task analysis method according to claim 4, characterized in that: After counting the number of files of the target process files in the process execution request obtained at the same time, the method further includes: Determine whether the number of files is greater than a preset file number threshold; If the number of files is not greater than the preset file number threshold, allocating the idle resources in the total hardware resources to a number of the target process files; If the number of files is greater than the preset file number threshold, a target number of target process files in all the target process files are submitted to the tail of a preset waiting queue, so that when there are idle resources, the target process files are extracted from the head of the preset waiting queue for analysis; wherein, the file acquisition time of the target number of target process files is later than that of other target process files in all the target process files.

6. The process task analysis method according to claim 1, characterized in that: After performing the parallel analysis operation on the target job using the computer resources, the method further includes: Counting the time required for analyzing each target job to obtain the analysis time corresponding to each target job; It is determined whether the analysis duration corresponding to each target job is greater than a preset duration threshold. If so, the execution status corresponding to the target job is determined as execution failure, and the target job is recycled.

7. The process task analysis method according to any one of claims 1 to 6, characterized in that: After performing the parallel analysis operation on the target job using the computer resources, the method further includes: Monitoring the execution status of each target job to obtain execution results corresponding to each target job; If the execution result is successful, executing the target job that has a dependency relationship with the target job corresponding to the execution result and has not been executed; If the execution result is execution failure, the target job corresponding to the execution result is recycled.

8. A process task analysis device, characterized in that: Applied to preset scheduling framework, including: A request acquisition module, used to acquire a process execution request sent by a preset process script, and generate a corresponding target process task based on a target process file and corresponding target execution parameters in the process execution request; A task splitting module is used to split the target process task into several target jobs based on preset splitting rules; the target job records the target resource quantity required by itself; and there is a sequential dependency relationship between the target jobs; The job sending module is used to send the target jobs to the target cluster according to the sequential dependency relationship, so that the target cluster allocates corresponding computer resources to the target jobs based on the target resource amount, and uses the computer resources to perform parallel analysis operations on the target jobs.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the process task analysis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the steps of the process task analysis method according to any one of claims 1 to 7 are implemented.

Images