Processing method, device and equipment for metagenome analysis task exception

Abstract

The invention belongs to the technical field of computer medical detection, solves the problems of computing resource waste and analysis period prolonging during processing of error tasks in existing metagenome analysis, and provides a processing method, device and equipment for task exception of metagenome analysis. Comprising the steps that a metagenome analysis task is divided into a plurality of modules to be executed according to task types, after each subtask is completed, a target file representing the execution result of the subtask is output, the total number of samples participating in the subtask and the number of the target files of the task execution result are compared, and first execution result information of task execution failure is determined; and according to the log of each task log corresponding to the first execution result information, determining an execution scheme of the second sub-task, including terminating the task or warning and continuing execution, according to the log of each task log corresponding to the first execution result information. And multi-sample parallel computing resource waste and analysis period prolonging can be avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer medical detection technology, and in particular to a method, device and equipment for processing abnormalities in a metagenomic analysis task. Background Art

[0002] The rapid development of high-throughput sequencing technology has promoted the in-depth study of metagenomics, but it has also brought about a sharp increase in data volume and the complexity of analysis tasks. In order to manage these large-scale data analysis tasks, domain-specific languages ​​(DSLs) such as Nextflow, Snakemake, and WDL are widely used. Although these DSLs are powerful and can automate task allocation, scheduling, and error recovery, they still have limitations in dealing with complex errors in metagenomic analysis.

[0003] During metagenomic analysis, due to the large scale and diverse characteristics of data, abnormal situations such as insufficient computing resources, task timeouts, platform anomalies, and data quality issues often occur, resulting in interruptions to the analysis process. Although existing technologies support automatic retry of errors, they lack effective solutions for errors that cannot be automatically repaired. In addition, once an error occurs in a task with large resource requirements and long execution time, it not only affects the current task, but also interferes with the operation of other parallel tasks. When the user restarts the process based on DSL, the previously interrupted task needs to be re-executed, resulting in a waste of computing resources and a prolonged analysis cycle. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a method, device and equipment for processing abnormalities in metagenomic analysis tasks, so as to solve the problems of waste of computing resources and prolonged analysis cycle in the methods used in the prior art for processing erroneous tasks in metagenomic analysis.

[0005] The technical solution adopted by the present invention is:

[0006] In one aspect, the present invention provides a method for processing an abnormality in a metagenomic analysis task, the method comprising:

[0007] In response to a task execution failure instruction of the first subtask, obtaining a first sample number corresponding to the target sample and a target file number corresponding to the task execution result of each target sample;

[0008] Obtaining first execution result information of the target sample for which task execution failed according to the first sample number and the target file number;

[0009] According to the first execution result information, collecting each task log corresponding to the first execution result information;

[0010] Outputting a task execution instruction for the second subtask according to the log category corresponding to each of the task logs;

[0011] The log categories are classified according to the impact of the analysis results of each subtask on the overall analysis results of the metagenome.

[0012] Preferably, the response to the task execution failure instruction includes:

[0013] Obtain each subtask corresponding to the analysis task of each sample;

[0014] Performing classification and statistics according to the task type of each of the subtasks to obtain the first sample numbers and the target samples corresponding to each of the subtasks one by one;

[0015] In response to the task execution failure instruction, the first number of samples corresponding to the first subtask is retrieved.

[0016] Preferably, generating the task log includes:

[0017] Obtaining the number of task executions of the first subtask that the target sample has executed and a first threshold number of preset task failures;

[0018] If the task execution times are less than the first threshold times, the target sample continues to re-execute the first subtask;

[0019] If the task execution times are equal to the first threshold times, the task log is output and the target sample is stopped from executing the first subtask.

[0020] Preferably, obtaining the first execution result information includes:

[0021] Obtain each target file corresponding to the target file number;

[0022] Obtaining the second number of files corresponding to the second execution result according to the file type of each of the target files;

[0023] The first sample number and the second file number are compared to obtain the first execution result information.

[0024] Preferably, generating the first execution result information includes:

[0025] Comparing the first sample number with the second file number to obtain a quantity difference;

[0026] If the quantity difference is not equal to zero, the first execution result information is true, and the statistical file type is a file of the first execution result.

[0027] Preferably, the task execution instruction of the second subtask includes:

[0028] If the log category is the first category, outputting a first execution instruction;

[0029] If the log category is the second category, outputting a second execution instruction;

[0030] The first execution instruction is to terminate the execution of the second subtask, and the second execution instruction is to output the first warning information and continue to execute the second subtask.

[0031] Preferably, if the log category is the first category, the first execution instruction includes a third instruction, a fourth instruction and a fifth instruction, including:

[0032] According to the third instruction, the system delays the execution of the fourth instruction until each of the target samples completes the current subtask;

[0033] According to the fourth instruction, the system terminates all processes;

[0034] According to the fifth instruction, the system generates a manual intervention signal after executing the fourth instruction.

[0035] On the other hand, the present invention also provides a device for processing abnormalities in a metagenomic analysis task, the device comprising:

[0036] A data acquisition module: used for acquiring the first sample number corresponding to the target sample and the target file number corresponding to the task execution result of each target sample in response to the task execution failure instruction of the first subtask;

[0037] A data analysis module: used for obtaining first execution result information of the target sample for which task execution failed according to the first sample number and the target file number;

[0038] A data processing module: used for collecting task logs corresponding to the first execution result information according to the first execution result information;

[0039] Task execution module: used for outputting the task execution instruction of the second subtask according to the log category corresponding to each of the task logs;

[0040] The log categories are classified according to the impact of the analysis results of each subtask on the overall analysis results of the metagenome.

[0041] On the other hand, the present invention also provides an electronic device, at least one processor, at least one memory and computer program instructions stored in the memory, which implement any of the above methods when the computer program instructions are executed by the processor.

[0042] A computer-readable storage medium stores computer program instructions, which implement any of the above methods when executed by a processor.

[0043] In summary, the beneficial effects of the present invention are as follows:

[0044] The method, device and equipment for processing abnormalities of metagenomic analysis tasks provided by the present invention include responding to a task execution failure instruction of a first subtask, wherein the metagenomic analysis task is divided into multiple modules according to the task type for execution, and each subtask outputs a target file representing the execution result of the subtask after completion, and compares the total number of samples participating in the subtask with the number of target files of the task execution result to determine the first execution result information of the task execution failure; and according to the logs of each task log corresponding to the first execution result information, determine the execution plan of the second subtask, including terminating the task or issuing a warning and continuing the execution. The present invention can avoid the waste of parallel computing resources and the extension of the analysis cycle of multiple samples by delaying the execution of failed tasks in the task execution and performing hierarchical processing on the execution of failed tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0046] Figure 1 Schematic diagram of a process for handling abnormalities in a metagenomic analysis task in Example 1 of the present invention;

[0047] Figure 2 This is a flow chart of obtaining the target number of samples for executing the first subtask in Embodiment 1 of the present invention;

[0048] Figure 3 This is a schematic diagram of a process for obtaining first execution result information in Embodiment 1 of the present invention;

[0049] Figure 4 This is a logical diagram of the failure of the execution of the first subtask in Embodiment 1 of the present invention;

[0050] Figure 5 This is a schematic diagram of the structure of a device for processing abnormalities in a metagenomic analysis task in Example 2 of the present invention;

[0051] Figure 6 Schematic diagram of the structure of an electronic device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of further restrictions, the elements defined by the phrase "comprising..." do not exclude the existence of other identical elements in the process, method, article or device comprising the elements. If there is no conflict, the various features of the present invention and the embodiments can be combined with each other, all within the protection scope of the present invention.

[0053] Example 1

[0054] Metagenomic analysis tasks include eight analysis steps: data quality control, species identification, metabolic function annotation, contigs assembly, gene prediction, gene abundance calculation, binning and downstream statistical analysis. Metagenomic analysis tasks are usually performed simultaneously on a large number of samples in a parallel manner. Therefore, the analysis task is decomposed into multiple independent subtasks according to each step. After each sample completes the current subtask, it automatically enters the next subtask; at the same time, these eight subtasks are divided into first-category subtasks and second-category subtasks according to the degree of influence of the analysis results on the overall analysis results. The analysis results corresponding to the first-category subtasks will directly determine the overall evaluation of the target samples, and the analysis results corresponding to the second-category subtasks will not directly affect the overall evaluation of the target samples. The first-category subtasks include: data quality control, species identification, metabolic function annotation, contigs assembly, gene prediction and gene abundance calculation, and the second-category subtasks include: binning and downstream statistical analysis. The present invention decomposes the analysis task into multiple independent subtasks according to each link. If a sample fails to execute one of the subtasks, after human intervention to re-cast the task, each sample directly restarts from the unexecuted task. This can avoid the need to restart execution from the first subtask after human intervention when the overall task belongs to a process, thereby shortening the task analysis cycle and saving computer resources.

[0055] See also Figure 1 , Figure 1 The flowchart is a method for handling abnormalities in a metagenomic analysis task, the method comprising:

[0056] S1: In response to a task execution failure instruction of a first subtask, obtaining a first sample number corresponding to a target sample and a target file number corresponding to a task execution result of each target sample;

[0057] Specifically, to facilitate the description in this article, the subtask in which the task execution fails is recorded as the first subtask, and the subtask after the first subtask is recorded as the second subtask. When the system detects that the first subtask of the target sample has failed to execute, the first sample number corresponding to the target sample that needs to execute the first subtask is obtained, and at the same time, the number of target files representing the task execution results corresponding to the output of the completed first subtask is obtained. It should be noted that: each time the sample completes a subtask, the system will generate identification information corresponding to the subtask. In addition to the target file with an identification mark, the identification information can also be a specific symbol, signal, etc., and the form of the identification information is not specifically limited here.

[0058] S2: obtaining first execution result information of the target sample for which task execution failed according to the first sample number and the target file number;

[0059] Specifically, the first sample number is compared with the target file number. The target file number may be a failure identification file indicating failed task execution and / or a success identification file indicating successful task execution. Based on the comparison result, the first execution result information indicating the existence of a failed task execution is determined, including the target sample number for failed task execution. If the target file carries a unique identification mark of the target sample, the first execution result information may also include the target sample for failed task execution.

[0060] S3: According to the first execution result information, collecting each task log corresponding to the first execution result information;

[0061] Specifically, when each subtask is executed successfully, only a specific file indicating successful task execution is output; when each task fails, a specific file indicating the failure of task execution and a task log of the failed task execution need to be output. Furthermore, the file format of the task log is optimized so that it carries a straightforward identification of the task execution, thereby directly using the task log as an identification file of failed task execution. The first execution result information is obtained by directly identifying the task log, thereby saving computing resources and storage resources.

[0062] S4: outputting a task execution instruction for the second subtask according to the log category corresponding to each of the task logs;

[0063] The log categories are classified according to the impact of the analysis results of each subtask on the overall analysis results of the metagenome.

[0064] Specifically, obtain the task log of failed task execution. The categories of task logs include task failures handled by termination errors and task failures handled by warning errors. If it is a task log handled by termination errors, the task execution instruction for the second subtask is that the system stops generating new tasks, performs manual intervention, and initiates task re-casting. The system then continues to execute each subtask. At this time, the subtasks that have been successfully executed by each sample do not need to be continued, and the unexecuted tasks are executed directly. If it is a warning task log, it will not affect the task execution of each target sample. Only the warning task log is recorded. No manual intervention is required, and each target sample directly executes the second subtask.

[0065] The method for processing the abnormality of a metagenomic analysis task provided by the present invention includes responding to a task execution failure instruction of a first subtask, dividing the metagenomic analysis task into multiple modules according to the task type for execution, outputting a target file representing the execution result of the subtask after each subtask is completed, comparing the total number of samples participating in the subtask with the number of target files of the task execution result, and determining the first execution result information of the task execution failure; and determining the execution plan of the second subtask according to the log of each task log corresponding to the first execution result information, including terminating the task or issuing a warning and continuing the execution. The present invention can avoid the waste of parallel computing resources and the extension of the analysis cycle of multiple samples by delaying the execution of failed tasks in task execution and performing hierarchical processing on the execution of failed tasks.

[0066] In one embodiment, see Figure 2 , the S1 comprises:

[0067] S11: Obtain each subtask corresponding to the analysis task of each sample;

[0068] S12: performing classification and statistics according to the task type of each of the subtasks to obtain the first sample numbers and the target samples corresponding to each of the subtasks one by one;

[0069] S13: In response to the task execution failure instruction, retrieve the first number of samples corresponding to the first subtask.

[0070] In one embodiment, obtaining the first number of samples includes: in response to an execution start instruction of the first subtask, recording the first number of samples corresponding to samples participating in the execution of the first subtask.

[0071] Specifically, the analysis task of each sample is decomposed into subtasks of multiple independent analysis links. For example, for sample A that needs to participate in the full-process analysis, the analysis task of sample A is decomposed into 8 subtasks, including data quality control, species identification, metabolic function annotation, contigs assembly, gene prediction, gene abundance calculation, binning and downstream statistical analysis. Similarly, all other samples are also operated to obtain the first sample number of each of the 8 subtasks involved in data quality control, species identification, metabolic function annotation, contigs assembly, gene prediction, gene abundance calculation, binning and downstream statistical analysis. Before starting to execute each subtask, the first sample number of the target sample corresponding to each subtask is entered into the system. Each target sample will generate a corresponding success identification file after each subtask is successfully executed, and a corresponding failure identification file will be generated if the execution fails.

[0072] In one embodiment, generating the task log includes:

[0073] Obtaining the number of task executions of the first subtask that the target sample has executed and a first threshold number of preset task failures;

[0074] If the task execution times are less than the first threshold times, the target sample continues to re-execute the first subtask;

[0075] If the task execution times are equal to the first threshold times, the task log is output and the target sample is stopped from executing the first subtask.

[0076] Specifically, multi-threaded parallel processing is used to analyze the tasks of each target sample. When a target sample executes any subtask, if a task fails, the system will automatically re-execute the subtask according to the task re-investment mechanism. Generally, the task re-investment mechanism provided by the DSL language is used. When the number of failures of the same target sample to execute the same subtask reaches the first threshold number, the execution will be deemed to have failed and the task log will be output. In other words, every time the system detects that the same target sample fails to execute the first subtask, the number of executions of the first subtask will be increased by 1. When the number of executions of the first subtask is equal to the first threshold number, the system will terminate the target sample and execute the first subtask again, and output the failed task log for the technical staff to refer to.

[0077] In one embodiment, see Figure 3 , said S2 comprises:

[0078] S21: Obtain each target file corresponding to the target file number;

[0079] S22: Obtaining a second file number corresponding to the second execution result according to the file type of each target file;

[0080] S23: Compare the first sample number and the second file number to obtain the first execution result information.

[0081] Specifically, collect the target files corresponding to the target file number, filter out the successful identification files for successful task execution, count the second file number corresponding to the successful identification files, and compare the first sample number with the second file number. For example, if the first sample number is N and the second file number is M, if M≠N, there is first execution result information of failed task execution. If M=N, it indicates that all target samples of the current subtask are successfully executed. After all the target samples corresponding to the next subtask are executed, continue to count the first sample number and the second file number of the next subtask, and so on until all samples complete all analysis tasks. By setting a target file for characterizing execution failure or success to be generated after each subtask is executed, the system can start to perform task execution result statistics after each subtask is executed. By counting whether the number of second files for successful task execution is equal to the number of first files, it can be quickly determined whether there is a task execution failure. If the number of first files is equal to the number of second files, there is no execution failure of the current subtask of the target sample. If the number of first files is not equal to the number of second files, there is a failure in the execution of the current subtask of the target sample. If manual intervention is required, it can be performed in time. At the same time, since each subtask can be regarded as an independent task, manual intervention will not affect the subtask that has been successfully executed, thereby shortening the metagenomic analysis cycle and reducing the use of computer resources.

[0082] In one embodiment, the S23 includes:

[0083] S231: Compare the first sample number and the second file number to obtain a quantity difference;

[0084] S232: If the quantity difference is not equal to zero, the first execution result information is true, and the statistical file type is a file of the first execution result.

[0085] Specifically, when the system detects that the execution of the first subtask of any target sample fails, the target sample will be terminated to continue executing the first subtask, but the subtasks being executed by other target samples will not be terminated until all target samples complete the subtasks being executed. At the same time, each target sample will not continue to execute the next task after executing the current subtask (the system will no longer generate new subtasks). When all target samples have completed the current subtask, the target files corresponding to the target samples that have executed the first subtask are collected, and the successful identification files for successful task execution are screened out. The number of second files corresponding to the successful identification files is counted, and the number of first samples is compared with the number of second files. For example, if the number of first samples is N and the number of second files is M, if M≠N, it means that there is first execution result information of failed task execution, then the target files corresponding to the first execution result, that is, the failure identification files, are obtained.

[0086] After each subtask is completed, a target file is generated to characterize the execution failure or success. After each subtask has executed all the target samples that need to be executed, it is quickly determined whether there is a task failure by directly comparing the number of files. If there is a target sample and the current subtask has failed to execute, it is determined whether manual intervention is needed. If the error type is a prohibited error, manual intervention is performed in time. Since each subtask can be regarded as an independent task, the successfully executed subtask can be directly skipped after manual intervention, thereby shortening the metagenomic analysis cycle and reducing the use of computer resources.

[0087] In a specific embodiment, the subtask type of each thread is detected in real time. If the subtasks being executed by each thread are all tasks after the first subtask, the successful identification files of the successful execution of the first subtask of each target sample are counted. If the second file number of the successful identification files is not equal to the first sample number of the target sample, the first execution result information is output, and each target sample executes the first subtask, and each target sample is divided into at least one batch according to the total number of threads to complete.

[0088] Specifically, all target samples that need to execute the first subtask are divided into at least one batch for processing according to the maximum processing thread. For example, the target samples that need to execute the first subtask include sample 1, sample 2, sample 3, sample 4, sample 5, sample 6, sample 7, sample 8, sample 9 and sample 10, totaling 10 samples. If the maximum processing thread is 5, samples 1-5 are taken as the first batch, and samples 6-10 are taken as the second batch. If the maximum processing thread is greater than 10, samples 1-10 are one batch, that is, if the first subtask is the first task, samples 1-10 start executing the first subtask at the same time.

[0089] For example, samples 1-10 all need to complete subtask A and subtask B, the maximum number of threads is 5, and the number of system re-throws K=3 when the task execution fails. When samples 1-5 start to execute task A, samples 6-10 wait in the background. The time required for samples 1-5 to execute task A is not necessarily the same. For example, when the time required for samples 1-10 to execute task A and task B is different, when any one of samples 1-5 completes task A, one of samples 6-10 starts to execute task A, and so on, until samples 1-10 have all completed task A, the successful execution files obtained by sample 1-10 executing task A are counted. At this time, since the time required for samples 1-10 to execute task A is different, and there may be samples that have re-thrown tasks when executing task A, therefore, when samples 1-10 have all completed task A (including successful execution and failed execution), there must be a sample that is executing task B. Since samples 1-10 have all completed task A, at this time, the system starts to count the successful execution file pairs obtained by sample 1-10 executing task A. The corresponding second file number, if the second file number is not equal to the first sample number, that is, the second file number is less than 10, such as sample 6 failed to execute, the second file number is equal to 9, because sample 6 failed to execute, and according to sample 6, the task log type corresponding to the execution of task A is to terminate the whole process, so a fourth instruction for terminating the whole process is generated, and because other threads are executing tasks at this time, a third instruction for delaying the execution of the fourth instruction is generated. After all samples have completed the current subtask, such as: when counting the successful execution files of task A, other threads are processing task B of samples 1-4. Under the control of the third instruction, after waiting for samples 1-4 to complete task B, the fourth instruction is executed, the system terminates all full processes, and generates a fifth instruction for manual intervention. The technician adjusts the parameters of sample 6 and re-invests the task. At this time, since samples 1-4 have completed task B, they execute task B of samples 5, 7, 8, and 9, and execute task A of sample 6. The above operation is repeated until all samples complete all subtasks.

[0090] In one embodiment, the S4 includes:

[0091] S41: If the log category is the first category, output a first execution instruction;

[0092] S42: if the log category is the second category, outputting a second execution instruction;

[0093] The first execution instruction is to terminate the execution of the second subtask, and the second execution instruction is to output the first warning information and continue to execute the second subtask.

[0094] Specifically, after the metagenomic analysis task is decomposed into 8 subtasks including data quality control, species identification, metabolic function annotation, contigs assembly, gene prediction, gene abundance calculation, binning and downstream statistical analysis, the subtasks that have a decisive influence on the overall analysis results are recorded as the first type of subtasks, and the subtasks that do not have a decisive influence on the overall analysis results are recorded as the second type of subtasks. If the first type of subtask fails to execute, the first type of task log is output. The first type of task log is a termination error and requires human intervention. If the second type of subtask fails to execute, the second type of task log is output. The second type of task log is a warning error and does not require human intervention. The next subtask can be executed. Among them, the first type of subtasks include data quality control, species identification, metabolic function annotation, contigs assembly, gene prediction and gene abundance calculation, and the second type of subtasks include binning and downstream statistical analysis.

[0095] In one embodiment, if the log category is the first category, the first execution instruction includes a third instruction, a fourth instruction, and a fifth instruction, including:

[0096] According to the third instruction, the system delays the execution of the fourth instruction until each of the target samples completes the current subtask;

[0097] According to the fourth instruction, the system terminates all processes;

[0098] According to the fifth instruction, the system generates a manual intervention signal after executing the fourth instruction.

[0099] Specifically, if the task that failed to execute the target sample is the first type of subtask, the system will trigger the termination of the entire process and enter the manual intervention stage. After the technicians adjust the analysis parameters, they will execute the unfinished subtasks again.

[0100] If the fourth instruction to terminate the entire process appears, the target sample whose subtask is terminated will no longer re-submit the task, and wait for other target samples to complete the current subtask and output a success identification file or a failure identification file. When the system obtains the latest target file output by all target samples, the system executes the fourth instruction to terminate the entire process, and triggers the execution of the fifth instruction to generate a manual intervention signal. After the technicians adjust the parameters of the target samples that failed to execute, they will manually re-submit each target sample according to the task node before the process terminated. Each target sample will start execution from the subtask that has not yet been executed; the execution time of the fourth instruction is delayed by the third instruction to avoid the failed task affecting the progress of other subtasks processed in parallel, thereby avoiding waste of computing resources.

[0101] In a specific embodiment, if Figure 4As shown, the number of samples to be analyzed is N=10. Because the analysis process is decomposed into multiple independent analysis modules, each sample needs to first select the subtask to be performed before analysis. Assuming that all samples need to execute the complete process, the target samples corresponding to each subtask are 10, that is, the first number of samples is 10.

[0102] Taking the three subtasks of executing data quality control (fastp software), species identification (MetaPhlAn software) and contigs assembly (SPAdes software) as examples, after each subtask is completed, the system will generate a successful identification file for the task execution, such as: generating an identification file of the sample_1_module_1_success.log type to indicate successful execution, or generating a failure identification file for the task execution failure, such as: generating an identification file of the sample_A_module_1_error.log type to indicate execution failure. In order to save computing resources, the task log generated by the task execution failure is directly used as the failure identification file. Figure 4As shown in the figure, the preset number of automatic task re-investment of the system is 3 times. After the system automatically re-invests the task, sample 6 still fails to execute, and finally outputs the failure identification file: sample_6_fastp_error.log. Because the task execution failure instruction is not detected, at this time, the system responds to the second instruction of the first execution instruction, delays the execution of the first instruction of the first execution instruction corresponding to the termination of the entire process, and collects the target files generated by the target samples corresponding to each subtask after the task is completed. At this time, the first subtask is the data quality control (fastp software) task. At this time, among the 10 samples, except for sample 6 that fails to execute, the data quality control (fastp software) tasks of the remaining samples are successfully executed, and 9 corresponding successful identification files are obtained, that is, the number of the first sample of the first subtask is not equal to the number of target files. The task log of sample 6 is analyzed by category. Since the data quality control task is a first-class subtask, when the task fails to execute, the entire process needs to be terminated for manual intervention. At this time, the system stops generating new tasks and allows sample 1 to complete contigs assembly ( The system executes the first instruction system to trigger the termination of the whole process, and the technician manually intervenes to modify the fastp task parameters of sample 6 so that sample 6 can obtain the sequencing data after the data quality control task; then the system re-invests the tasks of all target samples, sample 1 has completed all tasks in this round, and there is no need to continue to execute the tasks at this time. Samples 2-5 directly execute the contigs assembly (SPAdes software) task, sample 6 starts from the data quality control (fastp software) task, and sample 7-10 starts from the species identification (MetaPhlAn software) task. When all tasks are completed and no new errors are generated, the system ends the whole process normally. The present invention significantly reduces the interference of task errors on the overall analysis process, reduces resource waste, and effectively shortens the analysis cycle by adopting a delayed processing method when a task execution failure is detected.

[0103] In a preferred embodiment, when the difference between the number of second files and the number of first samples is not equal to zero, the first target file corresponding to the first execution result is collected; if the first file number of the first target file is not equal to the difference in number, the task re-investment information of the first target sample is detected, and the first execution result information is approved based on the task re-investment information.

[0104] Specifically, the system estimates the execution time required for the sample to complete each subtask according to the number of samples, sets the statistical time point of the target file corresponding to the task execution result based on the execution time, and preferably counts the successful identification file of the successful execution. If the second file number of the successful identification file counted at the statistical time point is not equal to the first sample number, further count the first file number of the failure identification file corresponding to the failed task execution. If the sum of the first file number and the second file number is not equal to the first sample number, the first execution result is calibrated for task re-investment, and it is detected whether there is a sample in the task execution that the system reinvests. If so, it is detected whether the number of re-investments exceeds the first threshold number. If not, the next statistical time point is set. If it exceeds, the task process execution is forcibly terminated, and the task log is output. It is determined whether manual intervention is required based on the task log type. By presetting the statistical time point, it is possible to avoid real-time detection of task execution information (including task type and execution result) of each thread, saving computing resources.

[0105] In one embodiment, if there is a target sample whose current subtask is executed before the first subtask, the first sample number includes:

[0106] Obtaining a first task sequence number corresponding to the current subtask of each target sample and a second task sequence number corresponding to the first subtask;

[0107] Compare each of the first task sequence numbers with the second task sequence number to obtain a first sequence number group that is greater than or equal to the second task sequence number;

[0108] The first sample number is updated using the number of sequence numbers included in the first sequence number group.

[0109] Specifically, taking the example that samples 1 to 10 all need to execute all subtasks, when sample 5 is executing the 5th subtask, samples 1-4 are executing the 6th subtask, samples 6-8 are executing the 5th subtask, and samples 9-10 are executing the 4th subtask. When sample 5 is executing the 5th subtask, the task execution failure instruction is triggered. At this time, the task sequence number of the subtask being executed by each target sample is compared with the subtask sequence number being executed by sample 5, and the first sequence number group includes: subtask sequence number 6 of samples 1-4 and subtask sequence number 5 of samples 5-8, a total of 8 sequence numbers, that is, the first sample number is updated from the initial preset 10 to 8; this can avoid system errors caused by the system collecting task logs according to the originally set first sample number 10, thereby improving system stability.

[0110] In one embodiment, obtaining the first number of samples includes:

[0111] Obtaining a target file corresponding to the task execution result of the current subtask of each target sample;

[0112] Compare the first task sequence number corresponding to each target file with the second task sequence number corresponding to the first subtask, and output a third file number corresponding to the target files whose first task sequence number is greater than or equal to the second task sequence number;

[0113] The first sample number is updated using the third file number.

[0114] Specifically, collect the target files output by the target samples after executing the current subtask, extract the first task number that identifies the task type in the target file, record the target samples whose first task number is greater than or equal to the second task number as valid samples, and finally take the number of all valid samples as the first sample number.

[0115] The method for processing the abnormality of a metagenomic analysis task provided by the present invention includes responding to a task execution failure instruction of a first subtask, dividing the metagenomic analysis task into multiple modules according to the task type for execution, outputting a target file representing the execution result of the subtask after each subtask is completed, comparing the total number of samples participating in the subtask with the number of target files of the task execution result, and determining the first execution result information of the task execution failure; and determining the execution plan of the second subtask according to the log of each task log corresponding to the first execution result information, including terminating the task or issuing a warning and continuing the execution. The present invention can avoid the waste of parallel computing resources and the extension of the analysis cycle of multiple samples by delaying the execution of failed tasks in task execution and performing hierarchical processing on the execution of failed tasks.

[0116] Example 2

[0117] See also Figure 5 The present invention further provides a device for processing an abnormality of a metagenome analysis task based on the method for processing an abnormality of a metagenome analysis task in Example 1, and the method comprises:

[0118] A data acquisition module: used for acquiring the first sample number corresponding to the target sample and the target file number corresponding to the task execution result of each target sample in response to the task execution failure instruction of the first subtask;

[0119] A data analysis module: used for obtaining first execution result information of the target sample for which task execution failed according to the first sample number and the target file number;

[0120] A data processing module: used for collecting task logs corresponding to the first execution result information according to the first execution result information;

[0121] Task execution module: used for outputting the task execution instruction of the second subtask according to the log category corresponding to each of the task logs;

[0122] The log categories are classified according to the impact of the analysis results of each subtask on the overall analysis results of the metagenome.

[0123] In one embodiment, the data acquisition module includes:

[0124] Task acquisition unit: acquires each subtask corresponding to the analysis task of each sample;

[0125] Task classification unit: performing classification and statistics according to the task type of each subtask, to obtain the first sample numbers and the target samples corresponding to each subtask one by one;

[0126] A data retrieving unit: in response to the task execution failure instruction, retrieves the first number of samples corresponding to the first subtask.

[0127] In one embodiment, generating the task log includes:

[0128] A task re-investment unit: obtaining the number of task executions of the first subtask that the target sample has executed and a first threshold number of preset task failures;

[0129] A times checking unit: if the task execution times are less than the first threshold times, the target sample continues to re-execute the first subtask;

[0130] File output unit: if the task execution times are equal to the first threshold times, output the task log and stop the target sample from executing the first subtask.

[0131] In one embodiment, obtaining the first execution result information includes:

[0132] File acquisition unit: acquires each target file corresponding to the target file number;

[0133] A quantity comparison unit: obtaining the number of second files corresponding to the second execution result according to the file type of each target file;

[0134] A result generating unit compares the first sample number and the second file number to obtain the first execution result information.

[0135] In one embodiment, generating the first execution result information includes:

[0136] Data difference unit: compares the first sample number and the second file number to obtain a quantity difference;

[0137] Data judgment unit: If the quantity difference is not equal to zero, the first execution result information is true, and the statistical file type is a file of the first execution result.

[0138] In one embodiment, the task execution instruction of the second subtask includes:

[0139] Instruction recognition unit: if the log category is the first category, output a first execution instruction;

[0140] The instruction analysis unit: if the log category is the second category, outputting a second execution instruction;

[0141] The first execution instruction is to terminate the execution of the second subtask, and the second execution instruction is to output the first warning information and continue to execute the second subtask.

[0142] In one embodiment, if the log category is the first category, the first execution instruction includes a third instruction, a fourth instruction, and a fifth instruction, including:

[0143] Second instruction unit: according to the third instruction, the system delays the execution of the fourth instruction until each of the target samples completes the current subtask;

[0144] First instruction unit: according to the fourth instruction, the system terminates all processes;

[0145] The third instruction unit: according to the fifth instruction, the system generates a manual intervention signal after executing the fourth instruction.

[0146] The device for processing abnormalities of metagenomic analysis tasks provided by the present invention comprises a task execution failure instruction in response to a first subtask, wherein the metagenomic analysis task is divided into multiple modules according to the task type for execution, and a target file representing the execution result of the subtask is output after each subtask is completed, and the total number of samples participating in the subtask is compared with the number of target files of the task execution result to determine the first execution result information of the task execution failure; and according to the logs of each task log corresponding to the first execution result information, the execution plan of the second subtask is determined, including terminating the task or issuing a warning and continuing the execution. The present invention can avoid the waste of parallel computing resources and the extension of the analysis cycle of multiple samples by delaying the execution of failed tasks in the task execution and performing hierarchical processing on the execution of failed tasks.

[0147] Example 3

[0148] The present invention provides an electronic device and a storage medium, see Figure 6 , comprising at least one processor, at least one memory and computer program instructions stored in the memory.

[0149] Specifically, the above-mentioned processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of an embodiment of the present invention, and the electronic device includes at least one of the following: a computer, a mobile terminal, a PC, a tablet computer, etc.

[0150] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside the data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory.

[0151] In one example, the electronic device may further include a communication interface and a bus, wherein the processor, the memory, and the communication interface are connected via the bus and communicate with each other.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing abnormalities in a metagenomic analysis task, characterized in that: The method comprises: In response to a task execution failure instruction of the first subtask, obtaining a first sample number corresponding to the target sample and a target file number corresponding to the task execution result of each target sample; Obtaining first execution result information of the target sample for which task execution failed according to the first sample number and the target file number; According to the first execution result information, collecting each task log corresponding to the first execution result information; Outputting a task execution instruction for the second subtask according to the log category corresponding to each of the task logs; The log categories are classified according to the impact of the analysis results of each subtask on the overall analysis results of the metagenome.

2. The method for processing abnormalities in a metagenomic analysis task according to claim 1, characterized in that: The response to the task execution failure instruction includes: Obtain each subtask corresponding to the analysis task of each sample; Performing classification and statistics according to the task type of each of the subtasks to obtain the first sample numbers and the target samples corresponding to each of the subtasks one by one; In response to the task execution failure instruction, the first number of samples corresponding to the first subtask is retrieved.

3. The method for processing abnormalities in a metagenomic analysis task according to claim 1, characterized in that: Generating the task log includes: Obtaining the number of task executions of the first subtask that the target sample has executed and a first threshold number of preset task failures; If the task execution times are less than the first threshold times, the target sample continues to re-execute the first subtask; If the task execution times are equal to the first threshold times, the task log is output and the target sample is stopped from executing the first subtask.

4. The method for processing abnormalities in a metagenomic analysis task according to claim 1, characterized in that: Acquiring the first execution result information includes: Obtain each target file corresponding to the target file number; Obtaining the second number of files corresponding to the second execution result according to the file type of each of the target files; The first sample number and the second file number are compared to obtain the first execution result information.

5. The method for processing abnormalities in a metagenomic analysis task according to claim 4, characterized in that: Generating the first execution result information includes: Comparing the first sample number with the second file number to obtain a quantity difference; If the quantity difference is not equal to zero, the first execution result information is true, and the statistical file type is a file of the first execution result.

6. The method for processing abnormalities in a metagenomic analysis task according to claim 4, characterized in that: The task execution instruction of the second subtask includes: If the log category is the first category, outputting a first execution instruction; If the log category is the second category, outputting a second execution instruction; The first execution instruction is to terminate the execution of the second subtask, and the second execution instruction is to output the first warning information and continue to execute the second subtask.

7. The method for processing abnormalities in a metagenomic analysis task according to claim 6, characterized in that: If the log category is the first category, the first execution instruction includes a third instruction, a fourth instruction and a fifth instruction, including: According to the third instruction, the system delays the execution of the fourth instruction until each of the target samples completes the current subtask; According to the fourth instruction, the system terminates all processes; According to the fifth instruction, the system generates a manual intervention signal after executing the fourth instruction.

8. A device for processing abnormalities in a metagenomic analysis task, characterized in that: The device comprises: A data acquisition module: used for acquiring the first sample number corresponding to the target sample and the target file number corresponding to the task execution result of each target sample in response to the task execution failure instruction of the first subtask; A data analysis module: used for obtaining first execution result information of the target sample for which task execution failed according to the first sample number and the target file number; A data processing module: used for collecting task logs corresponding to the first execution result information according to the first execution result information; Task execution module: used for outputting a task execution instruction of the second subtask according to the log category corresponding to each of the task logs; The log categories are classified according to the impact of the analysis results of each subtask on the overall analysis results of the metagenome.

9. An electronic device, characterized in that: At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images