Sequencing data analyzing and monitoring system based on distributed computing

Abstract

The invention provides a sequencing data analyzing and monitoring system based on distributed computing. The system comprises: a log monitoring module, a system parameter monitoring module, a real-time command monitoring module, and a storage space statistics module, wherein the log monitoring module is connected with a database, the system parameter monitoring module is connected with a CPU module, the real-time command monitoring module is connected with the database and the CPU module, and the storage space statistics module is connected with the database. The system can conveniently monitor all analysis tasks delivered on a cluster, can organize the analysis tasks in an organized form, and on the basis of running tasks delivered on the cluster, can perform statistics on samples, perform statistics on running sample tasks on the cluster, perform statistics on tasks, and perform statistics on delivery task workflows; by means of the technical solution, the CPU, the memory, and the cluster resource use condition of the delivery tasks can be visually obtained in a very simple mode, and analysis logs of system tasks and the space occupied by analysis can be output in real time.

Description

technical field [0001] The invention relates to the field of analysis of biological massive data, in particular to a sequencing data analysis and monitoring system based on distributed computing. Background technique [0002] In the analysis of biological mass data, there will be a large number of computing and data analysis tasks running on the cluster. At present, a core issue of high-throughput sequencing is data analysis. For example, a data needs to go through dozens of steps of data analysis before it can be obtained. The analysis results, including a series of operations such as data cleaning, data filtering, data comparison, data deduplication, data correlation analysis, database comparison and dimensionality reduction analysis, and some problems caused by this: for example, the sample size is huge , the user requires to get the analysis results within a limited time, and when the sample size increases, hundreds or even thousands of tasks will be performed at the sam...

AI Technical Summary

Problems solved by technology

[0002] In the analysis of biological mass data, there will be a large number of computing and data analysis tasks running on the cluster. At present, a core issue of high-throughput sequencing is data analysis. For example, a data needs to go through dozens of steps of data analysis before it can be obtained. The analysis results, including a series of operations such as data cleaning, data filtering, data comparison, data deduplication, data correlation analysis, database comparison and dimensionality reduction analysis, and some problems caused by this: for example, the sample size is huge , the user requires to get the analysis results within a limited time, and when the sample size increases, hundreds or even thousands of tasks will be performed at the same time, resulting in the difficulty of task management and archiving of task analysis results

Images