High-efficiency high-throughput gene sequencing data processing system

Abstract

The invention belongs to the technical field of gene sequencing, and particularly relates to an efficient and high-throughput gene sequencing data processing system. The data processing system comprises a data acquisition module, a data preprocessing module, a data storage and management module, a data comparison and alignment module, a variation detection module, a data integration and analysis module, a visualization and report module and a quality control and verification module, and the data acquisition module is used for extracting DNA or RNA and comprises phenol chloroform extraction, column purification and magnetic bead extraction; and detecting the concentration and the quality of the sample by using a spectrophotometer or a fluorescence quantifier, and carrying out pollution removal and degradation treatment on the extracted nucleic acid. According to the method, a comprehensive, accurate and efficient data processing solution can be provided for genomics research by optimizing the data processing flow, improving the data quality, enhancing the analysis capability and supporting automation.

Description

Technical Field

[0001] The present invention belongs to the technical field of gene sequencing, and in particular relates to an efficient and high-throughput gene sequencing data processing system. Background Art

[0002] Gene sequencing is the process of systematic nucleotide sequence analysis of the genome or specific gene regions of an organism, with the main purpose of revealing genetic information. Gene sequencing technologies are divided into types such as whole genome sequencing, exome sequencing, transcriptome sequencing, and target region sequencing. Its basic steps include sample preparation, library construction, sequencing, data preprocessing, data analysis, and result interpretation. Sequencing technologies include Illumina sequencing, PacBio sequencing, and Nanopore sequencing, and their application areas cover medical research, agriculture, environmental science, and forensics. Despite the continuous development of gene sequencing technology, data processing, interpretation, cost, and privacy and ethical issues remain major challenges.

[0003] The existing gene sequencing data processing system has a long data processing time, poor processing accuracy and reliability, and is not convenient for the system to detect and repair errors and deviations in the data, and cannot reveal the potential patterns and structures in the data. To this end, we propose an efficient and high-throughput gene sequencing data processing system to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to provide an efficient and high-throughput gene sequencing data processing system, which can provide a comprehensive, accurate and efficient data processing solution for genomics research by optimizing the data processing process, improving data quality, enhancing analysis capabilities and supporting automation.

[0005] The technical solution adopted by the present invention is as follows:

[0006] An efficient and high-throughput gene sequencing data processing system, the data processing system includes a data acquisition module, a data preprocessing module, a data storage and management module, a data comparison and alignment module, a variation detection module, a data integration and analysis module, a visualization and reporting module, and a quality control and verification module.

[0007] In a preferred embodiment, the data acquisition module extracts DNA or RNA including phenol-chloroform extraction, column purification, and magnetic bead extraction, uses a spectrophotometer or a fluorescence quantifier to detect the concentration and quality of the sample, removes contamination and degradation of the extracted nucleic acid, cuts the extracted DNA or RNA into fragments of appropriate length by enzyme digestion, ultrasound or other methods, amplifies the library using polymerase chain reaction to increase the concentration and amount of the target fragment, removes unconnected adapters and other impurities, uses a bioanalyzer to check the fragment size and concentration of the library, loads the constructed library into a flow slot, chip or flow pool of a sequencer, starts the sequencer, reads the sequence by monitoring the current changes generated when the DNA molecules pass through the nanopore, and the sequencer generates and stores the sequence data in real time in a built-in storage.

[0008] In a preferred embodiment, the data preprocessing module operation includes evaluating and screening the quality of sequencing data, importing the original FASTQ format sequencing data from the data storage system into the analysis environment, usually using a command line tool or a graphical user interface to generate a quality report to display the read quality, quality statistics of each base position and other important indicators, setting a threshold based on the quality control report, filtering out reads that do not meet the quality standards, using Trimmomatic or Cutadapt to perform read screening to remove low-quality sequences, filtering out reads shorter than a set length, cutting low-quality bases from both ends of the read according to the quality value, and adjusting the read length.

[0009] In a preferred embodiment, the operation of the data storage and management module includes receiving data that has been filtered, linker removed and quality checked from the data preprocessing module, storing it in FASTQ, BAM or VCF format, importing the received data into the data storage system using a script or data management tool, classifying and organizing the data according to projects, sample types, sequencing dates, etc., recording, managing and updating metadata about sequencing data, generating sample lists and experimental reports, establishing an automated backup system, regularly backing up data to different storage media, setting data access permissions according to user roles and responsibilities, and using single sign-on and role-based access control (RBAC) to manage user permissions.

[0010] In a preferred embodiment, the operation of the data comparison and alignment module includes selecting an appropriate reference genome according to the experimental purpose, downloading the reference genome sequence from Ensembl and UCSC Genome Browser, converting the reference genome sequence into a format suitable for comparison, using BWA and Bowtie2 to index the reference genome, generating an index file to improve the comparison efficiency, the index file contains preprocessed data of the reference genome, comparing the preprocessed sequencing reads with the indexed reference genome, using Samtools or Picard to sort the comparison results and remove duplicate reads, eliminating redundant data caused by PCR amplification, performing quality checks on the comparison results, evaluating the accuracy and coverage of the comparison, using IGV and UCSC Genome Browser to view the comparison results, checking the distribution and alignment of the reads on the genome, combining the comparison results with the genome annotation data, and performing variation detection, gene expression analysis, and structural variation identification.

[0011] In a preferred embodiment, the operation of the variation detection module includes selecting a suitable variation detection tool according to the analysis requirements, the variation detection tools include GATK, Samtools, FreeBayes, and Mutect2, using the variation detection tool to compare and process the results, identifying the variation sites in the sequencing data, performing quality filtering on the variation preliminarily detected, and removing possible false positives. The filtering criteria include variation quality score, sequencing depth, and variation frequency. ANNOVAR, SnpEff, and VEP are used to perform functional annotations on the variation, provide information on genes, protein functions, and disease associations that may be affected by the variation, integrate the variation detection results with the annotation information, and generate a final variation report. The generated report includes the variation site, variation type, annotation information, and possible biological significance. Sanger sequencing and genotyping are used to experimentally verify the detected important variation, and the variation detection results are stored in a database or file system.

[0012] In a preferred embodiment, the operation of the data integration and analysis module includes collecting raw data from different experiments, including sequencing data, variation data, and expression data, converting all data into a unified format, including FASTA, FASTQ, VCF, BAM, GTF, and BED, using BEDTools and Picard for format conversion and data processing, merging data from different experiments or samples to form a unified data set, standardizing the data, removing duplicate data entries, and performing data cleaning, checking and repairing missing values ​​and outliers, calculating basic statistics of the data, including mean, standard deviation, and variance, describing the distribution of the data, generating basic statistical charts of the data, performing differential analysis on data under different groups or conditions, identifying significant differences, analyzing the correlation between different data types, using Pearson correlation, Spearman rank correlation, and regression analysis to reveal the relationship between the data, and identifying biological pathways and networks related to the research objectives.

[0013] In a preferred embodiment, the operation of the visualization and reporting module includes converting data into CSV, TSV, and JSON formats, extracting key data points from the analysis results, including statistics, variant locations, and gene expression levels, selecting visualization tools, including R, Python, Tableau, Power BI, and Cytoscape, selecting chart types based on the data and analysis results, including scatter plots, histograms / bar charts, heat maps, box plots, volcano plots, and network diagrams, designing the appearance of the chart, including colors, labels, coordinate axes, and legends, adding interactive functions to required charts, including filtering, zooming, and floating prompts, planning the structure of the report, including introduction, methods, results, discussion, and conclusions, and inserting previously generated charts and tables into the report.

[0014] In a preferred embodiment, the operation of the quality control and verification module includes performing a preliminary check on the raw data, checking whether the sequencing data has missing reads or format errors, using FastQC to perform quality assessment on the sequencing data, detecting sequencing errors, GC content deviations, and sequence duplications, checking the quality of the alignment results, evaluating the alignment rate, alignment accuracy, and coverage, comparing the data consistency between different samples or experimental groups, performing quality control on the data processing process, ensuring that steps such as data cleaning, alignment, and variation detection are performed in accordance with a predetermined process, using control samples or control group data to check the accuracy of the data processing process, using standard samples or samples with known variations for verification, evaluating the accuracy and stability of the tool by comparison with other tools or known results, and independently verifying key results.

[0015] In a preferred embodiment, the operation of the data integration and analysis module also includes applying cluster analysis and principal component analysis (PCA) to discover potential patterns and structures in the data, using K-means clustering and hierarchical clustering to divide the data into groups with similar characteristics, training machine learning models for prediction and classification tasks, and evaluating model performance, including accuracy, recall, and F1 score.

[0016] The technical effects achieved by the present invention are:

[0017] The system covers the entire gene sequencing data processing flow from data collection to final report generation through efficient modular design. This structured processing method not only reduces data processing time, but also improves processing accuracy and reliability. The functions of each module focus on specific tasks and can quickly and efficiently complete data preprocessing, alignment, variant detection and analysis.

[0018] The quality control and verification module strictly monitors the entire data processing process to ensure that every step from sample extraction to data analysis meets quality standards. By using FastQC for preliminary quality checks, control sample verification and standard sample testing, the system can effectively detect and repair errors and deviations in the data, thereby improving the accuracy and reliability of the data;

[0019] The data integration and analysis module can process data from different experiments and samples, convert them into a standard format, and perform comprehensive analysis. Through standardized processing, data cleaning, statistical analysis, and differential analysis, the system can reveal potential patterns and structures in the data, which helps to discover meaningful biological information in large amounts of data and support subsequent in-depth analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of an efficient and high-throughput gene sequencing data processing system of the present invention. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0022] See also Figure 1 As shown, the present invention provides an efficient and high-throughput gene sequencing data processing system, which includes a data acquisition module, a data preprocessing module, a data storage and management module, a data comparison and alignment module, a variation detection module, a data integration and analysis module, a visualization and reporting module, and a quality control and verification module.

[0023] The data acquisition module extracts DNA or RNA including phenol-chloroform extraction, column purification, and magnetic bead extraction, uses a spectrophotometer or a fluorescence quantifier to detect the concentration and quality of the sample, removes contamination and degradation of the extracted nucleic acid, cuts the extracted DNA or RNA into fragments of appropriate length by enzyme digestion, ultrasound or other methods, amplifies the library using polymerase chain reaction to increase the concentration and amount of the target fragment, removes unconnected adapters and other impurities, uses a bioanalyzer to check the fragment size and concentration of the library, loads the constructed library into the flow slot, chip or flow pool of the sequencer, starts the sequencer, reads the sequence by monitoring the change in current generated when the DNA molecule passes through the nanopore, and the sequencer generates sequence data in real time and stores it in the built-in storage.

[0024] The data preprocessing module operation includes evaluating and screening the quality of sequencing data, importing the original FASTQ format sequencing data from the data storage system into the analysis environment, usually using command line tools or graphical user interfaces to generate quality reports that display the quality of reads, quality statistics for each base position, and other important indicators. According to the quality control report, thresholds are set to filter out reads that do not meet the quality standards, and Trimmomatic or Cutadapt is used for read screening to remove low-quality sequences, filter out reads shorter than the set length, cut low-quality bases from both ends of the read according to the quality value, and adjust the read length.

[0025] The operation of the data storage and management module includes receiving the filtered, connector-removed and quality-checked data from the data preprocessing module, storing it in FASTQ, BAM or VCF format, importing the received data into the data storage system using scripts or data management tools, classifying and organizing the data according to projects, sample types, sequencing dates, etc., recording, managing and updating metadata about sequencing data, generating sample lists and experimental reports, establishing an automated backup system, regularly backing up data to different storage media, setting data access permissions based on user roles and responsibilities, and using single sign-on and role-based access control (RBAC) to manage user permissions.

[0026] The operation of the data comparison and alignment module includes selecting an appropriate reference genome according to the experimental purpose, downloading the reference genome sequence from Ensembl and UCSC Genome Browser, converting the reference genome sequence into a format suitable for comparison, using BWA and Bowtie2 to index the reference genome, generating an index file to improve the comparison efficiency. The index file contains the preprocessed data of the reference genome, comparing the preprocessed sequencing reads with the indexed reference genome, using Samtools or Picard to sort the comparison results and remove duplicate reads, eliminating redundant data caused by PCR amplification, performing quality checks on the comparison results, evaluating the accuracy and coverage of the comparison, using IGV and UCSC Genome Browser to view the comparison results, checking the distribution and alignment of the reads on the genome, combining the comparison results with the genome annotation data, and performing variation detection, gene expression analysis, and structural variation identification.

[0027] The operation of the variation detection module includes selecting appropriate variation detection tools according to the analysis requirements. Variation detection tools include GATK, Samtools, FreeBayes, and Mutect2. Variation detection tools are used to compare and process the results, identify variation sites in sequencing data, perform quality filtering on the initially detected variation, and remove possible false positives. The filtering criteria include variation quality score, sequencing depth, and variation frequency. ANNOVAR, SnpEff, and VEP are used to perform functional annotations on variation, provide information about genes, protein functions, and disease associations that may be affected by variation, integrate variation detection results with annotation information, and generate a final variation report. The generated report includes variation sites, variation types, annotation information, and possible biological significance. Sanger sequencing and genotyping are used to experimentally verify the detected important variation, and the variation detection results are stored in a database or file system.

[0028] The operation of the data integration and analysis module includes collecting raw data from different experiments, including sequencing data, variant data, and expression data, converting all data into a unified format, including FASTA, FASTQ, VCF, BAM, GTF, and BED, using BEDTools and Picard for format conversion and data processing, merging data from different experiments or samples to form a unified data set, standardizing the data, removing duplicate data entries, and performing data cleaning, checking and repairing missing values ​​and outliers, calculating basic statistics of the data, including mean, standard deviation, and variance, describing the distribution of the data, generating basic statistical charts of the data, performing differential analysis on data under different groups or conditions, identifying significant differences, analyzing the correlation between different data types, using Pearson correlation, Spearman rank correlation, and regression analysis to reveal the relationship between the data, and identifying biological pathways and networks related to the research objectives.

[0029] The operation of the visualization and reporting module includes converting data into CSV, TSV, and JSON formats, extracting key data points from the analysis results, including statistics, variant locations, and gene expression levels, selecting visualization tools, including R, Python, Tableau, Power BI, and Cytoscape, selecting chart types based on data and analysis results, including scatter plots, bar charts / bar charts, heat maps, box plots, volcano plots, and network diagrams, designing the appearance of charts, including colors, labels, coordinate axes, and legends, adding interactive functions to required charts, including filtering, zooming, and floating prompts, planning the structure of the report, including introduction, methods, results, discussion, and conclusions, and inserting previously generated charts and tables into the report.

[0030] The operation of the quality control and verification module includes preliminary inspection of the raw data, checking whether the sequencing data has missing reads or format errors, using FastQC to perform quality assessment on the sequencing data, detecting sequencing errors, GC content deviations, and sequence duplications, checking the quality of the alignment results, evaluating the alignment rate, alignment accuracy, and coverage, comparing the data consistency between different samples or experimental groups, performing quality control on the data processing process, ensuring that steps such as data cleaning, alignment, and variant detection are carried out in accordance with the predetermined process, using control samples or control group data to check the accuracy of the data processing process, using standard samples or samples with known variants for verification, evaluating the accuracy and stability of the tool by comparing with other tools or known results, and independently verifying key results.

[0031] The data integration and analysis module also includes applying cluster analysis and principal component analysis (PCA) to discover potential patterns and structures in the data, using K-means clustering and hierarchical clustering to divide the data into groups with similar characteristics, training machine learning models for prediction and classification tasks, and evaluating model performance, including accuracy, recall, and F1 score.

[0032] In the present invention, the system covers the entire gene sequencing data processing flow from data collection to final report generation through efficient modular design. This structured processing method not only reduces data processing time, but also improves the accuracy and reliability of processing. The function of each module focuses on a specific task, and can quickly and efficiently complete data preprocessing, alignment, variant detection and analysis;

[0033] The quality control and verification module strictly monitors the entire data processing process to ensure that every step from sample extraction to data analysis meets quality standards. By using FastQC for preliminary quality checks, control sample verification, and standard sample testing, the system can effectively detect and fix errors and deviations in the data, thereby improving the accuracy and reliability of the data;

[0034] The data integration and analysis module can process data from different experiments and samples, convert them into a standard format, and conduct comprehensive analysis. Through standardization, data cleaning, statistical analysis, and differential analysis, the system can reveal potential patterns and structures in the data. This helps to discover meaningful biological information in large amounts of data and supports subsequent in-depth analysis.

[0035] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. An efficient and high-throughput gene sequencing data processing system, characterized in that: The data processing system includes a data acquisition module, a data preprocessing module, a data storage and management module, a data comparison and alignment module, a variation detection module, a data integration and analysis module, a visualization and reporting module, and a quality control and verification module.

2. The efficient and high-throughput gene sequencing data processing system according to claim 1, characterized in that: The data acquisition module extracts DNA or RNA including phenol-chloroform extraction, column purification, and magnetic bead extraction, uses a spectrophotometer or a fluorescence quantifier to detect the concentration and quality of the sample, removes contamination and degradation of the extracted nucleic acid, cuts the extracted DNA or RNA into fragments of appropriate length by enzyme cutting, ultrasound or other methods, amplifies the library by polymerase chain reaction to increase the concentration and amount of the target fragment, removes unconnected adapters and other impurities, uses a bioanalyzer to check the fragment size and concentration of the library, loads the constructed library into a flow slot, chip or flow pool of a sequencer, starts the sequencer, reads the sequence by monitoring the current change generated when the DNA molecule passes through the nanopore, and the sequencer generates and stores the sequence data in real time in a built-in storage.

3. The efficient and high-throughput gene sequencing data processing system according to claim 1, characterized in that: The data preprocessing module operation includes evaluating and screening the quality of sequencing data, importing the original FASTQ format sequencing data from the data storage system into the analysis environment, usually using a command line tool or a graphical user interface to generate a quality report to display the quality of the reads, the quality statistics of each base position and other important indicators, setting thresholds based on the quality control report, filtering out reads that do not meet the quality standards, using Trimmomatic or Cutadapt to perform read screening to remove low-quality sequences, filtering out reads shorter than a set length, cutting low-quality bases from both ends of the reads based on the quality values, and adjusting the read length.

4. The efficient and high-throughput gene sequencing data processing system according to claim 1, characterized in that: The operation of the data storage and management module includes receiving the data after filtering, linker removal and quality inspection from the data preprocessing module, storing it in FASTQ, BAM or VCF format, importing the received data into the data storage system using a script or data management tool, classifying and organizing the data according to projects, sample types, sequencing dates, etc., recording, managing and updating metadata about sequencing data, generating sample lists and experimental reports, establishing an automated backup system, regularly backing up data to different storage media, setting data access permissions according to user roles and responsibilities, and using single sign-on and role-based access control (RBAC) to manage user permissions.

5. The efficient and high-throughput gene sequencing data processing system according to claim 1, characterized in that: The operation of the data comparison and alignment module includes selecting an appropriate reference genome according to the experimental purpose, downloading the reference genome sequence from Ensembl and UCSC Genome Browser, converting the reference genome sequence into a format suitable for comparison, using BWA and Bowtie2 to index the reference genome, generating an index file to improve the comparison efficiency, the index file contains preprocessed data of the reference genome, comparing the preprocessed sequencing reads with the indexed reference genome, using Samtools or Picard to sort the comparison results and remove duplicate reads, eliminating redundant data caused by PCR amplification, performing quality checks on the comparison results, evaluating the accuracy and coverage of the comparison, using IGV and UCSC Genome Browser to view the comparison results, checking the distribution and alignment of the reads on the genome, combining the comparison results with the genome annotation data, and performing variation detection, gene expression analysis, and structural variation identification.

6. The efficient and high-throughput gene sequencing data processing system according to claim 1, characterized in that: The operation of the variation detection module includes selecting a suitable variation detection tool according to the analysis requirements, the variation detection tools include GATK, Samtools, FreeBayes, and Mutect2, using the variation detection tool to compare and process the results, identifying the variation sites in the sequencing data, performing quality filtering on the variation initially detected, and removing possible false positives, the filtering criteria include variation quality score, sequencing depth, and variation frequency, using ANNOVAR, SnpEff, and VEP to functionally annotate the variation, providing information about genes, protein functions, and disease associations that the variation may affect, integrating the variation detection results with the annotation information, and generating a final variation report, the generated report includes the variation site, variation type, annotation information, and possible biological significance, using Sanger sequencing and genotyping to experimentally verify the detected important variation, and storing the variation detection results in a database or file system.

7. The efficient and high-throughput gene sequencing data processing system according to claim 1, characterized in that: The operation of the data integration and analysis module includes collecting raw data from different experiments, including sequencing data, variation data, and expression data, converting all data into a unified format, including FASTA, FASTQ, VCF, BAM, GTF, and BED, using BEDTools and Picard for format conversion and data processing, merging data from different experiments or samples to form a unified data set, standardizing the data, removing duplicate data entries, and performing data cleaning, checking and repairing missing values ​​and outliers, calculating basic statistics of the data, including mean, standard deviation, and variance, describing the distribution of the data, generating basic statistical charts of the data, performing differential analysis on data under different groups or conditions, identifying significant differences, analyzing the correlation between different data types, using Pearson correlation, Spearman rank correlation, and regression analysis to reveal the relationship between the data, and identifying biological pathways and networks related to the research objectives.

8. The efficient and high-throughput gene sequencing data processing system according to claim 1, characterized in that: The operation of the visualization and reporting module includes converting data into CSV, TSV, and JSON formats, extracting key data points from the analysis results, including statistics, variant locations, and gene expression levels, selecting visualization tools, including R, Python, Tableau, Power BI, and Cytoscape, selecting chart types based on data and analysis results, including scatter plots, histograms / bar charts, heat maps, box plots, volcano plots, and network diagrams, designing the appearance of charts, including colors, labels, coordinate axes, and legends, adding interactive functions to required charts, including filtering, zooming, and floating prompts, planning the structure of the report, including introduction, methods, results, discussion, and conclusions, and inserting previously generated charts and tables into the report.

9. The efficient and high-throughput gene sequencing data processing system according to claim 1, characterized in that: The operation of the quality control and verification module includes preliminary inspection of the raw data, checking whether the sequencing data has missing reads or format errors, using FastQC to perform quality assessment on the sequencing data, detecting sequencing errors, GC content deviations, and sequence duplications, checking the quality of the alignment results, evaluating the alignment rate, alignment accuracy, and coverage, comparing the data consistency between different samples or experimental groups, performing quality control on the data processing process, ensuring that steps such as data cleaning, alignment, and variation detection are performed in accordance with a predetermined process, using control samples or control group data to check the accuracy of the data processing process, using standard samples or samples with known variations for verification, evaluating the accuracy and stability of the tool by comparison with other tools or known results, and independently verifying key results.

10. The efficient and high-throughput gene sequencing data processing system according to claim 1, characterized in that: The data integration and analysis module operation also includes applying cluster analysis and principal component analysis (PCA) to discover potential patterns and structures in the data, using K-means clustering and hierarchical clustering to divide the data into groups with similar characteristics, training machine learning models for prediction and classification tasks, and evaluating model performance, including accuracy, recall, and F1 score.

Images