Method, system and application for rapid batch automatic download of high-throughput sequencing data

Abstract

The present invention discloses a method for fast batch automatic downloading of high-throughput sequencing data, the method is based on snakemake language and MySQL database, and specifically includes the following steps: process triggering; capturing sample information in the process; uploading sample information to MySQL database; automatically downloading samples marked as wait in the database; modifying the sample status after the download starts in the database; synchronizing sample information to the MySQL database after the download is completed; modifying the database status of the downloaded sample; sending a summary email of the download; deleting sample data of completed projects. The method of the present invention can synchronize and download sample information in batches, and the whole process does not require human intervention, and can automatically trigger various links in the middle. During the processing process, the download summary information of the sample data will be automatically sent to the monitor in a timely manner to ensure that all operation steps can be traced, and the MySQL database is used to facilitate various information queries. The present invention also discloses a system and application for implementing the above method.

Description

Technical Field

[0001] The present invention belongs to the technical field of high-throughput sequencing, and relates to a method for rapidly and automatically downloading high-throughput sequencing data in batches, a data downloading system and applications thereof. Background Art

[0002] High-throughput sequencing, also known as "next-generation sequencing," is a revolution in traditional sequencing. Compared with traditional first-generation Sanger sequencing, the throughput of next-generation sequencing technology has increased by one to two orders of magnitude, and it can economically cover the genome / transcriptome at a high rate. As the performance of high-throughput sequencing instruments gradually stabilizes and their prices continue to fall, their applications are becoming more and more widespread. Therefore, research based on high-throughput sequencing data will show a rapid development trend in terms of quantity and application.

[0003] Since the amount of data produced by high-throughput sequencing is very large, the size of a single file ranges from a few megabytes to tens of gigabytes; data downloading is generally done manually and is limited by the time it takes to complete the data download. Human intervention is not always timely, which results in the data download process taking a considerable amount of time and requiring human monitoring, which consumes a lot of human involvement. Therefore, efficient and smooth downloading of high-throughput speed measurement data is one of the key steps that must be taken.

[0004] Since high-throughput sequencing data has the characteristics of large data volume and large number of samples, the download of high-throughput sequencing data must have the characteristics of multi-sample batch processing, smooth single sample download, real-time monitoring of single sample data download status, and rapid feedback of multi-sample data download summary information. At present, the conventional data download method is manual processing with the help of general data download tools, which can only be processed in batches. The download of high-throughput sequencing data with large sample volume may take several hours to several days. It is difficult to monitor the download status manually, and it is impossible to make rapid and accurate feedback on the download results. There is no task monitoring mechanism, making the download of high-throughput sequencing data a major bottleneck in related research.

[0005] The existing high-throughput sequencing data download process has the following defects: (1) Low process processing efficiency: there may be a long blank waiting period between receiving the process and the final download completion; (2) Inability to process samples in batches: only a single batch of samples can be downloaded each time, and multiple batches of samples cannot be processed at the same time; (3) Untimely feedback on download results: manual verification is required after the process is completed, and email feedback cannot be provided in a timely manner; (4) No error detection mechanism: There is no detection mechanism for whether a single sample has been downloaded / successfully; (5) No real-time monitoring mechanism: The status of data download cannot be monitored in real time and can only be verified manually. Summary of the invention

[0006] In order to overcome the deficiencies in the prior art, the present invention aims to provide a method, a data downloading system and an application for quickly and automatically downloading high-throughput sequencing data in batches based on snakemake language and MySQL database.

[0007] The method specifically comprises the following steps:

[0008] Step 1: Process trigger:

[0009] When the python script is started for the first time, it immediately retrieves the offline sample information entered into the sample offline database during the process from the start time of the first start to the previous 4 hours, and records the start time of the first start t0.

[0010] In addition, the Python script mounted on the server background will scan the process status in each running cycle according to the preset time interval. When a new high-throughput sequencing data offline process is identified, the offline sample information in the sample offline database is entered, and the start time t of this startup is recorded. n , and it will also automatically trigger the task of synchronizing the sample information to the database; each operation cycle refers to the time from the previous startup to the start time t n-1 To the start time t of this startup n , no previous start time t n-1 , then get the start time t of this startup n Up to 4 hours before;

[0011] Step 2: Sample information in the capture process:

[0012] Use the python script to capture the new sample information obtained from the sample database in step 1. Each sample information includes: download name, analysis name, project number, project type, service name, download path, etc.

[0013] The name of the sample when it is sent for testing.

[0014] The analytical name refers to the name of the sample at the time of analysis;

[0015] The project number mentioned refers to the project number corresponding to the sample;

[0016] The project type refers to the type corresponding to the sample project number (corresponding to the process called by the subsequent analysis, for example, the dictionary records the whole transcriptome sequencing: whole transcriptome sequencing git, the dictionary is used to clarify the process used for subsequent analysis, the settings in the process are used to obtain the CPU and mem expected to be consumed by the process, and the real-time CPU and mem of the current running node (multiple nodes) are obtained to determine the subsequent download path and analysis path);

[0017] The service name refers to the service name corresponding to the sample project number (related to the contract signed for the project);

[0018] The download path refers to the download address corresponding to the sample provided by the partner (generally only provided to the directory level);

[0019] The captured sample information is instantiated into projects. Based on the projects, the corresponding project types and real-time CPU and MEM information of multiple nodes are obtained to determine the node path for subsequent download analysis.

[0020] Step 3: Upload sample information to the MySQL database:

[0021] The sample information captured in step 2 is organized into a fixed table, and Python is used to connect to the MySQL database. After the database synchronization task is triggered, the organized sample information is uploaded to the MySQL database in the form of a table, and the download status is marked as waiting for download through the Python script; the table is a self-built table, and the information that needs to be included in the table is determined according to the actual needs of the scientific research and production process.

[0022] The MySQL database includes two tables: the RawData table and the Projects_download table. The RawData table is used to record the downloaded data information and the sample download status. The table header includes the serial number, sample, project analysis directory, sample download data path, download command, download status, etc. The Projects_download table is used to record project-related information. The table header includes the project analysis directory, operation status information record file, download start time, download end time, completion time, deletion status, etc.

[0023] Step 4: Automatically download samples marked as wait in the database:

[0024] The automatic download task set in the process scans the database cyclically at a fixed time interval. When the status of the sample information scanned in the MySQL database is wait, the automatic download of the sample is automatically triggered; the fixed time interval can be set according to actual needs, and can generally be set to scan once every 30 minutes;

[0025] Generate the corresponding download command according to the download path in the sample download information, update it to the database, deliver the command line through the Snakemake process, use the mirror to call different download tools for downloading at the same time, and no conflict occurs. It can accommodate up to 10 download commands to run in parallel. In this step, for some samples that require special processing, such as samples that need to be split, the corresponding split code can be generated by identifying the corresponding 'need to split' (yes or no) field in the database to perform Snakemake delivery and splitting; the mirror refers to different download command running environments, specifically including multiple python environments after data transmission software and data preprocessing splitting.

[0026] Step 5: Modify the sample status after downloading starts in the database:

[0027] When the sample download is triggered, the Python script changes the download status of the corresponding sample in the database that has been triggered to download to run through the SQL statement.

[0028] Step 6: Modify the status of the downloaded sample in the database

[0029] After the high-throughput sequencing data download task of the sample is completed, the Python script connects to the MySQL database and changes the download status of the sample to done, and waits to trigger subsequent analysis.

[0030] Step 7: Send the merged and downloaded summary email:

[0031] When all samples in a certain operation process are downloaded, the Python script automatically merges the samples and sends a summary email of the download status to the task monitor. The summary information includes: the number of samples, the download time, the storage occupied by the downloaded data, etc.

[0032] Step 8: After the cycle of steps 2 to 7 is completed, pause for 30 seconds and start the next cycle. The starting time of the next cycle is t n , read the last startup start time t n-1 , query t n-1 to n The sample information of the sample disembarkation in the sample disembarkation database is entered within the time period. If there is no new information entered during this time period, it will not be recorded. n If there is new data in this time period, update t n-1 t n , and so on.

[0033] After the sample data is downloaded, it can be used for subsequent analysis of the sample data. After the project analysis is completed, the completion time information is uploaded to the database. After the completion information is identified, the sample data of the completed project is deleted.

[0034] In a preferred embodiment of the present invention, the process triggering is fully automated through a python script to monitor the process status without manual intervention.

[0035] In a preferred embodiment of the present invention, the sample information is uploaded to the MySQL database using a Python script to connect the data, and the sample information in the summarized process is synchronized to the database.

[0036] The present invention also provides application of the above method in downloading high-throughput sequencing data.

[0037] The present invention also provides a data download system for realizing the method of fast batch automatic downloading of high-throughput sequencing data, the data download system comprising: an instantiated sample and project attribute calling module, a download verification module, a download implementation module, a sample information acquisition module, a download review module, an email sending module, and a log generation module;

[0038] The instantiated sample and project attribute calling module is implemented through the Sample function and the Project function. Sample and Project are instantiated samples and project attributes respectively. It is more convenient and quick to call the required attributes in the whole process, and the cycle is accelerated through multi-process (concurrent.futures.ThreadPoolExecutor);

[0039] The download verification module is implemented through the CHECK function to verify the sample download status during the process;

[0040] The download implementation module is implemented through the DELIVER function, which generates a download command in the process and delivers it to Snakemake for downloading;

[0041] The sample information acquisition module is implemented through the ORGANIZE function, which obtains the sample information during the process and standardizes its data content and type;

[0042] The download review module is implemented through the download_check function. In the process, the consistency of the sample offline database and the sample download database information is traversed at fixed time intervals (for example, 24 hours) to prevent sample omissions due to server freezes and other reasons;

[0043] The email sending module is implemented through the EmailSender function, and sends emails to the corresponding person in charge in the process. When the email sending module is called, different recipients and email bodies are specified;

[0044] The log generation module is implemented through the Log function, and outputs a successful or error log file in the process. When the above module is called, it can be output to the 'year-month-day.log' file in the specified path according to the current time.

[0045] The present invention also provides the above-mentioned method for automatically downloading high-throughput sequencing data, or the application of the above-mentioned data download system in situations related to data file merging and sorting, such as high-throughput sequencing quality control analysis, sequencer data download and transmission, and can flexibly determine the remaining operating resources of the node to maximize the use of resources.

[0046] The beneficial effects of the present invention include:

[0047] The present invention uses Snakemake code in combination with MySQL database to integrate the merged download process of high-throughput sequencing data, transmits information to the MySQL database in real time through a python script, and updates the download status of samples in the database to wait, run and done in real time according to the running status of the download task, including waiting for download, downloading and download completion, and ensures the seamless connection of the process operation through automated tasks. Snakemake mirror delivery is used to run multiple different or identical download commands at the same time, ensuring that the downloads do not interfere with each other while speeding up the operation efficiency; at the same time, multi-threading can be used to batch process samples in multiple processes, and after the data is downloaded, the python package smtplib is used to automatically send emails to quickly feedback the download summary information, and the task operation status and abnormal information can be quickly obtained in combination with the sample download status in the MySQL database.

[0048] The present invention operates with high efficiency, and the entire process including process triggering, information uploading to a database, data downloading, download status updating, and sending summary files, etc., is fully automated without human involvement.

[0049] The present invention can check the project progress in real time and delete the completed data in time without human intervention, thus avoiding the extra costs caused by accidental deletion, untimely deletion and the like.

[0050] The various steps described in the present invention are automatically triggered, and there is no waiting time between the steps, which greatly shortens the operation cycle of the entire process.

[0051] All operation steps of the present invention can be traced. If the data download is wrong or abnormal, snakemake will output the error log information of the corresponding step. The sample status information can be obtained by querying the error log information.

[0052] At the same time, the running speed of the multi-process, multi-sample parallel downloading method used in the present invention is nearly 10 times shorter than the conventional single-process multi-sample downloading time. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 It is a schematic diagram of the process of the present invention.

[0055] Figure 2 The figure is a schematic diagram of summarizing sample information in the sample information crawling process according to an embodiment of the present invention.

[0056] Figure 3 This is a schematic diagram of a sample downloading schematic diagram in modifying a database state in an embodiment of the present invention.

[0057] Figure 4 This is a schematic diagram of the sample downloading running state in modifying the database state in an embodiment of the present invention.

[0058] Figure 5 This is a schematic diagram of the sample download completion state in the database modification state according to an embodiment of the present invention.

[0059] Figure 6 This is a diagram showing the snakemake process for automatically downloading data from samples marked as wait in a database according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0061] The present invention provides a method for fast batch automatic downloading of high-throughput sequencing data based on snakemake language and MySQL database, and the method specifically includes the following steps: process triggering; capturing sample information in the process; uploading sample information to MySQL database; automatically downloading samples marked as wait in the database; modifying database status; synchronizing sample information to MySQL database after downloading; modifying database status; sending a summary email of downloads; deleting sample data of completed projects. The method of the present invention can synchronize and download sample information in batches, and the whole process does not require human intervention, and can automatically trigger various links in the middle. During the processing process, the download summary information of the sample data will be automatically sent to the monitor in a timely manner to ensure that all operation steps can be traced, and various information queries are convenient with the help of MySQL database.

[0062] The present invention provides a high-throughput sequencing quality control analysis method based on snakemake language for rapid batch automatic email feedback of results. The process of the method is as follows: Figure 1 As shown, the specific steps include:

[0063] Step 1: Process trigger:

[0064] When the python script is started for the first time, it immediately retrieves the offline sample information entered into the sample offline database during the process from the start time of the first start to the previous 4 hours, and records the start time of the first start t0.

[0065] Step 2: Sample information in the capture process:

[0066] Use the python script to capture the new sample information obtained from the sample database in step 1. The information of a single sample includes: the sample's name, sample analysis name, sample project number, sample project type, sample service name, sample download path, etc.

[0067] Step 3: Upload sample information to the MySQL database:

[0068] The sample information captured from the process is organized into a fixed table, and Python is used to connect to the MySQL database. After the database synchronization task is triggered, the organized sample information is uploaded to the MySQL database in the form of a table, and the download status is marked as waiting for download.

[0069] Step 4: Automatically download samples marked as wait in the database:

[0070] The automatic download task set in the process scans the database every 30 minutes. When the status of the sample information scanned in the MySQL database is wait, the automatic download of the sample is automatically triggered.

[0071] Step 5: Modify the sample status after downloading starts in the database:

[0072] When the sample download is triggered, the Python script changes the download status of the corresponding sample in the database that has been triggered to download to run.

[0073] Step 6: Modify the status of the downloaded sample in the database:

[0074] After the high-throughput sequencing data download task of the sample is completed, the Python script connects to the MySQL database and changes the download status of the sample to done, and waits to trigger subsequent analysis.

[0075] Step 7: Send the merged and downloaded summary email:

[0076] When all samples in the process are downloaded, the Python script automatically merges the samples and sends a summary email of the download status to the task monitor. The summary information includes: the number of samples, the download time, the storage occupied by the downloaded data, etc.

[0077] Step 8: After the cycle of steps 2 to 7 is completed, pause for 30 seconds and start the next cycle. The starting time of the next cycle is t n , read the last startup start time t n-1 , query t n-1 to n The sample information of the sample disembarkation in the sample disembarkation database is entered within the time period. If there is no new information entered during this time period, it will not be recorded. n If there is new data in this time period, update t n-1 t n , and so on.

[0078] After the sample download analysis of each process is completed, the sample data of the completed project will be deleted:

[0079] After the project analysis is completed, the completion time information is uploaded to the database, the completion information is identified, and the data of the corresponding sample is deleted.

[0080] Example

[0081] Taking samples A1, A2, A3, A4, and A5 in the XMXD-202307276934 process as an example, the process of the present invention is described:

[0082] 1. The first time the process is started, the ORGANIZE module is called to receive the data from the current time t1 to the previous 4 hours containing the sample information of A1, A2, A3, A4, and A5. The offline process XMXD-202307276934, for sample information, see Figure 2 , including serial number, sample, order number, 8-row serial number, database construction date, sales order number, task number, sequencing number, number, serial number, whether completed and confirmed, Sample, project type, target data volume, order data volume, output data volume, Reads (M), Q30 (%); use python code to organize sample information, find the corresponding sample download path, generate the corresponding download command and upload it to the MySQL database, and mark the sample download status as wait, see Figure 3 ;

[0083] 2. The second startup calls the DELIVER module to trigger the download task to automatically deliver Snakemake to download the A1, A2, A3, A4, and A5 sample data, and updates the sample download status to run in real time. Figure 4 , Figure 6 At the same time, use the EmailSender module to send a summary email of data downloads; call the ORGANIZE module again to search for database information and search for information from t1 to the current time t2 to repeat the process in 1;

[0084] Figure 4 The sample information of the database, including the download status of the sample;

[0085] Figure 6 Snakemake flow chart for automated downloading;

[0086] 3. The third time the CHECK module is called to output the log to detect the download status of the A1, A2, A3, A4, and A5 sample data. After the download is complete, the download status of the sample is updated to done in real time. Figure 5 ; Call the DELIVER module to trigger the download task to automatically deliver the Snakemake download sample. Download the sample whose status is marked as wait and repeat the process in 2; Call the ORGANIZE module to search the database information to search for information from t2 to the current time t3 and repeat the process in 1;

[0087] 4. After the subsequent analysis of the data is completed, the project status is marked as completed, which automatically triggers the data deletion task in the CHECK module to delete the sample data associated with the completed project.

[0088] Comparative Example

[0089] 1. In the prior art, the path and data information to be downloaded need to be obtained manually through emails;

[0090] 2. In the prior art, it is necessary to manually configure obsutil, ossutil or other download software parameters to download data. At the same time, due to the download speed limit, a maximum of five download tasks can be performed at the same time;

[0091] 3. In the prior art, it is necessary to organize and merge data based on the provided information and transmit it to a designated location for analysis.

[0092] In the present invention, for 1, these download paths, data information and other information can be automatically obtained without checking emails. For 2, different download methods can be automatically adjusted through preset values. The download speed is also limited, but no manual intervention is required. The undownloaded data can be automatically downloaded. For 3, the data can be merged before downloading, and a single data can be downloaded to reduce the risk of interruption of large data transmission due to network fluctuations. The data can be directly downloaded to the designated location and connected with the subsequent automated analysis, so there is no need to upload it to other locations separately.

[0093] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A method for fast batch automatic downloading of high-throughput sequencing data based on snakemake language and MySQL database, characterized in that: The method specifically comprises the following steps: Step 1: When the python script is started for the first time, the information of the samples that have been logged into the database of the samples logged in the process from the start time of the first startup to the previous 4 hours is immediately retrieved, and the start time of the first startup, t0, is recorded at the same time; Step 2: Use a python script to capture the sample information obtained in step 1, instantiate the sample information into a project, obtain the corresponding project type, the real-time cpu and mem information of multiple nodes, and determine the node path for subsequent download analysis. Step 3: Arrange the sample information captured in step 2 into a fixed table, use Python to connect to the MySQL database, upload the arranged sample information in the form of a table to the MySQL database, and mark the download status as waiting for download; Step 4: The preset automatic download task scans the database at a fixed time interval. When the sample information scanned in the MySQL database is wait, the automatic download of the sample is automatically triggered; In step 4, the automatic triggering of the download task is realized by a python script; the fixed time interval is 30 minutes; In step 4, the corresponding download command is generated according to the download path in the sample download information, updated to the database, and the command line is delivered through the Snakemake process. The image is used to call different download tools to download samples in parallel; the image refers to the transmission data software and the python environment for running multiple different download commands after data preprocessing and splitting; Step 5: When the sample download is triggered, the Python script changes the download status of the sample in the database that has been triggered to download to run; Step 6: After the high-throughput sequencing data download task of the sample is completed, the Python script connects to the MySQL database and changes the download status of the sample to done, and waits for the subsequent analysis to be triggered; Step 7: When all samples in the process are downloaded, the Python script automatically sends a summary email of the sample download status to the task monitor; Step 8: After the cycle of steps 2 to 7 is completed, pause for 30 seconds and start the next cycle. The starting time of the next cycle is t n , read the last startup start time t n-1 , query t n-1 to n The sample information of the sample disembarkation in the sample disembarkation database is entered within the time period. If there is no new information entered during this time period, it will not be recorded. n , if there is new data entered during this time period, update t n-1 t n .

2. The method for automatically downloading high-throughput sequencing data according to claim 1, characterized in that: In step 1, the python script is mounted on the server background, scans the process status once in each operation cycle, and records the start time t after identifying the new high-throughput sequencing data offline process. n , and will automatically trigger the task of synchronizing the sample information to the database; each operation cycle refers to the time from the previous startup to the start time t n-1 To the start time t of this startup n , no previous start time t n-1 , then get the start time t of this startup n Up to 4 hours ago.

3. The method for automatically downloading high-throughput sequencing data according to claim 1, characterized in that: In step 2, the sample information includes the name of the download, the name of the analysis, the project number, the project type, the service name, and the download path; wherein, The name of the sample when it is sent for testing. The analytical name refers to the name of the sample at the time of analysis; The project number mentioned refers to the project number corresponding to the sample; The project type refers to the type corresponding to the sample project number; The service name refers to the service name corresponding to the sample project number; The download path refers to the download address corresponding to the sample provided by the partner.

4. The method for automatically downloading high-throughput sequencing data according to claim 1, characterized in that: In step 3, the MySQL database includes 2 tables: RawData and Projects_download. The RawData table is used to record the downloaded data information and sample download status. The table header includes the serial number, sample, project analysis directory, sample download data path, download command, and download status. The Projects_download table is used to record project related information. The table header includes the project analysis directory, operation status information record file, download start time, download end time, completion time, and deletion status. In step 3, the download status is marked by the python script.

5. The method for automatically downloading high-throughput sequencing data according to claim 1, characterized in that: In step 7, the automatic sending of emails is implemented by the Python script that organizes the sample data download information and is implemented by the Python package smtplib.

6. The method for automatically downloading high-throughput sequencing data according to claim 1, characterized in that: The method further includes: analyzing the project instantiated with the sample data after the sample data is downloaded, uploading the completion time information to the database after the project analysis is completed, and deleting the sample data of the completed project after the completion information is identified.

7. Application of the method for automatically downloading high-throughput sequencing data as described in any one of claims 1 to 6 in data file merging and sorting including high-throughput sequencing quality control analysis and sequencer data download and transmission.

8. A data downloading system for implementing the method for automatically downloading high-throughput sequencing data as claimed in any one of claims 1 to 6, characterized in that: The data download system includes: instantiated sample and project attribute calling module, download verification module, download implementation module, sample information acquisition module, download review module, email sending module, log generation module; The instantiation sample and project attribute calling module is implemented through the Sample function and the Project function, and the required attributes are called in the whole process; The download verification module is implemented through the CHECK function to verify the sample download status during the process; The download implementation module is implemented through the DELIVER function, which generates a download command in the process and delivers it to Snakemake for downloading; The sample information acquisition module is implemented through the ORGANIZE function, which obtains the sample information during the process and standardizes its data content and type; The download review module is implemented by the download_check function, which checks the consistency of the sample offline database and the sample download database information at fixed time intervals in the process; The email sending module is implemented through the EmailSender function, which sends emails to the corresponding person in charge in the process. When the email sending module is called, different recipients and email bodies can be specified; The log generation module is implemented through the Log function, and outputs a successful or error log file in the process. When the log generation module is called, it is output to the 'year-month-day.log' file under the specified path according to the current time.

9. Application of the data download system as claimed in claim 8 in data file merging and sorting, including high-throughput sequencing quality control analysis, and sequencer data download and transmission.

Images