Parallel alignment method for analyzing large-scale third-generation sequencing data based on MPI technology

Abstract

This invention discloses a parallel alignment method for analyzing large-scale third-generation sequencing data based on MPI technology, comprising: S1, starting p processes, arbitrarily selecting one process as the master process, and the remaining processes as slave processes; S2, reading the reference genome into the master process, and generating a reference genome index using minimap2; S3, constructing a data structure based on the values ​​of the reference genome index; S4, the master process broadcasting the data structure constructed in step S3 using the MPI_Bcast function; S5, all slave processes calling the MPI_Bcast function to obtain the data structure broadcast by the master process; S6, segmenting all p processes to determine the third-generation sequencing data to be read and analyzed; S7, performing third-generation sequencing alignment on all processes through multi-level parallelism; S8, each process asynchronously inputting its alignment results into an independent file for storage; S9, merging the alignment results of each process stored in different files. This invention significantly reduces alignment time.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of bioinformatics, parallel programming, and computer applications, and more specifically, to a parallel alignment method for analyzing large-scale third-generation sequencing data based on MPI technology. Background Technology

[0002] DNA sequences are composed of four bases: A, T, C, and G. The order of these four bases represents the genetic information of an organism, and determining the base sequence in a biological sample is an indispensable part of a range of research applications. Over the past few decades, scientists have developed numerous sequencing technologies to determine the precise order of the four bases that make up the DNA chain and to unlock the genetic code of DNA samples. Continuous advancements in sequencing technology in terms of speed and cost reduction have greatly propelled the development of genomics. Unlike first- and second-generation sequencing, third-generation sequencing has increased the length of DNA sequence fragments by several orders of magnitude. However, the error rate of sequence data obtained using current third-generation sequencing technologies is approximately 15%, significantly higher than that of second-generation sequencing. Obtaining long-read sequence data has brought new breakthroughs to many challenges in the field of bioinformatics, greatly improving the quality of genome assembly and the analysis of genome structure.

[0003] Third-generation sequencing (NGS) has brought new opportunities, but it has also presented new challenges to sequence alignment technology. Sequence alignment is a very time-consuming process; the computational and time complexity of sequence alignment algorithms are directly proportional to the sequence length. The sequences generated by NGS are extremely long, making the sequence alignment process for NGS even more time-consuming. Furthermore, NGS's low sequencing cost and large data volume have resulted in massive amounts of long-read sequence data. Achieving fast and accurate alignment of large-scale sequences has become a major challenge for NGS long-read sequence alignment.

[0004] Therefore, developing efficient parallel programs for third-generation sequencing alignment and utilizing these tools for large-scale alignment of third-generation sequencing data is of great scientific significance. Summary of the Invention

[0005] The purpose of this invention is to provide a parallel alignment method for analyzing large-scale third-generation sequencing data based on MPI technology, so as to overcome the defects of existing technologies.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A parallel alignment method for analyzing large-scale third-generation sequencing data based on MPI technology includes the following steps:

[0008] S1. Start p processes, arbitrarily select one process as the master process, and use the remaining p-1 processes as slave processes;

[0009] S2. Read the reference genome into the main process and generate a reference genome index using minimap2;

[0010] S3. Construct a data structure based on the value of the reference genome index, and broadcast the length of each array in the data structure;

[0011] S4. The main process broadcasts the data structure constructed in step S3 through the MPI_Bcast function;

[0012] S5. All slave processes call the MPI_Bcast function to obtain the data structure broadcast by the master process, construct a genome index based on the data structure, and save it as a copy in the slave processes;

[0013] S6. Perform sequence segmentation on all p processes to determine the third-generation sequencing data to be read and analyzed;

[0014] S7. Perform third-generation sequencing alignment on all processes through multi-level parallelism;

[0015] S8. Each process asynchronously inputs the comparison results of this process into an independent file for storage;

[0016] S9. Merge the comparison results of each process stored in different files to obtain a comparison result file.

[0017] Furthermore, step S3 specifically includes the following steps:

[0018] S31. The main process determines the storage structure of the reference genome index, saves it to different arrays, and obtains the array length;

[0019] S32. The main process broadcasts the length of each array through the MPI_Bcast function, and the slave processes obtain the array length through the MPI_Bcast function and allocate corresponding memory space according to the obtained array length.

[0020] S33. The main process broadcasts the data of each array through the MPI_Bcast function, and the slave processes obtain the array data through the MPI_Bcast function and store it in the memory space allocated in step S32.

[0021] Furthermore, in step S33, the main process broadcasts only arrays of non-zero length using the MPI_Bcast function.

[0022] Furthermore, step S6, which involves sequentially segmenting all p processes, specifically includes the following steps:

[0023] S61. After all processes read the index or receive the index data, they then read in the length sequence for comparison.

[0024] S62. Process obtains the total query sequence file size. f Get the current total number of processes p, and then calculate the file size that each process needs to process.

[0025] S63, The starting position of the query sequence that the process computation needs to process. Where i represents the process ID, which starts from 0;

[0026] S64. The termination position of the query sequence that the process computation needs to process is: Where i represents the process ID, which starts from 0;

[0027] S65. The query file pointer of each process is moved to the starting position start determined in step S63. The process moves the pointer from the starting position start until it encounters the first sequence identifier ">" or "@", and takes this as the new starting position. Then, the MPI_Send function is used to send this position to the previous process.

[0028] S66. The process receives the position parameter passed from the next process by calling the MPI_Recv function, and adjusts the termination position end according to this parameter.

[0029] Furthermore, the step S7 of performing third-generation sequencing alignment for all processes specifically involves performing third-generation sequencing alignment after all processes have prepared their genome indexes and the sequences of the required lengths.

[0030] Furthermore, step S9 specifically includes: after all processes have finished inputting, merging the result files of each process using the cat command to obtain the final comparison result file.

[0031] Compared with existing technologies, the advantages of this invention are as follows: This invention provides a parallel alignment method for analyzing large-scale third-generation sequencing data based on MPI technology. It develops a multi-level parallel program through the Message Passing Interface (MPI), parallelizing the traditional large-scale third-generation sequencing method minimap2, significantly reducing alignment time from days or even weeks to minutes or hours. The program adopts a master-slave node model. First, the master node is responsible for reading and constructing the reference genome index and dividing it into arrays according to the structure of the genome index. Then, it broadcasts the arrays to the slave processes through the MPI function MPI_Bcast, and the slave processes store copies of the reference genome index. After all processes have read the reference genome index, the alignment sequences are partitioned. To balance the load, each process partitions the sequence files by dividing the file size by the number of processes, and then adjusts the start and end positions according to the start identifier of the genome sequence to ensure the correctness of the alignment. After partitioning the alignment sequences, the master and slave processes perform third-generation sequencing alignment together. To prevent write conflicts, each process has an independent output file, outputting asynchronously to improve I / O efficiency. Finally, the alignment files can be merged using the cat command to obtain the final results. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart of the parallel alignment method for analyzing large-scale third-generation sequencing data based on MPI technology, as described in this invention.

[0034] Figure 2 This invention represents the speedup of the parallel alignment method for analyzing large-scale third-generation sequencing data based on MPI technology.

[0035] Figure 3 This invention relates to the efficiency of a parallel alignment method for analyzing large-scale third-generation sequencing data based on MPI technology. Detailed Implementation

[0036] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0037] See Figure 1As shown, this embodiment discloses a parallel alignment method for analyzing large-scale third-generation sequencing data based on MPI technology, including the following steps:

[0038] Step S1: Start p processes, arbitrarily select one process as the master process, and use the remaining p-1 processes as slave processes.

[0039] Step S2: Read the reference genome into the main process and generate a reference genome index using minimap2.

[0040] Step S3: Construct a data structure based on the value of the reference genome index, and broadcast the length of each array in the data structure.

[0041] Specifically, step S3 may include:

[0042] Step S31: The main process determines the storage structure of the reference genome index, saves it to different arrays, and obtains the array length.

[0043] Step S32: The main process broadcasts the length of each array through the MPI_Bcast function, and the slave processes obtain the array length through the MPI_Bcast function and allocate corresponding memory space according to the obtained array length.

[0044] Step S33: The main process broadcasts array data to each process using the MPI_Bcast function, and the slave processes retrieve the array data and store it in the memory space allocated in step S32 using the MPI_Bcast function. To reduce communication, the main process only broadcasts arrays of non-zero length using the MPI_Bcast function.

[0045] Step S4: The main process broadcasts the data structure constructed in step S3 through the MPI_Bcast function.

[0046] Step S5: All slave processes call the MPI_Bcast function to obtain the data structure broadcast by the master process, construct a genome index based on the data structure, and save it as a copy in the slave processes.

[0047] Step S6: Perform sequence segmentation on all p processes to determine the third-generation sequencing data to be read and analyzed.

[0048] Specifically, step S6, which involves sequentially segmenting all p processes, includes:

[0049] Step S61: After all processes read the index or receive the index data, they then read in the length sequence for comparison.

[0050] Step S62: Obtain the total query sequence file size. fGet the current total number of processes p, and then calculate the file size that each process needs to process.

[0051] Step S63: The process calculates the starting position of the query sequence to be processed. Where i represents the process ID, which starts from 0.

[0052] Step S64: The process calculates the termination position of the query sequence to be processed. Where i represents the process ID, which starts from 0.

[0053] Step S65: The query file pointer of each process is moved to the starting position determined in step S63. In order to ensure the correctness of the query results, the process moves the pointer from the starting position until it encounters the first sequence identifier ">" or "@", and takes this as the new starting position. Then, the MPI_Send function is used to send this position to the previous process.

[0054] Step S66: The process receives the position parameter passed from the next process by calling the MPI_Recv function, and adjusts the termination position end according to this parameter.

[0055] Step S7: Perform third-generation sequencing alignment on all processes through multi-level parallel processing.

[0056] Specifically, performing third-generation sequencing alignment for all processes involves performing third-generation sequencing alignment after all processes have prepared their genome indexes and the sequences of the required lengths.

[0057] Step S8: Each process asynchronously inputs the comparison results of the process into an independent file for saving.

[0058] Specifically, since the number of processes may be large, in order to avoid write conflicts, each process has its own output file and only needs to output the comparison results to its own result file.

[0059] Step S9: Merge the comparison results of each process stored in different files to obtain a comparison result file.

[0060] Specifically, step S9 involves merging the result files of each process after all processes have finished inputting data, using the cat command to obtain the final comparison result file.

[0061] This invention develops a multi-level parallel program using a Message Passing Interface (MPI) to parallelize the traditional large-scale third-generation sequencing method minimap2, significantly reducing alignment time from days or even weeks to minutes or hours. The program employs a master-slave node model. The master node is responsible for reading and constructing the reference genome index, dividing it into arrays based on the index structure, and then broadcasting these arrays to slave processes via the MPI function MPI_Bcast. Slave processes store copies of the reference genome index. After all processes have read the reference genome index, sequence partitioning is performed. To balance the load, each process partitions the sequence files by dividing the file size by the number of processes, and then adjusts the start and end positions based on the start identifier of the genome sequence to ensure alignment accuracy. After partitioning the alignment sequences, the master and slave processes perform third-generation sequencing alignment together. To prevent write conflicts, each process has an independent output file, outputting asynchronously to improve I / O efficiency. Finally, the alignment files can be merged using the cat command to obtain the final results. Figure 2 As shown, the speedup gradually approaches linear speedup as the data size increases. The maximum speedup reached 52.753 on a 61GB dataset. Figure 3 As shown, the acceleration efficiency is stable. From 2 processes to 64 processes in parallel, the program running efficiency decreases relatively smoothly, and the program parallel efficiency is high.

[0062] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, the patent owner may make various modifications or alterations within the scope of the appended claims, as long as they do not exceed the protection scope described in the claims of the present invention, they shall be within the protection scope of the present invention.

Claims

1. A parallel alignment method for analyzing large-scale third-generation sequencing data based on MPI technology, characterized in that, Includes the following steps: S1. Start p processes, arbitrarily select one process as the master process, and use the remaining p-1 processes as slave processes; S2. Read the reference genome into the main process and generate a reference genome index using minimap2; S3. Construct a data structure based on the value of the reference genome index, and broadcast the length of each array in the data structure; S4. The main process broadcasts the data structure constructed in step S3 through the MPI_Bcast function; S5. All slave processes call the MPI_Bcast function to obtain the data structure broadcast by the master process, construct a genome index based on the data structure, and save it as a copy in the slave processes; S6. Perform sequence segmentation on all p processes to determine the third-generation sequencing data to be read and analyzed; S7. Perform third-generation sequencing alignment on all processes through multi-level parallelism; S8. Each process asynchronously inputs the comparison results of this process into an independent file for storage; S9. Merge the comparison results of each process stored in different files to obtain a comparison result file; Step S3 specifically includes the following steps: S31. The main process determines the storage structure of the reference genome index, saves it to different arrays, and obtains the array length; S32. The main process broadcasts the length of each array through the MPI_Bcast function, and the slave processes obtain the array length through the MPI_Bcast function and allocate corresponding memory space according to the obtained array length. S33. The main process broadcasts the data of each array through the MPI_Bcast function, and the slave processes obtain the array data through the MPI_Bcast function and store it in the memory space allocated in step S32. Step S6, which involves sequentially segmenting all p processes, specifically includes the following steps: S61. After all processes read the index or receive the index data, they then read in the length sequence for comparison. S62. Process obtains the total query sequence file size. f Get the current total number of processes p, and then calculate the file size that each process needs to process. S63, The starting position of the query sequence that the process computation needs to process. Where i represents the process ID, which starts from 0; S64. The termination position of the query sequence that the process computation needs to process is: Where i represents the process ID, which starts from 0; S65. The query file pointer of each process is moved to the starting position start determined in step S63. The process moves the pointer from the starting position start until it encounters the first sequence identifier ">" or "@", and takes this as the new starting position. Then, the MPI_Send function is used to send this position to the previous process. S66. The process receives the position parameter passed from the next process by calling the MPI_Recv function, and adjusts the termination position end according to this parameter.

2. The method for parallel alignment of large-scale third-generation sequencing data based on MPI technology according to claim 1, characterized in that: In step S32, the main process broadcasts only arrays of non-zero length using the MPI_Bcast function.

3. The method for parallel alignment of large-scale third-generation sequencing data based on MPI technology according to claim 1, characterized in that, The specific steps of performing third-generation sequencing alignment for all processes in step S7 are as follows: after all processes have prepared their genome indexes and the sequences of the required lengths, third-generation sequencing alignment is performed.

4. The method for parallel alignment of large-scale third-generation sequencing data based on MPI technology according to claim 1, characterized in that, Step S9 specifically includes: after all processes have finished inputting, merging the result files of each process using the cat command to obtain the final comparison result file.

Images