A multi-threaded method and system for gene assembly

Abstract

The present invention discloses a multithreaded method and system for gene assembly, which includes the following steps: S1: generating an ID docking table; S2: using a B+ tree index to extract base sequences corresponding to numbered read IDs from a hard disk or virtual memory; S3: matching multiple sequences after the docking relationship with a reference sequence in sequence; S4: reading base sequences of the next batch of numbered read IDs that can match the reference sequence; S5: outputting thread identifiers and matching small contigs after merging the output; S6: after the ID docking table is read, traversing the ID docking table, replacing the base sequences of scattered numbered read IDs in the thread identifier table with corresponding base sequence IDs, and outputting a single nucleotide variant (SNP) information table. The present invention performs calculations by forming a server cluster, using one of the servers as shared storage, and allowing the other servers to individually perform tasks, thereby achieving multi-machine parallel computing on the server, increasing the linear performance of the assembly algorithm, and infinitely reducing the calculation time.

Description

Technical Field

[0001] The present invention belongs to the technical field of bioinformatics, and in particular relates to a multi-threaded method and system for gene assembly. Background Art

[0002] Since Roche launched its first second-generation sequencer, the Roche 454, in 2005, life sciences have officially entered the era of high-throughput sequencing. The introduction of Illumina's sequencing platforms has significantly reduced the price of second-generation sequencing, making high-throughput sequencing widely available across various life science research fields. To this day, second-generation short-read sequencing technology maintains a dominant position in the global sequencing market.

[0003] In the field of bioinformatics, genome assembly is a core issue in genome analysis. Although second-generation sequencing technology has greatly increased sequencing throughput, the individual sequences it obtains are very short, often only 50 to 300 base pairs. Furthermore, due to the deeper sequencing coverage, the number of sequences that need to be processed in genome projects has increased rapidly. These shorter individual sequences need to be spliced ​​into longer, complete sequences for further analysis to improve the accuracy of species annotation analysis.

[0004] Due to the limitations of assembly technology and the complexity of actual conditions, current assembly algorithms usually require a server with extremely high memory and CPU configurations. When assembling species genes from scratch, only one server can be used for calculations, which requires enormous time and computing power costs. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-threaded method and system for gene assembly to achieve linear performance increase of the assembly algorithm, form a server cluster for calculation, use one server as shared storage, and the other servers each take on tasks for multi-threaded parallel execution, thereby promoting bioinformatics analysis methods to formally enter the era of server cluster computing, formally realize multi-machine parallel computing on servers, and infinitely reduce computing time.

[0006] The object of the present invention is achieved by adopting the following technical solution: a multi-threaded method for gene assembly, comprising the following steps:

[0007] S1: Assign custom numbers to the sequencing files and use the neural network fuzzy matching algorithm to obtain the corresponding relationship between the base sequences, i.e., the ID docking table;

[0008] S2: Start multiple threads. Each thread reads the corresponding relationship in the ID docking table row by row, puts the thread identifier in front of the row, and uses the B+ tree index to extract the base sequence corresponding to the read ID from the hard disk or virtual memory respectively.

[0009] S3: Match the multiple sequences after docking with the reference sequence in sequence, merge those that can be aligned, and discard those that cannot be aligned;

[0010] S4: Continue reading the base sequences of the next batch of read IDs that can match the reference sequence, and extract the corresponding sequences to match them again with the reference sequence of this batch;

[0011] S5: If a thread has already started computing the matches for the next batch of sequences, then the extraction of the next batch of sequences is terminated, and the thread identifier and the small contig output after the matching merge are output;

[0012] S6: If no thread has started calculating the matching of the next batch of sequences, continue to extract the next batch of sequences that match the benchmark sequence and match them with the benchmark sequence of this batch until one of the sequences in the batch has been calculated by other threads. Then terminate this calculation and output the thread identifier and the small contig after the matching merge output;

[0013] S7: After the ID docking table is read, the ID docking table is traversed, the base sequences of the scattered read IDs in the thread identification table are replaced with the corresponding base sequence IDs, and the single nucleotide variation SNP information table is output.

[0014] Furthermore, step S1 specifically includes: using a neural network fuzzy matching algorithm to output the front-to-back matching correspondence between the sequencing fragment reads in the form of an ID docking table, wherein the expression of the ID docking table is ab, c, d, wherein a, b, c, and d represent base sequence IDs, and the meaning of the expression is: there is a front-to-back alignment matching relationship between sequence a and sequences b, c, and d.

[0015] Furthermore, the step S2 specifically includes: when the thread starts calculation, a label is added before the number read ID in the ID docking table included in the thread range, indicating that the base sequences 1-n are matched one by one. When the first thread performs calculation and matches No. 1 with other docking base sequences, a label is added before the first row of the ID docking table as the thread identification number, indicating the first calculation of the first thread, and the corresponding sequence is extracted according to the number read ID based on the B+ tree index.

[0016] Furthermore, step S3 specifically includes: matching the multiple sequences after the docking relationship with the reference sequence in sequence; when the overlapping part of the sequence after the docking relationship and the reference sequence has 80% or more completely matched bases, it is considered that the sequence has an overlap region and can be merged into one sequence; when the number of completely matched bases in the overlapping part of the sequence after the docking relationship and the reference sequence is less than 80%, the sequence is discarded.

[0017] Furthermore, step S4 specifically includes: when the sequence to be tested can match the reference sequence, continue to search the ID docking table of the sequence to be tested, and extract the sequence number read ID corresponding to the sequence to be sequenced. This batch uses the sequence to be tested as the reference, and other sequences are matched with the sequence to be tested in turn.

[0018] Furthermore, the step S5 specifically includes: when the sequence to be tested can match the reference sequence, continue to search the ID docking table of the sequence to be tested; if the thread identification number of the sequence to be tested has been calculated before the ID docking table, stop the calculation, and output the thread identification number calculated this time and the base sequence of the reference sequence corresponding to the identification; the sequence identification of the small overlapping group samllcontig is recorded as contig 1-1.

[0019] Furthermore, step S6 specifically includes: after the calculation of sequencing fragment reads No. 1 is completed, the first thread starts the second calculation, searches for sequencing fragment reads that are not identified in the ID docking table, and marks the thread identifier in front of them. If there are other sequences that can match the unidentified sequence to be tested, and none of them are identified by other threads, then the base sequences docked with the other sequences are extracted at the same time and spliced ​​together until each sequence is retrieved with an identifier, the calculation is terminated, and the spliced ​​base sequence is output. The small contig sequence of the small contig is identified as contig 1-2, and so on.

[0020] Furthermore, step S7 specifically includes: after all ID docking tables are calculated, all ID docking tables marked with thread identifiers are retrieved, and the output thread identifier table is optimized, wherein the thread identifier also represents the number of the output small contig. When the sequencing fragment reads are extracted for one-to-one matching, the site information of the single nucleotide variation SNP can be output: the identifier of the small contig, the full length, the single nucleotide variation SNP variation base site, the base of the output contig at the variation site, and the base information replaced after the mutation.

[0021] A multi-threaded system for gene assembly includes an ID docking table generation module and a reads multi-threaded matching module, wherein the ID docking table generation module is used to achieve the correspondence between base sequences with front-to-back matching alignment relationships; the reads multi-threaded matching module is used to achieve multi-threaded sequencing fragment read matching and assemble them into small contigs.

[0022] The beneficial effects of the present invention are as follows: the present invention realizes multi-machine parallel computing by forming a server cluster for computing, using one of the servers as shared storage, and the other servers each taking on tasks to execute, thereby achieving linear performance increase of the assembly algorithm, infinitely reducing computing time, and promoting bioinformatics analysis methods to formally enter the era of server cluster computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0024] Figure 1 A flowchart for the invention;

[0025] Figure 2 This is a schematic diagram of head alignment;

[0026] Figure 3 This is a schematic diagram of tail alignment or staggered alignment;

[0027] Figure 4 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0029] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0030] The following embodiments of the present invention are described in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0031] Example 1:

[0032] See Figure 1 , a multithreaded method for gene assembly, comprising the following steps:

[0033] S1: Assign custom numbers to the sequencing files and use the neural network fuzzy matching algorithm to obtain the corresponding relationship between the base sequences, i.e., the ID docking table;

[0034] S2: Start multiple threads. Each thread reads the corresponding relationship in the ID docking table row by row, puts the thread identifier in front of the row, and uses the B+ tree index to extract the base sequence corresponding to the read ID from the hard disk or virtual memory respectively.

[0035] S3: Match the multiple sequences after docking with the reference sequence in sequence, merge those that can be aligned, and discard those that cannot be aligned;

[0036] S4: Continue reading the base sequences of the next batch of read IDs that can match the reference sequence, and extract the corresponding sequences to match them again with the reference sequence of this batch;

[0037] S5: If the matching of the next batch of sequences has already started calculation by a thread (that is, there is a thread identifier in front of the ID docking table), then the extraction of the next batch of sequences is terminated, and the thread identifier and the small contig output after the matching and merging are output;

[0038] S6: If there is no thread started to calculate the matching of the next batch of sequences (that is, there is no thread identifier in the ID matching table), continue to extract the next batch of sequences that match the benchmark sequence and match them with the benchmark sequence of this batch until one of the sequences in the batch has been started to calculate by other threads. Then terminate this calculation and output the thread identifier and the small contig output after the matching is merged.

[0039] S7: After the ID docking table is read, the ID docking table is traversed, the base sequences of the scattered read IDs in the thread identification table are replaced with the corresponding base sequence IDs, and the single nucleotide variation SNP information table is output.

[0040] In this embodiment, the step S1 specifically includes: using a neural network fuzzy matching algorithm to output the front-to-back matching correspondence between reads in the form of an ID docking table, similar to 1-3, 9, 18, 66, 77, 105, where the numbers represent the base sequence IDs, and the meaning of this row of the docking table is: there is a front-to-back alignment matching relationship between sequence 1 and sequences 3, 9, 18, 66, 77, and 105. Further, step S1 can be regarded as generating an ID docking table, the purpose of which is to obtain the correspondence between overlapping base sequences. Using a neural network fuzzy matching algorithm, the front-to-back matching correspondence between reads is output in the form of an ID docking table.

[0041] In this embodiment, step S2 specifically includes: when the thread starts calculation, a label is added before the number read ID in the ID docking table included in the thread range, indicating that the base sequences 1-n are matched one by one. When the first thread performs calculation and matches number 1 with other docking base sequences, a 1-1 label is added before the first row of the ID docking table as the thread identification number, indicating the first calculation of the first thread, and the corresponding sequence is extracted according to the number read ID based on the B+ tree index.

[0042] In this embodiment, step S3 specifically includes matching sequences 3, 9, 18, 66, 77, and 105 with reference sequence 1. When the sequencing reads of sequence 3 and 66 share a majority of bases with sequence 1, i.e., 80% or more of the bases are completely matched in the overlapped portion, and only a small number of SNPs are present, sequences 1, 3, and 66 are considered to have an overlapped region and can be merged into one sequence. If the overlapped region of sequences 9, 18, and 105 with sequence 1 contains a large number of different bases, i.e., the number of completely matched bases in the overlapped portion is less than 80%, these three sequences are discarded.

[0043] In this embodiment, step S4 specifically includes: when No. 3 can match No. 1, continue to search the ID docking table of sequence No. 3, and extract the sequence corresponding to sequence No. 3. This batch uses sequence No. 3 as the benchmark, and other sequences are matched with sequence No. 3 in sequence.

[0044] In this embodiment, the step S5 specifically includes: when No. 3 can match No. 1, continue to search the ID docking table of sequence No. 3, and find that the third thread has started calculating the thread identification number 3-1 before the ID docking table, then stop the calculation, output the thread identification number {1-1: 3-1, 66, 77} of this calculation and the base sequence No. 1 corresponding to the identification, and the small overlapping group samll contig sequence is identified as contig1-1.

[0045] In this embodiment, step S6 specifically includes: after the calculation of sequencing fragment reads No. 1 is completed, the first thread starts the second calculation, searches for sequencing fragment reads that are not identified in the ID docking table, such as 9-101, 102, and puts thread identification 1-2 in front of them. If No. 101 and No. 102 can match No. 9 and are not identified by other threads, then the base sequences docked with No. 101 and No. 102 are extracted at the same time, such as 101-1007, 1008, 102-1009, and 1010 for splicing. The calculation is terminated until a sequence with an identification is retrieved. If No. 1010 has been calculated by other threads 8-15, this calculation is terminated, and the identification number {1-2: 8-15, 1007, 1008, 1009} and the base sequence after No. 9 and 101, 102 are spliced ​​are output. The small contig sequence is identified as contig1-2, and so on.

[0046] In this embodiment, step S7 specifically includes: after all ID docking tables are calculated, all ID docking tables marked with thread identifiers are retrieved, and the output thread identifier table is optimized. For example, if threads 4-7 and 8-9 have started the calculation of read 66 and read77 respectively, {1-1: 3-1, 66, 77} in the identifier table is corrected to {1-1: 3-1, 4-7, 8-9}. The thread identifier also represents the number of the output small contig. When extracting sequencing fragment reads for one-to-one matching, the SNP site information can be output: the identifier of the small contig, the full length, the SNP variant base site, the base of the output contig at the variant site, the base replaced after the mutation, and other information.

[0047] Furthermore, steps S2 to S7 can be viewed as a reads multi-threaded matching module, which reads the numbered read IDs in the ID docking table in multiple threads, extracts the corresponding numbered read IDs from the hard disk or virtual memory using a B+ tree index, and matches them with the benchmark sequencing fragment reads in turn. Very similar sequencing fragment reads are matched and extended with the benchmark sequence to form small contigs.

[0048] Specifically, when extracting small contigs based on the path, the alignment is as follows: Figure 2 and Figure 3 Two cases are shown, where Figure 2 For head alignment, Figure 3 For tail alignment or staggered alignment, Figure 2K1, K2, and K3 are the variant sites in the original SNP information table. If the alignment is head alignment, the original SNP variant information is directly output. If the variant sites of the reference sequence and the remaining sequences overlap, the replaced base information is merged and output. Figure 3 Where K1, K2, K3, K4, and K5 represent the variant sites in the original SNP information table, L represents the full length of the reference sequence, and O represents the overlap between the upper and lower bases after alignment. If the alignment is tail-aligned (staggered), the variant site in the reference sequence remains unchanged, while the variant sites in the remaining sequences are adjusted to L-O+K3, L-O+K4, and L-O+K5, respectively. If the variant sites in the reference sequence and the remaining sequences overlap, the replaced base information is merged and output.

[0049] See Figure 4 Based on the same inventive concept, the present invention also proposes a multi-threaded system for gene assembly to implement the above-mentioned multi-threaded method for gene assembly. The system includes an ID docking table generation module and a reads multi-threaded matching module, wherein the ID docking table generation module is used to realize the correspondence between base sequences with front-to-back matching alignment relationships; the reads multi-threaded matching module is used to realize multi-threaded read matching and assemble into small contigs.

[0050] The present invention has at least the following technical effects:

[0051] The present invention realizes parallel computing of multiple servers by forming a server cluster for computing, using one of the servers as shared storage and other servers individually taking on tasks to execute, thus achieving linear performance increase of the assembly algorithm and infinite reduction of computing time, thus pushing bioinformatics analysis methods into the era of server cluster computing.

[0052] It should be noted that for the aforementioned embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are preferred embodiments, and the actions involved are not necessarily required by this application.

[0053] The above embodiments describe the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Without departing from the spirit and scope of the present invention, modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention should be within the scope of protection of the appended claims.

Claims

1. A multithreaded method for gene assembly, characterized in that: The steps include: S1: Assign custom numbers to the sequencing files and use the neural network fuzzy matching algorithm to obtain the corresponding relationship between the base sequences, i.e., the ID docking table; S2: Start multiple threads. Each thread reads the corresponding relationship in the ID docking table row by row, puts the thread identifier in front of the row, and uses the B+ tree index to extract the base sequence corresponding to the read ID from the hard disk or virtual memory respectively. S3: Match the multiple sequences after docking with the reference sequence in sequence, merge those that can be aligned, and discard those that cannot be aligned; S4: Continue reading the base sequences of the next batch of read IDs that can match the reference sequence, and extract the corresponding sequences to match them again with the reference sequence of this batch; S5: If a thread has already started computing the matches for the next batch of sequences, then the extraction of the next batch of sequences is terminated, and the thread identifier and the small contig output after the matching merge are output; S6: If no thread has started calculating the matching of the next batch of sequences, continue to extract the next batch of sequences that match the benchmark sequence and match them with the benchmark sequence of this batch until one of the sequences in the batch has been calculated by other threads. Then terminate this calculation and output the thread identifier and the small contig after the matching merge output; S7: After the ID docking table is read, the ID docking table is traversed, the base sequences of the scattered read IDs in the thread identification table are replaced with the corresponding base sequence IDs, and the single nucleotide variation SNP information table is output.

2. A multi-threaded method for gene assembly according to claim 1, characterized in that: The step S1 specifically includes: using a neural network fuzzy matching algorithm to output the front-to-back matching correspondence between the sequencing fragment reads in the form of an ID docking table, wherein the expression of the ID docking table is ab, c, d, where a, b, c, and d represent base sequence IDs, and the meaning of the expression is: there is a front-to-back alignment matching relationship between sequence a and sequences b, c, and d.

3. A multi-threaded method for gene assembly according to claim 1, characterized in that: The step S2 specifically includes: when the thread starts calculation, a label is added before the number read ID in the ID docking table included in the thread range, indicating that the base sequences 1-n are matched one by one; when the first thread performs calculation and matches base sequence 1 with other docking base sequences, a label is added before the first row of the ID docking table as the thread identification number, indicating the first calculation of the first thread, and the corresponding sequence is extracted according to the number read ID based on the B+ tree index.

4. A multi-threaded method for gene assembly according to claim 1, characterized in that: The step S3 specifically includes: matching the multiple sequences after the docking relationship with the reference sequence in sequence. When the overlapping part of the sequence after the docking relationship and the reference sequence has 80% or more completely matched bases, it is considered that the sequence has overlapped regions and can be merged into one sequence; when the number of completely matched bases in the overlapping part of the sequence after the docking relationship and the reference sequence is less than 80%, the sequence is discarded.

5. A multi-threaded method for gene assembly according to claim 1, characterized in that: The step S4 specifically includes: when the sequence to be tested can match the reference sequence, continue to search the ID docking table of the sequence to be tested, and extract the sequence number read ID corresponding to the sequence to be tested. This batch uses the sequence to be tested as the reference, and other sequences are matched with the sequence to be tested in turn.

6. A multi-threaded method for gene assembly according to claim 1, characterized in that: The step S5 specifically includes: when the sequence to be tested can match the reference sequence, continue to search the ID docking table of the sequence to be tested; if the thread identification number of the sequence to be tested has been calculated before the ID docking table, stop the calculation, and output the thread identification number calculated this time and the base sequence of the reference sequence corresponding to the identification; the sequence identification of the small overlapping group samll contig is recorded as contig 1-1.

7. A multi-threaded method for gene assembly according to claim 1, characterized in that: The step S6 specifically includes: after the calculation of sequencing fragment reads No. 1 is completed, the first thread starts the second calculation, searches for sequencing fragment reads that are not identified in the ID docking table, and marks the thread identifier in front of them. If there are other sequences that can match the unidentified sequence to be tested, and none of them are identified by other threads, the base sequences docked with the other sequences are extracted at the same time and spliced ​​together until each sequence is retrieved with an identifier, the calculation is terminated, and the spliced ​​base sequence is output. The small contig sequence is identified as contig 1-2, and so on.

8. A multi-threaded method for gene assembly according to claim 1, characterized in that: The step S7 specifically includes: after all ID docking tables are calculated, all ID docking tables marked with thread identifiers are retrieved, and the output thread identifier table is optimized, wherein the thread identifier also represents the number of the output small contig. When the sequencing fragment reads are extracted for one-to-one matching, the site information of the single nucleotide variation SNP can be output: the identifier of the small contig, the full length, the single nucleotide variation SNP variation base site, the base of the output contig at the variation site, and the base information replaced after the mutation.

9. A multi-threaded system for gene assembly, for implementing a multi-threaded method for gene assembly according to any one of claims 1 to 8, characterized in that: It includes an ID docking table generation module and a reads multi-threaded matching module, wherein the ID docking table generation module is used to realize the correspondence between base sequences with front-to-back matching alignment relationships; the reads multi-threaded matching module is used to realize multi-threaded sequencing fragment reads matching and assemble them into small contigs.

Images