Sequence alignment memory estimation and parameter tuning method based on BWA and BWA*

By establishing memory estimation and parameter selection models, optimizing the sequence alignment method of BWA and BWA*, the balance problem of memory overhead and comparison speed is solved, efficient sequence alignment is achieved, and the memory size of different computing terminals is adapted to significantly improve the Comparison efficiency.

CN118981373APending Publication Date: 2024-11-19CHONGQING INST OF GREEN & INTELLIGENT TECH CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410993821.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing sequence alignment methods based on BWA and BWA* have inefficient problems in memory estimation and parameter tuning, especially in the balance between large memory overhead and comparison speed, which makes ordinary computers unable to load huge memory. Overhead affects comparison efficiency.

Method used

By establishing memory estimation model A, sa_interval parameter selection model B and serial and parallel selection model C, we automatically tune the sa_interval and occ_interval parameters, combining the memory size of the calculation terminal and the number of query sequences, we optimize the sequence comparison method of BWA and BWA*. Select the optimal sa_interval and threading mode to meet memory constraints and improve the alignment speed.

Benefits of technology

It realizes efficient sequence comparison on computing terminals with different memory sizes, significantly improves the comparison speed and efficiency, reduces memory overhead, and adapts to the needs of various computing environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981373A_ABST
    Figure CN118981373A_ABST
Patent Text Reader

Abstract

The invention relates to a sequence alignment memory estimation and parameter tuning method based on BWA and BWA *, and belongs to the field of biological information and big data, and the method comprises the following steps: S1, building a memory estimation model A, a sainterval parameter selection model B and a series-parallel connection selection model C; s2, obtaining the available memory size, the reference sequence file size and the query sequence number of the computing terminal; s3, the value range of the sainterval parameters is obtained through the model A; s4, the model B is selected to calculate the optimal value of the sainterval; s5, judging whether a multi-thread mode is adopted for calculation or not by utilizing the model C; and S6, setting a sequence alignment method based on BWA or BWA * in combination with the optimal value of the sainterval and a thread mode, and completing sequence alignment by using the set method. According to the method and the device, the optimal sainterval parameter can be selected according to the query task and the terminal configuration, and whether multiple threads are suitable to be used or not is judged, so that the calculation speed of sequence alignment can be improved while the memory insufficiency is prevented.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a sequence alignment memory estimation and parameter tuning method based on BWA and BWA*, belongs to the field of bioinformatics and big data, and is particularly suitable for sequence alignment memory estimation and parameter tuning based on BWA and BWA*. Background Art

[0002] Sequence alignment is a basic task in bioinformatics, which is crucial for genome annotation, variation detection and evolutionary research. It mainly compares DNA, RNA or protein sequences with reference sequences in the database to identify similarities and differences. In simple terms, sequence alignment is to align the query sequence with the reference genome, and conduct further analysis on the target based on the similarity between the two.

[0003] There are many sequence alignment tools available. Most sequence alignment processes generate indexes before alignment and then use the indexes to speed up alignment. The most commonly used indexing methods are BWT and hash table methods. However, the hash table method is inefficient and limited in its suitability for certain scenarios. For BWA (Burrows-Wheeler Alignment) and BWA-MEM based on the BWT (Burrows-Wheeler Transform) indexing method, BWA has higher accuracy than BWA-MEM, while BWA-MEM is faster.

[0004] Since the BWA-based sequence alignment method needs to call the ALN and SAMSE algorithms separately to implement the two-step processing, based on this, some scholars proposed the BWA* method, which significantly improved the alignment speed by improving the algorithm flow of the SAMSE step of BWA, optimizing the sa_interval parameter, and adding multithreading to the SAMSE step, surpassing BWA-MEM in the alignment task.

[0005] However, in the BWT index, the complete SA (Suffix Array) table and OCC table are very large. Therefore, when generating the index, the SA table and OCC table are not completely saved, but one row is saved every certain number of rows. Among them, the parameters that control the size of the interval are sa_interval and occ_interval, that is, the SA table saves one row of data every sa_interval rows, and the OCC table saves one row of data every occ_interval rows. Therefore, the larger the sa_interval and occ_interval parameters are, the smaller the generated SA table and OCC table will be. However, it is worth noting that since only a part of the SA table and OCC table are generated in the indexing stage, this part of the data will be used to restore the data that was not generated in the indexing stage during the comparison. Therefore, the less data generated in the indexing stage, the more data needs to be restored during the comparison, that is, the larger the sa_interval and occ_interval parameters are, the longer it takes to restore the data during the comparison. Analysis shows that for BWA and BWA*, sa_interval only affects the SAMSE algorithm, and occ_interval affects both the ALN algorithm and the SAMSE algorithm.

[0006] In the application of existing technologies, the occ_interval parameter is mainly analyzed and adjusted, while occ_interval is not set. For example, Meysam Roodi et al. studied the impact of the parameter sa_interval on the performance of BWA-MEM in 2019, and concluded that the smaller the sa_interval, the faster the BWA-MEM speed. They used the human genome GRCh38 as the reference genome and NA12878 as the query sequence. When sa_interval was set to 1, the entire SA table was read into the memory. Although the comparison speed was improved to a certain extent, the memory overhead it brought was also very large, with a peak memory of up to 56.1GB. Ordinary computers simply cannot handle such a huge memory overhead. Summary of the invention

[0007] In view of this, the present invention provides a sequence alignment memory estimation and parameter tuning method based on BWA and BWA*. In order to adapt to computing terminals with different memory sizes for efficient sequence alignment, on the basis of the sequence alignment method based on BWA and BWA*, it is intended to use the sa_interval parameter and the occ_interval parameter to establish an optimization model and a memory estimation model for automatic tuning and selection.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A method for memory estimation and automatic parameter tuning of sequence alignment, comprising the following steps:

[0010] S1: Establish model A for memory estimation, model B for sa_interval parameter selection, and model C for series-parallel selection;

[0011] S2: Obtain the available memory size of the computing terminal, the size of the reference sequence file, and the number of query sequences;

[0012] S3: Taking the available memory size of the computing terminal as a constraint, combined with the reference sequence file size and the number of query sequences, the value range of the sa_interval parameter is obtained through model A;

[0013] S4: Set the value range of the sa_interval parameter calculated in step S3 as the sampling interval, and select model B to calculate the optimal sa_interval value with the fastest comparison speed;

[0014] S5: using model C to determine whether to use multi-threaded mode for calculation;

[0015] S6: Combine the optimal value of sa_interval and the thread mode to set the sequence alignment method based on BWA or BWA*, and use the set method to complete the sequence alignment.

[0016] Furthermore, both the BWA-based and BWA*-based sequence alignment methods need to call the ALN and SAMSE algorithms respectively to implement the two-step processing. During the entire process, the memory estimation needs to be satisfied that the memory size does not exceed the available memory size. Therefore, the memory estimation model A is specifically:

[0017]

[0018] Among them, MEM represents the available memory size; f represents the reference sequence file size, in GB; ref_c, bwt_c, occ_c and sa_c represent the compression ratios of the reference sequence, BWT transformation, OCC table and SA table respectively, which are fixed parameters corresponding to the sequence alignment tool; the occ_interval parameter and sa_interval parameter are both powers of 2; OTHER represents the memory overhead required for the query sequence in the BWA or BWA* sequence alignment method, in GB.

[0019] Preferably, in the process of ALN algorithm processing based on the sequence alignment method of BWA or BWA*, the memory estimation model A can be simplified as follows:

[0020]

[0021] Preferably, in the process of SAMSE algorithm processing of the BWA sequence alignment method, the memory estimation model A can be simplified as follows:

[0022]

[0023] Preferably, during the processing of the SAMSE algorithm by the BWA* sequence alignment method, the memory estimation model A can be simplified as follows:

[0024]

[0025] Furthermore, since the ALN algorithm processing process is independent of sa_interval, in order to improve the comparison efficiency, it is necessary to optimize the sa_interval corresponding to the SAMSE algorithm processing process. The sa_interval parameter selected for model B is:

[0026]

[0027] in, Indicates the time to read the SA table; b×sa_interval×N b Indicates the time when the corresponding position of the SA table is converted to the reference sequence position; represents the number of loop iterations of SA table conversion; B is the number of reads processed in each loop iteration, which is a set constant of 262144; reads is the number of query sequences; a, b, c are unknown coefficients; T is the processing time of SAMSE algorithm.

[0028] Further, the step S4 is specifically as follows:

[0029] S401: The value range of the sa_interval parameter calculated in step S3 is used as a sampling interval, and the SAMSE algorithm is executed by using a computing terminal to count the SAMSE algorithm processing time T corresponding to different sa_interval parameters;

[0030] S402: Calculate the undetermined coefficients a, b, and c in model B using regression analysis based on the data in step S401;

[0031] S403: Model B is a check mark function. According to the minimum value of the check mark function, When , the corresponding contrast efficiency is the highest, which is the optimal value.

[0032] Preferably, the undetermined coefficient b and the number of iterations N are b Can be used and Scaling is performed to improve the precision of regression analysis.

[0033] Further, the series-parallel selection model C is:

[0034]

[0035] Among them, T s and T p They represent the running time of the SAMSE algorithm in single thread and multi-thread respectively; T0 represents the running time of serial calculation outside the loop; is the loop iteration threshold, is the coefficient to be determined; C s and C p Respectively represent the running time of the parallel and serial parts in the loop without multithreading; T ∞ and T μ They represent the overhead caused by parallelism inside and outside the loop respectively; p is the number of threads, which is a constant 2.

[0036] Further, the step S5 is specifically as follows:

[0037] S501: The computing terminal executes the SAMSE algorithm multiple times without using multithreading to measure the running time C of the parallel and serial parts in the loop s and C p ;

[0038] S502: The computing terminal executes the SAMSE algorithm multiple times to measure the running time T of the single thread and the multi-thread s and T p ;

[0039] S503: Combine the data of step S501 and step S502 to As a variable, using univariate linear regression, the constant coefficient T is fitted ∞ and T μ ;

[0040] S504: Calculate the conditions that meet the requirements The minimum value of

[0041] S505: Utilization Calculate the minimum value of And compare whether it is greater than the number of query sequences. If it is greater, single thread is used, otherwise multi-thread is used.

[0042] The beneficial effects of the present invention are as follows: the present invention provides a sequence alignment memory estimation and parameter tuning method based on BWA and BWA*, according to the memory size of the computing terminal and the number of query sequences, on the basis of the sequence alignment method based on BWA and BWA*, the sa_interval parameter and the occ_interval parameter are used to establish inequality constraint models A, B, and C to respectively realize memory estimation, sa_interval parameter estimation, and automatic selection of thread tasks, thereby realizing efficient sequence alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the purpose and technical solution of the present invention, the present invention provides the following drawings for explanation:

[0044] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0045] In order to make the technical solution, the purpose of the embodiments and the advantages of the system architecture of the present invention clearer, the preferred embodiments of the present invention will be described in detail below in conjunction with Appendix 1.

[0046] Example 1: The existing computer experimental platform is configured as follows: the operating system is 64-bit Ubuntu 20.04, the compiler is GCC 9.4.0, the processor is Intel (R) Core (TM) i7-12700, the CPU main frequency is 4.9GHz, the number of cores is 12, and the available memory is 31.1GB. The task of comparing the query sequence consisting of the first ten million sequences of the human genome GRCh38.p14 and SRR10067255 sequencing sequence with a size of 3.3GB. The present invention provides a "sequence alignment memory estimation and parameter tuning method based on BWA*", and the specific steps are:

[0047] Step 1: Establish model A for memory estimation, model B for sa_interval parameter selection, and model C for series-parallel selection.

[0048] Among them, for the processing of ALN and SAMES algorithms, a memory estimation model A is established respectively;

[0049] In the process of ALN algorithm processing based on BWA* sequence alignment method, the memory estimation model A can be simplified as follows:

[0050]

[0051] In the process of SAMSE algorithm processing, the memory estimation model A of the BWA* sequence alignment method can be simplified as follows:

[0052]

[0053] Wherein, MEM represents the available memory size; f represents the reference sequence file size, in GB; ref_c, bwt_c, occ_c and sa_c represent the compression ratios of the reference sequence, BWT transformation, OCC table and SA table respectively, which are fixed parameters corresponding to the sequence alignment tool; the occ_interval parameter and sa_interval parameter are both powers of 2.

[0054] The sa_interval parameter is chosen for model B as:

[0055]

[0056] in, Indicates the time to read the SA table; b×sa_interval×N b Indicates the time when the corresponding position of the SA table is converted to the reference sequence position; represents the number of loop iterations of SA table conversion; B is the number of reads processed in each loop iteration, which is a set constant of 262144; reads is the number of query sequences; a, b, c are unknown coefficients; T is the processing time of SAMSE algorithm.

[0057] The model C selected by series-parallel connection is:

[0058]

[0059] Among them, T s and T p They represent the running time of the SAMSE algorithm in single thread and multi-thread respectively; T0 represents the running time of serial calculation outside the loop; is the loop iteration threshold, is the coefficient to be determined; C s and C p Respectively represent the running time of the parallel and serial parts in the loop without multithreading; T ∞ and T μ They represent the overhead caused by parallelism inside and outside the loop respectively: p is the number of threads, which is a constant 2.

[0060] Step 2: Get the available memory size of the computing terminal as MEM = 31.1 GB, the reference sequence file size as f = 3.3 GB, and the number of query sequences as reads = 10*10 6 .

[0061] Step 3: Taking the available memory of 31.1 GB as a constraint, use model A to calculate the ALN algorithm process, select occ_interval = 128, and calculate the SAMES algorithm process to obtain the selection range of sa_interval: sa_interval ≥ 2.

[0062] Step 4: Set the value range of the sa_interval parameter calculated in step 3 as the sampling interval, and select model B to calculate the optimal sa_interval value with the fastest comparison speed.

[0063] Specifically:

[0064] S401: sa_interval≥2 is used as a sampling interval, and a computing terminal is used to execute a SAMSE algorithm, and the SAMSE algorithm processing time T corresponding to different sa_interval parameters is counted;

[0065] S402: Calculate the undetermined coefficients a, b, and c in model B using regression analysis based on the data in step S401;

[0066] S403: Model B is a check mark function. According to the minimum value of the check mark function, When the corresponding contrast efficiency is the highest, the optimal value is 8;

[0067] Step 5: Use model C to determine whether to use multi-threaded mode calculation.

[0068] Specifically:

[0069] S501: The computing terminal executes the SAMSE algorithm multiple times without using multithreading to measure the running time C of the parallel and serial parts in the loop s and C p :

[0070] S502: The computing terminal executes the SAMSE algorithm multiple times to measure the running time T of the single thread and the multi-thread s and T p ;

[0071] S503: Combine the data of step S501 and step S502 to As a variable, using univariate linear regression, the constant coefficient T is fitted ∞ and T μ ;

[0072] S504: Calculate the conditions that meet the requirements The minimum value of is 2.445;

[0073] S505: The number of query sequence reads is 10*106 Much larger than the threshold Therefore multi-threaded computing is used.

[0074] Step 6: Combine the optimal sa_interval value of 8 and the multi-threaded mode to set the BWA*-based sequence alignment method and use the set method to complete the sequence alignment.

[0075] Table 1 Performance acceleration comparison experimental results of Example 1

[0076] SAMSE time with default parameters and single thread The method of the present invention has sa_interval=8, SAMSE time under 4 threads 75.020s 17.917s

[0077] Example 2: Based on the computing terminal of Example 1, the comparison task of the query sequence consisting of the first two million sequences of the human genome GRCh38.p14 and SRR6889465 sequence sequence with a size of 3.3GB is performed. The present invention provides a "sequence alignment memory estimation and parameter tuning method based on BWA", and the steps that are roughly the same as those in Example 1 are not repeated here, and the specific steps can be described as follows:

[0078] Step 1: Establish model A for memory estimation, model B for sa_interval parameter selection, and model C for series-parallel selection.

[0079] Among them, in the process of SAMSE algorithm processing of BWA sequence alignment method, the memory estimation model A can be simplified as follows:

[0080]

[0081] Step 2: Get the available memory size of the computing terminal as MEM = 31.1 GB, the reference sequence file size as f = 3.3 GB, and the number of query sequences as reads = 2*10 6 .

[0082] Step 3: Taking the available memory of 31.1 GB as a constraint, use model A to calculate the ALN algorithm process, select occ_interval = 128, and calculate the SAMES algorithm process to obtain the selection range of sa_interval: sa_interval ≥ 2.

[0083] Step 4: Set sa_interval ≥ 2 as the sampling interval, select model B and calculate the optimal sa_interval value of 16, which has the fastest matching speed.

[0084] Among them, zoom

[0085] Step 5: Use model C to determine whether to use multi-threaded mode calculation: reads 2*10 6It is much larger than the threshold of 513278, so multi-threaded calculation is used;

[0086] Step 6: Combine the optimal sa_interval value 16 and the multi-threaded mode to set the BWA-based sequence alignment method and use the set method to complete the sequence alignment.

[0087] Table 2 Performance acceleration comparison experimental results of Example 2

[0088] SAMSE time with default parameters and single thread The SAMSE time of the method of the present invention under sa_interval=16 and 4 threads 31.346s 9.756s

[0089] It can be seen that the method of the present invention can improve the efficiency of sequence alignment, which is significantly improved compared with the existing BWA and BWA* sequence alignment methods.

[0090] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A method for memory estimation and parameter tuning of sequence alignment based on BWA and BWA*, characterized in that: The following steps are involved: S1: Establish model A for memory estimation, model B for sa_interval parameter selection, and model C for series-parallel selection; S2: Obtain the available memory size of the computing terminal, the size of the reference sequence file, and the number of query sequences; S3: Taking the available memory size of the computing terminal as a constraint, combined with the reference sequence file size and the number of query sequences, the value range of the sainterval parameter is obtained through model A; S4: Set the value range of the sa_interval parameter calculated in step S3 as the sampling interval, and select model B to calculate the optimal sa_interval value with the fastest comparison speed; S5: using model C to determine whether to use multi-threaded mode for calculation; S6: Combine the optimal value of sa_interval and the thread mode, set the sequence alignment method based on BWA or BWA*, and complete the sequence alignment using the set method; The sequence alignment methods based on BWA and BWA* need to call the ALN and SAMSE algorithms respectively to implement the two-step processing. During the whole process, the memory estimation needs to be satisfied that it does not exceed the available memory size. Therefore, the memory estimation model A is specifically: Wherein, MEM represents the available memory size; f represents the reference sequence file size, in GB; ref_c, bwt_c, occ_c and sa_c represent the compression ratios of the reference sequence, BWT transformation, OCC table and SA table respectively, which are fixed parameters corresponding to the sequence alignment tool; the occ_interval parameter and sa_interval parameter are both powers of 2; OTHER represents the memory overhead required for the query sequence in the BWA or BWA* sequence alignment method, in GB; The sa_interval parameter is chosen for model B as: in, Indicates the time to read the SA table; b×sainterval×N b Indicates the time when the corresponding position of the SA table is converted to the reference sequence position; represents the number of loop iterations of SA table conversion; B is the number of reads processed in each loop iteration, which is a set constant of 262144; reads is the number of query sequences; a, b, c are undetermined coefficients; T is the processing time of SAMSE algorithm; The model C selected by series-parallel connection is: Among them, T s and T p They represent the running time of the SAMSE algorithm in single thread and multi-thread respectively; T0 represents the running time of serial calculation outside the loop; is the loop iteration threshold, is the coefficient to be determined; C s and C p Respectively represent the running time of the parallel and serial parts in the loop without multithreading; T ∞ and T μ They represent the overhead caused by parallelism inside and outside the loop respectively; p is the number of threads, which is a constant 2.

2. The method for memory estimation and parameter tuning of sequence alignment based on BWA and BWA* according to claim 1, characterized in that: In the process of SAMSE algorithm processing of BWA sequence alignment method, the memory estimation model A can be simplified as follows:

3. The method for memory estimation and parameter tuning of sequence alignment based on BWA and BWA* according to claim 1, characterized in that: In the process of SAMSE algorithm processing of BWA* sequence alignment method, the memory estimation model A can be simplified as follows:

4. The method for memory estimation and parameter tuning of sequence alignment based on BWA and BWA* according to claim 1, characterized in that: The step S4 is specifically as follows: S401: The value range of the sa_interval parameter calculated in step S3 is used as a sampling interval, and the SAMSE algorithm is executed by using a computing terminal to count the SAMSE algorithm processing time T corresponding to different sa_interval parameters; S402: Calculate the undetermined coefficients a, b, and c in model B using regression analysis based on the data in step S401; S403: Model B is a check mark function. According to the minimum value of the check mark function, When , the corresponding contrast efficiency is the highest, which is the optimal value.

5. The method for memory estimation and parameter tuning of sequence alignment based on BWA and BWA* according to claim 4, characterized in that: The undetermined coefficient b and the number of iterations N b Can be used and Scaling is performed to improve the precision of regression analysis.

6. The method for memory estimation and parameter tuning of sequence alignment based on BWA and BWA* according to claim 1, characterized in that: The step S5 is specifically as follows: S501: The computing terminal executes the SAMSE algorithm multiple times without using multithreading to measure the running time C of the parallel and serial parts in the loop s and C p ; S502: The computing terminal executes the SAMSE algorithm multiple times to measure the running time T of the single thread and the multi-thread s and T p ; S503: Combine the data of step S501 and step S502 to As a variable, using univariate linear regression, the constant coefficient T is fitted ∝ and T μ ; S504: Calculate the conditions that meet the requirements The minimum value of S505: Utilization Calculate the minimum value of And compare whether it is greater than the number of query sequences. If it is greater, single thread is used, otherwise multi-thread is used.