Efficient space optimization dynamic programming algorithm for large-scale sequence matching

Abstract

The invention provides an efficient space optimization dynamic programming algorithm for large-scale sequence matching. The efficient space optimization dynamic programming algorithm comprises the following steps: inputting and preprocessing; carrying out sequence blocking; initializing a global score matrix; initializing a one-dimensional dynamic programming array, performing dynamic programming calculation in a block, calculating a matching score, performing pruning judgment, updating the one-dimensional dynamic programming array, and updating a global score matrix; according to the sequence of the blocks, combining scores, processing matching dependence among the blocks, and calculating a global optimal score; and outputting a result. According to the method, the space complexity can be reduced from O (mn) to O (n), the calculation efficiency is remarkably improved, the requirement for calculation resources is greatly reduced on the premise that the matching precision is guaranteed, and the method is suitable for DNA sequence alignment, text similarity calculation and other scenes needing to process super-long sequences.

Description

Technical Field

[0001] The present invention relates to the fields of computer algorithms and bioinformatics, and in particular to an improved dynamic programming algorithm based on space optimization technology for efficiently solving large-scale sequence matching problems, such as DNA sequence alignment, text similarity calculation, etc. Background Art

[0002] Sequence matching has wide applications in computer science, bioinformatics, and natural language processing. Common sequence matching algorithms include Edit Distance and Longest Common Subsequence. These algorithms usually use dynamic programming methods, and their time and space complexities are both O(mn), where m and n are the lengths of the sequences to be matched.

[0003] When the sequence length reaches millions, the traditional dynamic programming algorithm needs to construct a huge two-dimensional matrix, which results in excessive memory consumption and cannot be run on general computers. At the same time, the calculation time also increases significantly with the increase of sequence length, which is difficult to meet the needs of practical applications. Therefore, how to reduce the space and time complexity of the dynamic programming algorithm has become an urgent problem to be solved. Summary of the invention

[0004] The present invention aims to provide an efficient space-optimized dynamic programming algorithm for large-scale sequence matching. By introducing space compression technology, block optimization strategy, pruning and heuristic search methods, the algorithm's space requirements and computing time are significantly reduced, thereby improving the efficiency of large-scale sequence matching.

[0005] To achieve the above object, the present invention proposes the following technical solutions:

[0006] An efficient space-optimized dynamic programming algorithm for large-scale sequence matching includes the following steps:

[0007] Step 1, input and preprocessing: input the sequences S and T to be matched; set the block size L, matching threshold θ and other related parameters;

[0008] Step 2, Sequence Blocking: Calculate the Number of Blocks Divide the sequences S and T into k blocks and l blocks respectively: S = [S 1 ,S 2 ,…,S k ],T=[T 1 ,T 2 ,…,T l ];

[0009] Step 3, Initialization: Initialize the global score matrix D global, the size is (k+1)×(l+1), and the initial value is 0;

[0010] Step 4, block matching calculation: parallel calculation for each block pair (S p ,T q ), and execute the following steps in parallel:

[0011] Step 4.1, initialize the one-dimensional dynamic programming array D block , the length is L+1, and the initial value is 0;

[0012] Step 4.2, dynamic programming calculation within the block:

[0013] (1) For i from 1 to L:

[0014] Save the previous state D prev =D block ;

[0015] (2) For j from 1 to L:

[0016] Calculate the matching score:

[0017]

[0018] Pruning judgment: calculating the maximum possible gain If score+max_gain<θ, prune and skip subsequent calculations;

[0019] Update D block [j]: D block [j] = score;

[0020] Step 4.3, update the global score matrix and update D global [p][q]=D block [L];

[0021] Step 5: Inter-block dependency processing: merge D in the order of blocks global The score in , handles the matching dependency between blocks, and calculates the global optimal score D global [p][q];

[0022] For p from 1 to k, q from 1 to l:

[0023]

[0024] Step 6, result output: the global optimal matching score is D global [k][l]; Get the matching path by backtracking.

[0025] The present invention significantly reduces the space and time complexity of the dynamic programming algorithm in large-scale sequence matching problems by applying space optimization technology, and has the following beneficial effects:

[0026] (1) Reduce space complexity: Reducing space complexity makes it possible to process very long sequences in a limited memory environment.

[0027] (2) Improve computational efficiency: Through block parallel computing and pruning strategies, unnecessary calculations can be reduced and the overall efficiency of the algorithm can be improved.

[0028] (3) Strong applicability: The algorithm can be applied to different types of sequence matching problems and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the traditional dynamic programming matrix in the prior art, showing the two-dimensional matrix that the traditional dynamic programming algorithm needs to construct in the sequence matching problem, with a space complexity of O(mn).

[0030] Figure 2 This is a schematic diagram of the space compression technology described in an embodiment of the present invention, which shows how to use a one-dimensional dynamic programming array and differential storage technology to compress a two-dimensional matrix into a one-dimensional array to reduce space requirements.

[0031] Figure 3 This is a schematic diagram of sequence segmentation and parallel computing according to an embodiment of the present invention, which describes a process of segmenting a sequence into several blocks and performing parallel computing on the matching between the blocks.

[0032] Figure 4 This is a flowchart of the algorithm described in the embodiment of the present invention, which shows the overall process of the algorithm of the present invention, including the steps of input and preprocessing, sequence segmentation, initialization, block matching calculation, inter-block dependency processing and result output. Input sequences S and T, lengths m and n respectively; block size L; matching score function match(); penalty value penalty; threshold θ. Output global optimal matching score D global [k][l], match the path. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] 1. The design ideas and principles of the present invention include the following three aspects:

[0035] 1. Application of space compression technology

[0036] 1.1 One-dimensional dynamic programming array

[0037] The traditional dynamic programming algorithm needs to construct a two-dimensional matrix D(i,j) of size m×n. However, when calculating D(i,j), only D(i-1,j-1), D(i-1,j), and D(i,j-1) are needed. Therefore, the two-dimensional matrix can be compressed into a one-dimensional array D(j) and updated using rolling array or double buffer technology.

[0038] Update formula:

[0039] Implementation details: Use two one-dimensional arrays D of length n+1 prev and D new , updated alternately.

[0040] The traditional two-dimensional dynamic programming matrix is ​​compressed into a one-dimensional array. Using rolling array or double buffer technology, during the calculation process, only the data of the current row and the previous row need to be stored, and the space complexity is reduced to O(n).

[0041] 1.2 Differential Storage

[0042] Principle: During the dynamic programming process, only the difference between adjacent elements ΔD(j)=D(j)-D(j-1) is stored, and a smaller data type can be used for storage to further save space.

[0043] Implementation details: Define a difference array ΔD(j). When updating D(j), only ΔD(j) needs to be calculated and stored. ΔD(j) is used to restore the absolute value of D(j) as needed.

[0044] 2. Application of block optimization strategy

[0045] 2.1 Sequence Blocking

[0046] Method: Divide the sequences S and T into subsequence blocks of length L respectively:

[0047] S=[S 1 ,S 2 ,…,S k ],T=[T 1 ,T 2 ,…,T l ]

[0048] in,

[0049] Advantages: By dividing a long sequence into multiple subsequence blocks of length L, a large-scale matching problem can be decomposed into multiple small-scale matching problems, reducing the amount of calculation and facilitating the use of parallel computing technology.

[0050] 2.2 Inter-block dependency processing

[0051] Necessity: In block matching calculation, the dependency between adjacent blocks needs to be considered to ensure the accuracy of the global optimal solution.

[0052] Implementation details:

[0053] In the calculation block S p and T q When matching, initialize the boundary conditions and use the end value of the previous block as the start value of the current block.

[0054] By maintaining the matching scores and path information between blocks, the matching results of the entire sequence are guaranteed to be correct.

[0055] 2.3 Parallel Computing

[0056] Method: Use multithreading or GPU to process different block pairs (S p ,T q ) for parallel computing.

[0057] Implementation details:

[0058] Multithreading: Use a thread pool or parallel computing library (such as OpenMP, MPI) to manage threads and allocate computing resources to each block pair.

[0059] GPU acceleration: Use CUDA or OpenCL to deploy the kernel function of block calculation on the GPU to fully utilize the parallel computing capability of the GPU.

[0060] Advantages: Parallel computing uses multi-threading or GPU acceleration to process different blocks in parallel, significantly improving computing efficiency and shortening the total computing time.

[0061] 3. Application of pruning and heuristic search

[0062] 3.1 Threshold Pruning

[0063] Principle: During the dynamic programming process, if the potential optimal score of the current path cannot exceed the known threshold θ, the calculation of the path is terminated to avoid unnecessary calculations.

[0064] Implementation details:

[0065] The dynamic threshold θ is set and can be adjusted dynamically according to actual needs.

[0066] When calculating D(j), calculate the current possible maximum score:

[0067]

[0068] If max_possible_score < θ, prune and skip subsequent calculations.

[0069] 3.2 Heuristic Search

[0070] Principle: Utilize the statistical characteristics of the sequence or domain knowledge to design a heuristic function, give priority to calculating the areas where high matching degrees may exist, reduce computational overhead, and improve the practicality of the algorithm.

[0071] Implementation details:

[0072] Character frequency analysis: Count the frequency of each character in the sequence and give priority to matching high-frequency character areas.

[0073] k-mer index: extract k-mer substrings of fixed length, create indexes, and quickly locate possible matching areas.

[0074] Heuristic scoring formula:

[0075] H(i,j)=α·match(S i ,T j )+β·similarity(S i:i+k ,T j:j+k )

[0076] Among them, α and β are weight parameters, and k is the substring length.

[0077] 3.3 Early Termination

[0078] Principle: When certain conditions are met (such as reaching the expected matching score or time limit), the calculation is terminated early and the result is returned directly.

[0079] Second, the final design of the algorithm of the present invention is as follows Figure 4 As shown, it includes the following 6 steps:

[0080] Step 1: Input and preprocessing

[0081] Enter the sequences S and T to be matched.

[0082] Set the block size L, matching threshold θ, and other parameters.

[0083] Step 2: Sequence Chunking

[0084] Calculate the number of blocks:

[0085]

[0086] Divide the sequences S and T into k blocks and l blocks respectively:

[0087] S=[S 1 ,S 2 ,…,S k ],T=[T 1 ,T 2 ,…,T l ]

[0088] Step 3: Initialization

[0089] Initialize the global score matrix D global , the size is (k+1)×(l+1), and the initial value is 0.

[0090] Step 4: Block matching calculation (parallel execution)

[0091] Parallel computing: For each block pair (S p ,T q ), and execute the following steps in parallel:

[0092] Step 4.1: Initialize the one-dimensional dynamic programming array D block , length is L+1, and the initial value is 0.

[0093] Step 4.2: Dynamic programming calculation within the block

[0094] For i from 1 to L:

[0095] Save the previous state D prev =D block .

[0096] For j from 1 to L:

[0097] Calculate the matching score:

[0098]

[0099] Pruning judgment: calculating the maximum possible gain If score+max_gain<θ, prune and skip subsequent calculations.

[0100] Update D block [j]: D block [j] = score

[0101] Step 4.3: Update the global score matrix and update D global [p][q]=D block [L].

[0102] Step 5: Inter-block dependency processing

[0103] Merge D in the order of blocks global The scores in , process the matching dependencies between blocks, and calculate the global optimal score.

[0104] For p from 1 to k, q from 1 to l:

[0105]

[0106] Step 6: Output the results

[0107] The global optimal matching score is D global [k][l].

[0108] By backtracking, the matching path is obtained.

Claims

1. An efficient space-optimized dynamic programming algorithm for large-scale sequence matching, characterized in that: The following steps are involved: Step 1, input and preprocessing: input the sequences S and T to be matched; Set the block size L, matching threshold θ and other related parameters; Step 2, Sequence Blocking: Calculate the Number of Blocks Divide the sequences S and T into k blocks and l blocks respectively: S = [S 1 , S 2 , ..., S k ],T=[T 1 , T 2 , ..., T l ]; Step 3, Initialization: Initialize the global score matrix D global , the size is (k+1)×(l+1), and the initial value is 0; Step 4, block matching calculation: parallel calculation for each block pair (S p , T q ), and execute the following steps in parallel: Step 4.1, initialize the one-dimensional dynamic programming array D block , the length is L+1, and the initial value is 0; Step 4.2, dynamic programming calculation within the block: (1) For i from 1 to L: Save the previous state D prev =D block ; (2) For j from 1 to L: Calculate the matching score: Pruning judgment: calculating the maximum possible gain If score+max_gain<θ, prune and skip subsequent calculations; Update D block [j]: D block [j] = score; Step 4.3, update the global score matrix, D global [p][q]=D block [L]; Step 5: Inter-block dependency processing: merge D in the order of blocks global The score in , handles the matching dependency between blocks, and calculates the global optimal score D global [p][q]; For p from 1 to k, q from 1 to l: Step 6, result output: the global optimal matching score is D global [k][l]; Get the matching path by backtracking.

2. The efficient space-optimized dynamic programming algorithm for large-scale sequence matching according to claim 1, characterized in that: In step 4, a difference array ΔD(j) is defined. When updating D(j), it is only necessary to calculate and store the difference ΔD(j) between adjacent elements, ΔD(j)=D(j)-D(j-1). ΔD(j) is used to restore the absolute value of D(j) as needed.

3. The efficient space-optimized dynamic programming algorithm for large-scale sequence matching according to claim 1, characterized in that: In the calculation block S p and T q When matching, the boundary conditions are initialized, and the end value of the previous block is used as the start value of the current block. By maintaining the matching scores and path information between blocks, the matching results of the entire sequence are guaranteed to be correct.

4. The efficient space-optimized dynamic programming algorithm for large-scale sequence matching according to claim 1, characterized in that: Using multithreading or GPU, different block pairs (S p , T q ) for parallel computing.

5. The efficient space-optimized dynamic programming algorithm for large-scale sequence matching according to claim 1, characterized in that: Design a heuristic function to prioritize the calculation of areas where there may be high matching degrees, including: (1) Character frequency analysis: Count the frequency of each character in the sequence and prioritize matching high-frequency character areas; (2) k-mer index: extract k-mer substrings of fixed length, create indexes, and quickly locate possible matching areas; (3) Heuristic scoring: H(i, j) = α·match(S i , T j )+β·similarity(S i：i+k , T j：j+k ), where α and β are weight parameters and k is the substring length.

Images