Efficient spatial optimization dynamic programming algorithm for large-scale sequence matching

Abstract

The application provides a high-efficiency space optimization dynamic programming algorithm for large-scale sequence matching, comprising the following steps: input and pretreatment; sequence block; initialization of a global score matrix; initialization of a one-dimensional dynamic programming array, in-block dynamic programming calculation, calculation of a matching score, pruning judgment, updating of the one-dimensional dynamic programming array, updating of the global score matrix; according to the order of blocks, score merging, processing of inter-block matching dependency, calculation of a global optimal score; and result output. The application can reduce the space complexity from O(mn) to O(n), significantly improve the calculation efficiency, greatly reduce the demand for calculation resources on the premise of ensuring the matching accuracy, and is suitable for scenarios such as DNA sequence alignment and text similarity calculation that need to process super-long sequences.

Description

Technical Field

[0001] This invention relates to the fields of computer algorithms and bioinformatics, and in particular to an improved dynamic programming algorithm based on spatial optimization techniques for efficiently solving large-scale sequence matching problems, such as DNA sequence alignment and text similarity calculation. Background Technology

[0002] Sequence matching problems have wide applications in computer science, bioinformatics, and natural language processing. Common sequence matching algorithms include edit distance and longest common subsequence. These algorithms typically employ dynamic programming, with both time and space complexity of O(mn), where m and n are the lengths of the sequences to be matched, respectively.

[0003] When the sequence length reaches the millions, traditional dynamic programming algorithms require constructing massive two-dimensional matrices, resulting in excessive memory consumption and making them unusable on general-purpose computers. Simultaneously, computation time increases significantly with sequence length, failing to meet the demands of practical applications. Therefore, reducing the space and time complexity of dynamic programming algorithms has become a pressing issue. Summary of the Invention

[0004] This invention aims to provide an efficient space-optimized dynamic programming algorithm for large-scale sequence matching. By introducing space compression techniques, block optimization strategies, pruning, and heuristic search, the algorithm significantly reduces space requirements and computation time, thereby improving the efficiency of large-scale sequence matching.

[0005] To achieve the above objectives, the present invention proposes the following technical solution:

[0006] An efficient space-optimization dynamic programming algorithm for large-scale sequence matching includes the following steps:

[0007] Step 1, Input and Preprocessing: Input the sequences S and T to be matched; set the block size L, matching threshold θ, and other relevant parameters;

[0008] Step 2, Sequence Blocking: Calculate the number of blocks Divide sequences S and T into k blocks and l blocks respectively: S = [S 1 ,S 2 ,…,S k ],T=[T 1 ,T 2 ,…,T l ];

[0009] Step 3, Initialization: Initialize the global score matrix D globalThe size is (k+1)×(l+1), and the initial value is 0;

[0010] Step 4, Block Matching Calculation: Parallel computation is performed on each block pair (S p ,T q The following steps are executed in parallel:

[0011] Step 4.1, initialize the one-dimensional dynamic programming array D block The length is L+1, and the initial value is 0;

[0012] Step 4.2, Intra-block dynamic programming calculation:

[0013] (1) For i from 1 to L:

[0014] Save the previous state D prev =D block ;

[0015] (2) For j from 1 to L:

[0016] Calculate the matching score:

[0017]

[0018] Pruning decision: Calculate the maximum possible gain If score + max_gain < θ, then prune and skip subsequent calculations;

[0019] Update D block [j]:D block [j] = score;

[0020] Step 4.3, update the global score matrix, update D global [p][q]=D block [L];

[0021] Step 5, inter-block dependency handling: Merge D according to the block order. global The score is calculated, inter-block matching dependencies are handled, and the globally optimal score D is calculated. global [p][q];

[0022] For p from 1 to k, and q from 1 to l:

[0023]

[0024] Step 6, Output the result: The global optimal matching score is D. global [k][l]; By backtracking, the matching path is obtained.

[0025] This invention significantly reduces the space and time complexity of dynamic programming algorithms in large-scale sequence matching problems by applying space optimization techniques, and has the following specific benefits:

[0026] (1) Reduce space complexity: Reducing space complexity makes it possible to process very long sequences in a limited memory environment.

[0027] (2) Improve computational efficiency: By using block parallel computing and pruning strategies, unnecessary computations are reduced, thereby improving the overall efficiency of the algorithm.

[0028] (3) Strong applicability: The algorithm can be applied to different types of sequence matching problems and has a wide range of application prospects. Attached Figure Description

[0029] Figure 1 The diagram illustrates the traditional dynamic programming matrix in existing technologies, showing the two-dimensional matrix that the traditional dynamic programming algorithm needs to construct in the sequence matching problem, with a space complexity of O(mn).

[0030] Figure 2 This is a schematic diagram of the space compression technology described in an embodiment of the present invention, illustrating how a two-dimensional matrix can be compressed into a one-dimensional array by using a one-dimensional dynamic programming array and differential storage technology, thereby reducing space requirements.

[0031] Figure 3 This is a schematic diagram of sequence segmentation and parallel computing according to an embodiment of the present invention, which describes the process of dividing a sequence into several blocks and performing parallel computing on the matching between blocks.

[0032] Figure 4 This is a flowchart illustrating the algorithm described in this embodiment of the invention, showing the overall process of the algorithm, including steps such as input and preprocessing, sequence segmentation, initialization, block matching calculation, inter-block dependency handling, and result output. Input sequences S and T have lengths m and n respectively; block size L; matching score function match(); penalty value; threshold θ. Output the globally optimal matching score D. global [k][l] matches the path. Detailed Implementation

[0033] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] I. The design concept and principles of this invention include the following three main aspects:

[0035] 1. Application of space compression technology

[0036] 1.1 One-dimensional dynamic programming array

[0037] Traditional dynamic programming algorithms require constructing a two-dimensional matrix D(i,j) of size m×n. However, when computing D(i,j), only D(i-1,j-1), D(i-1,j), and D(i,j-1) are needed. Therefore, the two-dimensional matrix can be compressed into a one-dimensional array D(j) and updated using a rolling array or double buffer technique.

[0038] Updated formula:

[0039] Implementation details: Use two one-dimensional arrays D of length n+1. prev and D new Alternating updates.

[0040] The traditional two-dimensional dynamic programming matrix is ​​compressed into a one-dimensional array. By using a rolling array or double buffer technique, only the data of the current row and the previous row need to be stored during the calculation process, and the space complexity is reduced to O(n).

[0041] 1.2 Differential Storage

[0042] Principle: In the dynamic programming process, only the difference between adjacent elements ΔD(j)=D(j)-D(j-1) is stored, which can be stored using a smaller data type, further saving space.

[0043] Implementation details: Define a difference array ΔD(j). When updating D(j), only ΔD(j) needs to be calculated and stored. The absolute value of D(j) can be recovered using ΔD(j) as needed.

[0044] 2. Application of block optimization strategies

[0045] 2.1 Sequence Blocking

[0046] Method: Divide sequences S and T into subsequence blocks of length L respectively:

[0047] S = [S 1 ,S 2 ,…,S k ],T=[T 1 ,T 2 ,…,T l ]

[0048] in,

[0049] Advantages: By dividing a long sequence into multiple subsequence blocks of length L, a large-scale matching problem can be decomposed into multiple smaller matching problems, reducing the amount of computation and facilitating the use of parallel computing techniques.

[0050] 2.2 Inter-block dependency handling

[0051] Necessity: In block matching calculation, it is necessary to consider the dependencies between adjacent blocks to ensure the accuracy of the global optimal solution.

[0052] Implementation details:

[0053] In computation block S p and T q During matching, the boundary conditions are initialized, and the end value of the previous block is used as the start value of the current block.

[0054] By maintaining the matching scores and path information between blocks, the matching results of the entire sequence are ensured to be correct.

[0055] 2.3 Parallel Computing

[0056] Method: Utilize multithreading or GPU to process different block pairs (S p ,T q Parallel computation can be performed.

[0057] Implementation details:

[0058] Multithreading: Use thread pools or parallel computing libraries (such as OpenMP, MPI) to manage threads and allocate computing resources to each block pair.

[0059] GPU acceleration: Utilize CUDA or OpenCL to deploy block computation kernel functions on the GPU, fully leveraging the GPU's parallel computing capabilities.

[0060] Advantages: Parallel computing utilizes multi-threading or GPU acceleration to process different blocks in parallel, significantly improving computational efficiency and shortening the total computation time.

[0061] 3. Applications of pruning and heuristic search

[0062] 3.1 Threshold Pruning

[0063] Principle: In dynamic programming, if the potential optimal score of the current path cannot exceed the known threshold θ, the calculation of the path is terminated to avoid unnecessary calculations.

[0064] Implementation details:

[0065] A dynamic threshold θ can be set and adjusted dynamically according to actual needs.

[0066] When calculating D(j), calculate the current maximum possible score:

[0067]

[0068] If max_possible_score < θ, then prune and skip subsequent calculations.

[0069] 3.2 Heuristic Search

[0070] Principle: By utilizing the statistical characteristics of sequences or domain knowledge, a heuristic function is designed to prioritize the computation of regions that may have a high degree of matching, thereby reducing computational overhead and improving the practicality of the algorithm.

[0071] Implementation details:

[0072] Character frequency analysis: Statistically analyze the frequency of each character in the sequence and prioritize matching high-frequency character regions.

[0073] k-mer index: Extract fixed-length k-mer substrings, build an index, and quickly locate possible matching regions.

[0074] Heuristic scoring formula:

[0075] H(i,j)=α·match(S i ,T j )+β·similarity(S i:i+k ,T j:j+k )

[0076] Where α and β are weight parameters, and k is the substring length.

[0077] 3.3 Early Termination

[0078] Principle: Under certain conditions (such as reaching the expected matching score or time limit), the calculation is terminated early and the result is returned directly.

[0079] II. The final design of the algorithm of this invention is as follows: Figure 4 As shown, it includes the following 6 steps:

[0080] Step 1: Input and Preprocessing

[0081] Input the sequences S and T to be matched.

[0082] Set the block size L, matching threshold θ, and other parameters.

[0083] Step 2: Sequence Blocking

[0084] Calculate the number of blocks:

[0085]

[0086] Divide sequences S and T into k blocks and l blocks respectively:

[0087] S = [S 1 ,S 2 ,…,S k ],T=[T 1 ,T 2 ,…,T l ]

[0088] Step 3: Initialization

[0089] Initialize the global score matrix D global The size is (k+1)×(l+1), and the initial value is 0.

[0090] Step 4: Block matching computation (parallel execution)

[0091] Parallel computation: for each block pair (S) p ,T q The following steps are executed in parallel:

[0092] Step 4.1: Initialize the one-dimensional dynamic programming array D block The length is L+1, and the initial value is 0.

[0093] Step 4.2: Intra-block dynamic programming calculation

[0094] For i from 1 to L:

[0095] Save the previous state D prev =D block .

[0096] For j from 1 to L:

[0097] Calculate the matching score:

[0098]

[0099] Pruning decision: Calculate the maximum possible gain If score + max_gain < θ, then prune and skip subsequent calculations.

[0100] Update D block [j]:D block [j] = score

[0101] Step 4.3: Update the global score matrix, update D global [p][q]=D block [L].

[0102] Step 5: Inter-block dependency handling

[0103] Merge D according to the block order. global The score is calculated by handling the matching dependencies between blocks and then calculating the global optimal score.

[0104] For p from 1 to k, and q from 1 to l:

[0105]

[0106] Step 6: Output Results

[0107] The global optimal matching score is D global [k][l].

[0108] By backtracking, the matching path is obtained.

Claims

1. An efficient spatial optimization dynamic programming method for large-scale biological sequence matching, characterized in that, Includes the following steps: Step 1, Input and Preprocessing: Input the sequences S and T to be matched; Set the block size L, the matching threshold θ, and other relevant parameters; Step 2, Sequence Blocking: Calculate the number of blocks Divide sequences S and T into k blocks and l blocks respectively: S = [S 1 ,S 2 ,…,S k ],T=[T 1 ,T 2 ,…,T l ]; Step 3, Initialization: Initialize the global score matrix D global The size is (k+1)×(l+1), and the initial value is 0; Step 4, Block Matching Calculation: Using multithreading or GPU, for each distinct block pair (S p ,T q Parallel computation is performed, executing the following steps in parallel: Step 4.1, initialize the one-dimensional dynamic programming array D block The length is L+1, and the initial value is 0; Step 4.2, Intra-block dynamic programming calculation: (1) For i from 1 to L: Save the previous state D prev =D block ; (2) For j from 1 to L: Calculate the matching score: Pruning decision: Calculate the maximum possible gain if Then prune and skip subsequent calculations; Update D block [j]:D block [j] = score; Step 4.3, update the global score matrix, D global [p][q]=D block [L]; Step 5, inter-block dependency handling: Merge D according to the block order. global The score is calculated, inter-block matching dependencies are handled, and the globally optimal score D is calculated. global [p][q]; For p from 1 to k, and q from 1 to l: Step 6, Output the result: The global optimal matching score is D. global [k][l]; By backtracking, the matching path is obtained.

2. The efficient spatial optimization dynamic programming method for large-scale biological sequence matching according to claim 1, characterized in that, In step 4, the difference ΔD(j) between adjacent elements is defined. When updating D(j), it is only necessary to calculate and store the difference ΔD(j) between adjacent elements, ΔD(j) = D(j) - D(j-1). As needed, the absolute value of D(j) can be recovered using ΔD(j).

3. The efficient spatial optimization dynamic programming method for large-scale biological sequence matching according to claim 1, characterized in that, In computation block S p and T q During matching, boundary conditions are initialized, the end value of the previous block is used as the start value of the current block, and the matching results of the entire sequence are ensured to be correct by maintaining the matching scores and path information between blocks.

4. The efficient spatial optimization dynamic programming method for large-scale biological sequence matching according to claim 1, characterized in that, Design heuristic functions to prioritize calculating regions that are likely to have high matching degrees, specifically including: (1) Character frequency analysis: Statistically analyze the frequency of each character in the sequence and prioritize matching high-frequency character regions; (2) k-mer index: Extract fixed-length k-mer substrings, build an index, and quickly locate possible matching regions; (3) Heuristic scoring: H(i,j)=α·match(S i ,T j )+β·similarity(S i:i+k ,T j:j+k ), where α and β are weight parameters, and k is the substring length.

Images