Insertion, deletion, substitution error correction encoding method applied to DNA storage

Abstract

The application discloses an insertion, deletion and substitution error correction coding method applied to DNA storage, and comprises the following steps: embedding the edit distance between two sequences into the normal distance between two corresponding embedding vectors to realize deep embedding of the edit distance; using a greedy search based on deep embedding, constructing code words with an interval of 3 edit distances from each other in the embedding space, so that one edit error can be corrected; and using an efficient tree search algorithm in the embedding space to speed up the correction process of the damaged code word. In addition, in order to improve the decoding efficiency, a long sequence decoding method based on a depth-first search method is provided, the method has an early stopping criterion, and the decoding speed can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of data storage using deoxynucleotides (DNA), and more particularly to an insertion, deletion, and substitution error correction coding method applicable to DNA storage. Background Technology

[0002] Readings obtained from the original strand are typically affected by insertion, deletion, and substitution errors (IDS). In the early stages of DNA storage research, error-correcting codes such as Reed-Solomon (RS) codes, fountain codes, and low-density parity-check codes (LDPC) were established. More recently, to accommodate IDS channels, some methods have introduced synchronization markers and watermarks to correct synchronization errors or generate implicitly disjoint segments. Within related fields, there are also synchronization error-correcting codes based on complex mathematical and combinatorial techniques.

[0003] Traditional error-correcting coding methods construct codewords through random selection, but this algorithm ignores the inherent edit distance property of the sequence. Furthermore, it's impossible to strike a balance between coding efficiency and error-correcting capability when choosing codeword length. For small codeword lengths N, the constant terms in the redundancy dominate, leading to low coding efficiency; while for larger codeword lengths N, the error-correcting capability is limited, only correcting single errors and lacking applicability. Second, the computational complexity of bounded distance decoders (BDD) for calculating edit distance is considerable. Moreover, most existing search algorithms are primarily designed for traditional distances and are not suitable for edit distances. Third, encoding long sequences requires concatenating segment codewords to achieve the desired original chain length. However, the readout sequence may carry synchronization errors from the original chain. During decoding, existing methods simply divide the readout into segments of length n, which can lead to synchronization mismatches, affecting decoding accuracy and even causing decoding failure. Summary of the Invention

[0004] The purpose of this invention is to provide an insertion, deletion, and substitution error correction coding method applicable to DNA storage, in order to solve the errors existing in sequencing reads under low redundancy and high coding efficiency, and to use segmented error correction to decode long sequences.

[0005] To achieve the objectives of this invention, the technical solution provided by this invention is as follows: An insertion, deletion, and replacement error correction coding method for DNA storage includes the following steps: Step 1: Based on the deep embedding framework that uses the squared Euclidean distance between embedding vectors as an approximation of the edit distance, the optimization objective is modified to focus the embedding model on small edit distances, where small edit distances are defined as edit distances < 3. Step 2: Based on deep embeddings that use the squared Euclidean distance between embedding vectors as an approximation of the edit distance, perform a greedy search for codewords selectively; Step 3: Based on the codewords constructed by greedy search, construct a Kd tree for the embedding vector of the damaged codeword segment. Use the tree search algorithm to query its k nearest neighbors in the tree, thereby identifying the codeword with the smallest edit distance to the segment and realizing segment correction. Step 4: In long sequence decoding, a segmented correction method is used to recover segment boundaries from global information. Based on a depth-first search method, a recursive ternary decoding tree is constructed to reliably decode the long sequence. Each node has three child nodes: a left child node, a middle child node, and a right child node. The left child node captures deletion errors in the segment, the middle child node captures errors such as absence of data or replacement errors, and the right child node captures insertion errors. Then, the child nodes of the child nodes undergo the same operation to decode the first segment of the undecoded sequence until the undecoded sequence is empty. By comparing the distance records after decoding, a suitable leaf node is selected as the solution. Then, the decoded codeword records are connected to form the decoded sequence. Based on the distance records, an early stopping criterion is used to terminate the branch with a large distance record early.

[0006] In step 1, the embedding model is made to focus more on small edit distances, specifically as follows: (1.1) Let s and t This represents two letters of length {A, T, G, C}. n The sequence, let Represent the edit distance between them; determine an embedding function. This makes the mapping embedding vector and With close editing distance d Traditional distance ; (1.2) It is a trainable deep embedding model with 10 1D-CNN layers and one normalization layer, where... These are learnable parameters in deep embedding models. It is a predefined loss function that represents the difference between the predicted distance and the true edit distance. The training can be expressed as the optimization of the following formula: .

[0007] (1.3) Optimize the above equation using a Siamese neural network to obtain the optimized embedding model. The sequences are mapped to their corresponding embedding vectors, and the squared Euclidean distance between the embedding vectors is used as an approximation of the edit distance. The Euclidean distance is defined as follows: .

[0008] (1.4) Modify the loss function to emphasize predictions for small edit distances. Specifically, the trained model should obtain an accurate prediction of one edit distance and ensure that the predicted distance is greater than 2 when the edit distance is greater than or equal to 2. Define the loss function as follows:

[0009] Among them, indicator function when When the condition is met, the value is 1; otherwise, the value is 0.

[0010] In step 2, the selective greedy search of codewords specifically involves the following steps: (2.1) When constructing codewords, the candidate set is first initialized to a sequence length of... n The set of all possible sequences is denoted as A( n )={A,T,G,C} n In the embedding vector Estimate the statistical distribution; (2.2) From the candidate set, codewords with fewer adjacent sequences are selected iteratively, while the candidate set is updated by filtering out codewords and their adjacent sequences within an edit distance sphere of radius 2, and the iterative process is repeated to select codewords until the candidate set becomes empty.

[0011] Specifically, the step of preferentially selecting codewords with fewer adjacent sequences is as follows: By embedding vector Using the Euclidean sphere centered on the given codeword s The edit distance sphere, the last layer of the embedding network uses batch normalization, which forces the embedding vectors to... u Follows a multivariate normal distribution N ( 0, The PDF formula is:

[0012] Utilize all possible embedding vectors Estimate the covariance matrix Apply the above formula to each embedding vector N ( 0, PDF can be easily calculated. During each iteration, codewords are selected from the sequence whose embedding vector has the lowest PDF value, prioritizing codewords with fewer adjacent sequences.

[0013] In step 3, a Kd-tree is constructed from the embedding vector of the corrupted codeword fragment using a tree search method. The specific steps are as follows: (3.1) Through the embedding vector corresponding to the codeword Construct a Kd-tree when correcting fragments First, the segment is mapped to its corresponding deep embedding vector. ; (3.2) Then, using the embedding vector, Querying the Kd-tree k The nearest neighbors found in the search k Among the nearest neighbors, the codeword with the smallest edit distance to the segment is identified as the corrected one. In order to eliminate the approximation error introduced by the deep embedding of the edit distance and improve the reliability of obtaining the nearest neighbor in the tree search method, the number of search neighbors k is increased, and then the edit distance is used for double confirmation, which eliminates the approximation error introduced by the deep embedding of the edit distance and improves the reliability.

[0014] In step 4, the recursive decoding of the first segment of the undecoded sequence involves the following steps: r It is a reading sequence, and the root node is characterized as an undecoded sequence. r An empty decoded codeword record and an empty distance record; the left child of the root node contains an undecoded sequence. Decoded codeword record Distance Record ,in for The segment correction is that the middle and right child nodes of the root are similar, but are respectively used... n and n + 1 Replace n -1, the left child node captures deletion errors in the first segment, the middle child node captures no-existence or replacement errors in the first segment, and the right child node captures insertion errors in the first segment. Then, recursively, the child nodes of the child nodes undergo the same operation, decoding length is... n -1, n , n The first segment of the undecoded sequence with +1, where each child node inherits its parent node's record list and appends its new record element.

[0015] In step 4, the step of selecting a suitable leaf node as the solution is to directly select the leaf node with the minimum distance record, and use it as the suitable leaf node.

[0016] Compared with the prior art, the beneficial effects of this invention are: (1) High coding efficiency: The edit distance deep embedding method used in this invention expresses the inherent structural properties of edit distance in the embedding space. By optimizing codeword design and selection through the embedding space, the problem of limited information rate under small codeword length is overcome, thereby storing more effective data in a limited storage space. Compared with random codeword selection, the proposed greedy search expands the cardinality of the encoded codewords. A larger cardinality of encoded codewords directly corresponds to lower redundancy and higher coding efficiency.

[0017] (2) High-efficiency error correction capability: In the embedding space, the correction process of damaged codeword segments is accelerated through an effective tree search algorithm.

[0018] (3) High reliability: The present invention provides a more refined and effective method for handling synchronization errors, ensuring that synchronization errors can be accurately handled during the decoding process, thereby improving the reliability of decoding.

[0019] (4) Low complexity: In terms of time complexity, the encoder of the encoding method proposed in this invention has a time complexity of... O ( 1 The decoder time complexity in the segmented correction process is at most O(n log n). O ( n This is far superior to the time complexity of brute-force sorting. O ( n 2 |C ( n ) | ). Attached Figure Description

[0020] Figure 1 This is a schematic diagram of a deep embedding model based on edit distance from a Siamese neural network. Figure 2 This is a flowchart of a codeword greedy search based on deep embedding. Figure 3 The flowchart for segmented correction based on deep embedding is shown below. Figure 4 The flowchart shows the long sequence decoding based on DFS. Figure 5 This diagram illustrates the time taken to correct 100,000 segments by randomly modifying the codeword once, with different k values ​​and brute-force search. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] This invention provides a method for correcting insertion, deletion, and replacement errors in DNA storage, comprising the following steps: Step 1: Using a trainable deep embedding model, the edit distance between two sequences is embedded into the regular distance between two corresponding embedding vectors, achieving deep embedding of the edit distance. Based on the deep embedding framework that uses the squared Euclidean distance between embedding vectors as an approximation of the edit distance, the optimization objective is modified to give the embedding model higher predictive ability for smaller edit distances (edit distance < 3).

[0023] Step 2: Based on deep embeddings that use the squared Euclidean distance between embedding vectors as an approximation of the edit distance, perform a greedy search for codewords.

[0024] Step 3: Based on the codewords constructed using greedy search, construct a Kd-tree for the embedding vector of the damaged codeword fragment. Use a tree search algorithm to query its k nearest neighbors in the tree, thereby identifying the codeword with the smallest edit distance to the fragment and achieving fragment correction.

[0025] Step 4: In long sequence decoding, a segmented correction method is used to recover segment boundaries from global information. Based on the depth-first search (DFS) method, a recursive ternary decoding tree is constructed to reliably decode the long sequence. Each node has three child nodes: a left child node, a middle child node, and a right child node. The left child node captures deletion errors in the segment, the middle child node captures errors such as missing segments or replacement errors, and the right child node captures insertion errors. Then, the corresponding undecoded sequence is recursively decoded for each child node, becoming its child node. This continues until the undecoded sequence is empty. By comparing the distance records after decoding, a suitable leaf node is selected as the solution. Then, the decoded codeword records are concatenated to form the decoded sequence. Based on the distance records, an early stopping criterion is used to terminate branches that exceed three times the statistical average edit distance.

[0026] Preferably, to make the embedding model more focused on small edit distances, the specific adjustments are as follows: (1.1) Let s and t This represents two letters of length {A, T, G, C}. n The sequence, let This indicates the edit distance between them. For example... Figure 1 As shown, an embedding function is determined. This makes the traditional distance between embedding vectors It can be used to estimate the edit distance between original sequences. d .

[0027] (1.2) Let It is a trainable deep embedding model with 10 1D-CNN layers and one normalization layer. These are the learnable parameters in deep models. It is a predefined loss function that represents the difference between the predicted distance and the true edit distance. The training can be expressed as an optimization of the following formula.

[0028] .

[0029] (1.3) Utilizing Siamese neural networks, such as Figure 1 Optimize the above formula to obtain an optimized embedding model. Sequences are mapped to their corresponding embedding vectors. The squared Euclidean distance between the embedding vectors is used as an approximation of the edit distance. The Euclidean distance is defined as follows: .

[0030] (1.4) To focus more on edit distance balls and adjacent sequences, the loss function is modified to emphasize predictions of small edit distances. Specifically, the model should be trained to obtain an accurate prediction of one edit distance and ensure that the predicted distance is greater than 2 when the edit distance is greater than or equal to 2. The loss function is defined as:

[0031] Among them, indicator function when When the time condition is met, the value is 1; otherwise, the value is 0. Preferably, the selective codeword greedy search, such as Figure 2 As shown, the specific steps are as follows: (2.1) When constructing codewords, the candidate set is first initialized to a sequence length of... n The set of all possible sequences is denoted as A( n )={A,T,G,C} n In the embedding vector The above estimates the statistical distribution.

[0032] (2.2) Subsequently, from the candidate set, codewords with fewer adjacent sequences are selected iteratively, while the candidate set is updated by filtering out codewords and their adjacent sequences within an edit distance sphere of radius 2, and the iterative process is repeated to select codewords until the candidate set becomes empty.

[0033] Furthermore, such as Figure 3 As shown, the specific operation of prioritizing codewords with fewer adjacent sequences is as follows: By embedding vector Using the Euclidean sphere centered on the given codeword s The edit distance sphere. The last layer of the embedding network uses batch normalization, which forces the embedding vectors to... u Follows a multivariate normal distribution N ( 0, The PDF formula is:

[0034] Utilize all possible embedding vectors Estimate the covariance matrix Apply the above formula to each embedding vector N ( 0, The PDF can be easily calculated using any of these methods. During each iteration, codewords are selected from the sequences whose embedding vectors have the lowest PDF values. Codewords with fewer adjacent sequences are preferred.

[0035] Specifically, for the embedding vector of the corrupted codeword fragment, a Kd-tree is constructed, and the tree search method is used. The specific steps are as follows: (3.1) Through the embedding vector corresponding to the codeword Construct a K-dimensional tree (Kd-tree). When correcting fragments... First, the segment is mapped to its corresponding deep embedding vector. .

[0036] (3.2) Then, using the embedding vector, Querying the Kd-tree k The nearest neighbors. In the retrieved... k Among the nearest neighbors, the codeword with the smallest edit distance to the segment is identified as the corrected one. To eliminate the approximation error introduced by the deep embedding of the edit distance and improve the reliability of obtaining the nearest neighbor in the tree search method, the number of search neighbors k is increased, and then double verification is performed using the edit distance. This eliminates the approximation error introduced by the deep embedding of the edit distance and improves reliability.

[0037] like Figure 4 As shown, the specific steps for recursively decoding the first segment of the undecoded sequence are: Let r be the reading sequence, and the root node be characterized by the undecoded sequence. r An empty decoded codeword record and an empty distance record. The left child of the root node contains an undecoded sequence. Decoded codeword record Distance Record , ,in for The segment correction. The middle and right child nodes of the root are similar, but respectively using... n and n + 1 Replace n -1. The left child node captures deletion errors in the first segment, the middle child node captures no-existence or replacement errors in the first segment, and the right child node captures insertion errors in the first segment. Then, recursively, the child nodes of the child nodes undergo the same operation, decoding a length of... n -1, n , n The first segment of the undecoded sequence with +1. Each child node inherits its parent node's list of records and appends its new record element.

[0038] Furthermore, the specific steps for selecting a suitable leaf node as the solution are as follows: directly select the leaf node with the minimum distance record, and use it as the suitable leaf node.

[0039] The following specific embodiments illustrate the feasibility of the data decoding error correction method for DNA data storage provided in this invention.

[0040] As the codeword length n increases, the increase in codeword count becomes more pronounced. Specifically, when n = 11, the greedy search based on deep embedding identifies 16.8% more codewords compared to the method of randomly selecting codewords. The coderate curve of the encoding scheme proposed in this invention is very close to the optimal redundancy of the ideal order.

[0041] By leveraging deep embedding-based fragment correction and performing tree search on the embedding vectors to reduce time complexity, codewords corrupted by a single edit operation can be corrected. The search neighborhood is increased to... When correcting 100,000 modified codewords, the number of failed corrections was 0. Furthermore, Increasing from 1 to 5 results in the smallest increase in time cost, such as... Figure 5 As shown.

[0042] DFS decoding has the ability to successfully decode failed segments and can recover segment boundaries after encountering a failed segment. Without applying external ECC, the error rate for long sequence decoding is already below 0.5%, but by integrating external RS codes, the DFS decoding method can even achieve a zero error rate at a code rate of 67.5%, surpassing the best performance of existing methods.

[0043] Finally, it should be noted that the above embodiments are merely illustrative and explanatory of the present invention, and are not intended to limit the present invention to the scope of the described embodiments. Furthermore, those skilled in the art will understand that the present invention is not limited to the above embodiments, and many more variations and modifications can be made based on the teachings of the present invention, all of which fall within the scope of protection claimed by the present invention.

Claims

1. A coding method for error correction during insertion, deletion, and replacement in DNA storage, characterized in that, Includes the following steps: Step 1: Based on the deep embedding framework that uses the squared Euclidean distance between embedding vectors as an approximation of the edit distance, the optimization objective is modified to focus the embedding model on small edit distances, where small edit distances are defined as edit distances < 3. Step 2: Based on deep embeddings that use the squared Euclidean distance between embedding vectors as an approximation of the edit distance, perform a greedy search for codewords selectively; Step 3: Based on the codewords constructed by greedy search, construct a Kd tree for the embedding vector of the damaged codeword fragment. Use the tree search algorithm to query its k nearest neighbors in the tree, thereby identifying the codeword with the smallest edit distance to the fragment and realizing fragment correction. Step 4: In long sequence decoding, a segmented correction method is used to recover segment boundaries from global information. Based on the depth-first search method, a recursive ternary decoding tree is constructed to reliably decode the long sequence. Each node has three child nodes: left child node, middle child node, and right child node. The left child node captures deletion errors in the segment, the middle child node captures errors such as absence or replacement errors in the segment, and the right child node captures insertion errors in the segment. Then, the child nodes of the child nodes are recursively processed in the same way to decode the first segment of the undecoded sequence until the undecoded sequence is empty. By comparing the distance records after decoding, a suitable leaf node is selected as the solution. Then, the decoded codeword records are connected to form a decoded sequence. Based on the distance records, the early stopping criterion is adopted to terminate the branch with a large distance record in advance. In step 1, the embedding model is made to focus more on small edit distances, specifically by: (1.1) Let s and t This represents two letters of length {A, T, G, C}. n The sequence, let Represent the edit distance between them; determine an embedding function. This makes the mapping embedding vector and With close editing distance d Traditional distance ; (1.2) It is a trainable deep embedding model with 10 1D-CNN layers and one normalization layer, where... These are learnable parameters in deep embedding models. It is a predefined loss function that represents the difference between the predicted distance and the true edit distance. The training can be expressed as the optimization of the following formula: ； (1.3) Optimize the above equation using a Siamese neural network to obtain the optimized embedding model. The sequences are mapped to their corresponding embedding vectors, and the squared Euclidean distance between the embedding vectors is used as an approximation of the edit distance. The Euclidean distance is defined as follows: ； (1.4) Modify the loss function to emphasize predictions for small edit distances. Specifically, the trained model should obtain an accurate prediction of one edit distance and ensure that the predicted distance is greater than 2 when the edit distance is greater than or equal to 2. Define the loss function as follows: ； Among them, indicator function when When the condition is met, the value is 1; otherwise, the value is 0.

2. The method for error correction coding in DNA storage for insertion, deletion, and replacement according to claim 1, characterized in that, In step 2, the selective greedy search of codewords specifically involves the following steps: (2.1) When constructing codewords, the candidate set is first initialized to a sequence length of... n The set of all possible sequences is denoted as A( n )={A,T,G,C} n In the embedding vector Estimate the statistical distribution; (2.2) From the candidate set, codewords with fewer adjacent sequences are selected iteratively, while the candidate set is updated by filtering out codewords and their adjacent sequences within an edit distance sphere of radius 2, and the iterative process is repeated to select codewords until the candidate set becomes empty.

3. The method for error correction coding in DNA storage for insertion, deletion, and replacement according to claim 2, characterized in that, The specific operation of prioritizing the selection of codewords with fewer adjacent sequences is as follows: By embedding vector Using the Euclidean sphere centered on the given codeword s The edit distance sphere, the last layer of the embedding network uses batch normalization, which forces the embedding vectors to... u Follows a multivariate normal distribution The PDF formula is: ； Utilize all possible embedding vectors Estimate the covariance matrix Apply the above formula to each embedding vector N ( 0, PDF can be easily calculated. During each iteration, codewords are selected from the sequence whose embedding vector has the lowest PDF value, prioritizing codewords with fewer adjacent sequences.

4. The method for error correction coding in DNA storage for insertion, deletion, and replacement according to claim 2, characterized in that, In step 3, a Kd-tree is constructed from the embedding vector of the corrupted codeword fragment. The tree search method is then used, and the specific steps are as follows: (3.1) Through the embedding vector corresponding to the codeword Construct a Kd-tree when correcting fragments First, the segment is mapped to its corresponding deep embedding vector. ; (3.2) Then, using the embedding vector, Querying the Kd-tree k The nearest neighbors found in the search k Among the nearest neighbors, the codeword with the smallest edit distance to the segment is identified as the corrected one; in order to eliminate the approximation error introduced by the deep embedding of the edit distance, the number of search neighbors k is increased, and then double confirmation is performed using the edit distance, thus eliminating the approximation error introduced by the deep embedding of the edit distance.

5. The method for error correction coding in DNA storage for insertion, deletion, and replacement according to claim 1, characterized in that, In step 4, the first segment of the undecoded sequence is recursively decoded. The specific steps are as follows: It is a reading sequence, and the root node is characterized as an undecoded sequence. An empty decoded codeword record and an empty distance record; the left child of the root node contains an undecoded sequence. Decoded codeword record Distance Record ,in for The segment correction is that the middle and right child nodes of the root are similar, but are respectively used... n and n +1 Replace The left child node captures deletion errors in the first segment, the middle child node captures no-existence or replacement errors in the first segment, and the right child node captures insertion errors in the first segment. Then, recursively, the child nodes of the child nodes undergo the same operation, decoding a length of... , , The first segment of the undecoded sequence, each child node inherits the record list of its parent node and appends its new record elements.

6. The method for error correction coding in DNA storage for insertion, deletion, and replacement according to claim 1, characterized in that, In step 4, the step of selecting a suitable leaf node as the solution is to directly select the leaf node with the minimum distance record, and use it as the suitable leaf node.

Images