A DNA storage clustering method for dynamically updating a hash index

Abstract

The application discloses a DNA storage clustering method for dynamically updating a hash index, comprising selecting DNA fragments at three positions of an initial sequence, converting the three DNA fragments into corresponding hash values through hash code calculation, and mapping the hash values into three hash tables. In a subsequent sequence to be clustered, DNA fragments are selected at the three positions, hash codes are calculated respectively, and collision detection is performed, so as to determine whether the sequence belongs to the same cluster. When the number of sequences in the cluster reaches 5, the core index set is updated once, so as to avoid the error index from leading to the error shunting of the sequence. The DNA storage clustering method for dynamically updating the hash index can significantly improve the similarity of the sequences in the cluster due to the timely updating of the index, and avoids the error shunting of the sequence caused by the error index. The method has the advantages of improving the accuracy and reliability of the clustering, and provides good support for large-scale DNA data storage.

Description

Technical Field

[0001] The present invention relates to the technical field of clustering methods, and in particular to a DNA storage clustering method for dynamically updating hash indexes. Background Art

[0002] With the continuous advancement of science and technology, the total amount of data created by human society is growing exponentially. Therefore, finding a more efficient and reliable new storage medium has become a top priority. Compared to traditional media, DNA storage is becoming an ideal storage method due to its small size, long storage time, and low energy consumption. DNA storage uses encoding algorithms to map data to DNA sequences, and then uses sequencing and decoding processes to restore the original data. It has the potential to become a low-cost and stable storage solution for massive data storage. In 2017, Erlich et al. proposed a fountain encoding scheme using Luby transform coding to construct DNA sequences. This scheme encoded a 2.14MB file into 72,000 oligonucleotide sequences and stored them in a DNA pool. Sequencing and decoding experiments were then completed, achieving complete data recovery. In 2022, Ping et al. designed a "yin-yang" encoding and decoding scheme, inspired by the ancient concept of yin and yang. This yin-yang encoding scheme provides a high-density and highly stable conversion method for DNA storage, and was validated through in vitro and in vivo experiments, demonstrating its feasibility.

[0003] Index-based DNA clustering methods create data structures from fragments of DNA sequences, and are used to address the clustering problem of large-scale DNA sequence data. This method uses indexes to quickly match and align DNA sequences, thereby achieving efficient clustering tasks. However, during the data reading process of DNA storage, due to insufficient focus sensitivity of the sequencer at the beginning and end of the sequencing task, the in vitro stored sequences are prone to errors such as base insertions, substitutions, and deletions at the front and back ends. Therefore, constructing indexes at the front and back ends of the sequence can lead to incorrect index construction. This incorrect indexing directly leads to the incorrect diversion of the sequences to be clustered, thereby increasing the redundancy of the sequences within the clusters after clustering and reducing the clustering effect. Summary of the Invention

[0004] In response to the above-mentioned defects in the existing technology, this application proposes a DNA storage clustering method with dynamically updated hash indexes, which can reduce the redundancy of sequences within the cluster by more than 10% and also increase the sequence reconstruction rate to more than 99%.

[0005] To achieve the above objectives, the technical solution of the present application is: a DNA storage clustering method for dynamically updating hash indexes, comprising:

[0006] Step 1: Create a front hash table, a middle hash table, and a back hash table. Each table includes a core index value Baseindex and a core key value Basevalue.

[0007] Step 2: Build an initial core index, select DNA fragments from three specific positions, and obtain the hash code corresponding to each DNA fragment;

[0008] Step 3: Map the hash codes of the three DNA fragments to the front hash table, middle hash table, and back hash table respectively through hash mapping;

[0009] Step 4: During hash mapping, the hash code values ​​of the three positions are used as the core key value Basevalue, and the index of the sequence is written into the corresponding hash table as the core index value Baseindex;

[0010] Step 5: For the subsequent sequences to be clustered, the DNA fragments at three specific positions are still taken and the hash code value of each position is obtained;

[0011] Step 6: Use conflict detection to find sequences with the same core key value Basevalue, and then determine whether they belong to the same cluster;

[0012] Step 7: Map the hash code value and index of the three positions of the sequence to be clustered to the front hash table, middle hash table and back hash table respectively as the core key value Basevalue and core index value Baseindex of a new cluster.

[0013] Furthermore, the above method also includes:

[0014] Step 8: When the number of sequences in each cluster reaches N, all sequences in the cluster vote on three positions to update the Basevalue value of the cluster in the front hash table, middle hash table, and back hash table, while the Baseindex value remains unchanged;

[0015] Step 9: After each sequence to be clustered belongs to the corresponding cluster, clusters with fewer than N-2 sequences are further classified; if there are no sequences in the same cluster, they are directly discarded;

[0016] Step 10: Get each sequence in the cluster after clustering.

[0017] Furthermore, the hash code is obtained as follows:

[0018] x0a (n-1) +x1a (n-2) +x2a (n-3) +...+x (n-2) a+x (n-1)a 0 (1)

[0019] Here, x is the base value, A is assigned 1, T is assigned 2, G is assigned 3, C is assigned 4, n is the base length of the DNA fragment, and a is equal to 2.

[0020] Furthermore, a conflict detection method is used to find out whether there are sequences with the same core key value, specifically:

[0021] If the hash code value at two positions of the sequence to be clustered conflicts with the core key value Basevalue in the corresponding two hash tables and the core index value Baseindex is consistent, then it means that the sequence to be clustered and the core sequence belong to the same cluster; if the hash code value at one position conflicts with the core key value Basevalue, or there is no hash conflict, then it means that the sequence to be clustered and the core sequence do not belong to the same cluster.

[0022] Furthermore, if the hash code value conflicts with the core key value Basevalue, a linear probing method will be used to resolve the conflict.

[0023] Furthermore, the specific three positions refer to the front-middle, middle, and back-middle positions of the sequence.

[0024] By adopting the above technical solution, the present invention can achieve the following technical effects:

[0025] 1. The present invention has achieved ideal results in four reliability aspects, namely normalized mutual information, adjusted Rand coefficient, adjusted mutual information, and FMI coefficient;

[0026] 2. The present invention reduces the redundancy ratio in the cluster by updating the core index set in real time, improves the accuracy of the DNA storage data reading process, and thus improves the clustering effect of reconstructing the original sequence;

[0027] 3. The invention adopts hash conflict detection to search index and real-time update index, which avoids sequence error diversion and reduces redundancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Flowchart for a DNA storage clustering method with dynamically updated hash index. DETAILED DESCRIPTION

[0029] The following will provide a clear and complete description of the technical solutions implemented in the present invention in conjunction with the accompanying drawings. It should be understood that the embodiments described are only part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0030] In this method, three DNA fragments are selected from the sequence and hash codes are calculated for collision detection. During the clustering process, the core index set is updated when the number of sequences within a cluster reaches N (which can be 5). When the number of sequences within a cluster reaches N-2 (which can be 3), the cluster is considered an erroneous cluster and discarded. In this example, a DNA storage clustering method with dynamically updated hash indexes was used to cluster a sequence file (27 sequences to be clustered), and the implementation conditions were as described above.

[0031] Step 1: Create a front hash table, a middle hash table, and a back hash table. Each table includes a core index value Baseindex and a core key value Basevalue.

[0032] Step 2: Build an initial core index, select DNA fragments from three specific positions, and obtain the hash code corresponding to each DNA fragment;

[0033] Specifically, three DNA fragments are selected from the middle front, middle, and middle back positions, such as TGCACTCA, TCAGCAGC, and ACGATCAC, and their hash codes are calculated;

[0034] Step 3: Use hash mapping to map the hash codes of the three DNA fragments to the front hash table, middle hash table, and back hash table respectively.

[0035] Specifically, the hash codes of the three DNA fragments are 641, 606, and 532;

[0036] Step 4: During hash mapping, the hash code values ​​of the three positions are used as the core key value Basevalue, and the index of the sequence is written into the corresponding hash table as the core index value Baseindex;

[0037] Step 5: For the subsequent sequences to be clustered, the DNA fragments at three specific positions are still taken and the hash code value of each position is obtained;

[0038] Step 6: Use conflict detection to find sequences with the same core key value Basevalue, and then determine whether they belong to the same cluster;

[0039] Specifically, if the hash code value at two positions of the sequence to be clustered conflicts with the core key value Basevalue in the corresponding two hash tables and the core index value Baseindex is consistent, then it means that the sequence to be clustered and the core sequence belong to the same cluster; if the hash code value at one position conflicts with the core key value Basevalue, or there is no hash conflict, then it means that the sequence to be clustered and the core sequence do not belong to the same cluster; if the hash code value conflicts with the core key value Basevalue, then the linear probing method will be used to resolve the conflict.

[0040] Step 7: Map the hash code value and index of the three positions of the sequence to be clustered to the front hash table, middle hash table and back hash table respectively as the core key value Basevalue and core index value Baseindex of a new cluster.

[0041] Step 8: When the number of sequences in each cluster reaches 5, all sequences in the cluster vote on three positions to update the Basevalue value of the cluster in the front hash table, middle hash table, and back hash table, while the Baseindex value remains unchanged;

[0042] Step 9: After each sequence to be clustered belongs to the corresponding cluster, clusters with less than 3 sequences are classified; if there are no sequences in the same cluster, they are discarded directly;

[0043] Step 10: Get each sequence in the cluster after clustering.

[0044] The specific clustering sequence is shown in Table 1

[0045] Table 1 DNA sequences

[0046]

[0047]

[0048]

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A DNA storage clustering method with dynamic update of hash index, characterized in that: include: Step 1: Create a front hash table, a middle hash table, and a back hash table. Each table includes a core index value Baseindex and a core key value Basevalue. Step 2: Build an initial core index, select DNA fragments from three specific positions, and obtain the hash code corresponding to each DNA fragment; Step 3: Map the hash codes of the three DNA fragments to the front hash table, middle hash table, and back hash table respectively through hash mapping; Step 4: During hash mapping, the hash code values ​​of the three positions are used as the core key value Basevalue, and the index of the sequence is written into the corresponding hash table as the core index value Baseindex; Step 5: For the subsequent sequences to be clustered, the DNA fragments at three specific positions are still taken and the hash code value of each position is obtained; Step 6: Use conflict detection to find sequences with the same core key value Basevalue, and then determine whether they belong to the same cluster; Step 7: Map the hash code value and index of the three positions of the sequence to be clustered to the front hash table, middle hash table and back hash table respectively as the core key value Basevalue and core index value Baseindex of a new cluster; Step 8: When the number of sequences in each cluster reaches N, all sequences in the cluster vote on three positions to update the Basevalue value of the cluster in the front hash table, middle hash table, and back hash table, while the Baseindex value remains unchanged; Step 9: After each sequence to be clustered belongs to the corresponding cluster, clusters with fewer than N-2 sequences are further classified; if there are no sequences in the same cluster, they are directly discarded; Step 10: Get each sequence in the cluster after clustering; The hash code is obtained as follows: x0a (n-1) +x1a (n-2) +x2a (n-3) +...+x (n-2) a+x (n-1) a 0 Where x is the base assignment, A is assigned 1, T is assigned 2, G is assigned 3, C is assigned 4, n is the length of the DNA fragment in bases, and a is equal to 2; Use conflict detection to find sequences with the same core key value, specifically: If the hash code value at two positions of the sequence to be clustered conflicts with the core key value Basevalue in the corresponding two hash tables and the core index value Baseindex is consistent, then it means that the sequence to be clustered and the core sequence belong to the same cluster; if the hash code value at one position conflicts with the core key value Basevalue, or there is no hash conflict, then it means that the sequence to be clustered and the core sequence do not belong to the same cluster.

2. The DNA storage clustering method for dynamically updating hash indexes according to claim 1, characterized in that: If the hash code value conflicts with the core key value Basevalue, a linear probing method will be used to resolve the conflict.

3. The DNA storage clustering method for dynamically updating hash indexes according to claim 1, characterized in that: The specific three positions refer to the front-middle, middle, and back-middle positions of the sequence.

Images