A system and encoding and decoding method suitable for PB-level data DNA storage

Abstract

The present application discloses a system and encoding / decoding method suitable for DNA storage of PB-level data. The data storage system includes multiple DNA clusters. The DNA cluster contains a data chain formed after binary data encoding, an XOR redundant chain generated by the data chain, an address chain of the DNA cluster at the storage system location, a basic information chain of the stored file, and a redundant chain generated by the XOR of the address chain and the file information chain. When a save operation is performed on a file with a preset suffix, a user-mode monitoring program receives an execution instruction and copies the file to a working directory, and encodes or decodes the file according to the file suffix. The result of the encoding or decoding operation is copied back to the original file directory. The present application enables the total capacity of the DNA database to reach above the PB level, allowing DNA storage to realize the application potential of high data density. At the same time, it can realize the automatic encoding of binary data and the automatic decoding of base sequences, streamline and automate the entire encoding and decoding process, and facilitate subsequent data management and archiving.

Description

Technical Field

[0001] The present application belongs to the field of information technology, and in particular relates to a system and encoding / decoding method suitable for PB-level data DNA storage. Background Art

[0002] With the rapid development of the information industry, the amount of data generated by various industries, represented by the internet, is growing. DNA (deoxyribonucleic acid) storage is a new data storage method. It encodes binary files into a sequence consisting of the four DNA bases A (adenine), T (thymine), G (guanine), and C (cytosine). DNA chains are then synthesized according to this sequence to store data. Due to the structural characteristics of DNA itself, encoding binary data into base sequences requires consideration of certain constraints. The number of bases per DNA chain needs to be controlled within a certain scale. However, the total amount of data that needs to be stored continues to increase with the development of the information industry, resulting in an increasing proportion of indexes in the entire DNA storage system.

[0003] In related work, Cao et al. (Cao B, Zhang X, Cui S, et al. Adaptive coding for DNA storage with high storage density and low coverage [J]. NPJ systems biology and applications, 2022, 8 (1): 23.) stored the address and data in two separate DNA pools, the address was stored in the index pool, and the data was stored in the archive pool. When reading the data, the sequences in all index pools were first read, sequenced and decoded. Then, the access request was synthesized into a complementary sequence of the address bit, and the DNA sequence to be retrieved was separated by PCR amplification and purification in the archive pool. Finally, the separated DNA sequence was sequenced and decoded into the original information to obtain the original data; Wang et al. (Wang P, Mu Z, Sun L, et al. Hidden addressing encoding for DNA storage [J]. Frontiers in Bioengineering and Biotechnology, 2022, 10: 916615.) The index is optimized on the model of the original index data, and a part of the data segment is used to replace the index to distinguish between sequences. The fountain code is used to encode the data and address, and the sequences that do not meet the conditions are filtered out. In this way, the actual data storage density is increased.

[0004] At present, as represented by the above-mentioned document [1], the data structure stored in traditional technology is generally index plus data. However, the disadvantage of this technology is that the pre-set large total storage space will make the index occupy a higher proportion in the DNA chain, thereby reducing the data density. Although the document [2] uses the data block itself to replace the index, so that the index itself no longer occupies bases and improves the data storage density, only a 4-base data segment is used as the actual index, and the total storage space of the entire system is not high. In the existing DNA storage method, the base sequence occupied by the address is too long. The larger the storage system, the longer the address and the lower the data density, which limits the expansion of the DNA storage scale.

[0005] At the same time, there is currently no file system that can automatically encode and decode binary data and base sequences. The operations required for manual encoding and decoding are cumbersome, and when the amount of related data information reaches a certain scale, it will cause management problems. Summary of the Invention

[0006] The purpose of this application is to address the deficiencies of the above-mentioned prior art and provide a system and encoding and decoding method suitable for PB-level data DNA storage, so as to at least solve one of the above-mentioned prior art problems.

[0007] In order to solve the above technical problems, the technical solution adopted in this application is:

[0008] A system suitable for DNA storage of petabyte-level data, characterized in that the system includes multiple DNA clusters, each of which contains a data chain formed by binary data encoding, an XOR redundant chain generated by the data chain, an address chain of the DNA cluster's location in the storage system, a basic information chain of the stored file, and a redundant chain generated by the XOR of the address chain and the file information chain; wherein:

[0009] The two ends of the data chain are specific file random primers. After removing the primers, the beginning is the address segment, which determines the logical position of the data chain in the DNA cluster; the middle segment of the data chain is the base sequence of the stored data; and the end of the data chain is the error correction code.

[0010] The XOR redundant chain structure generated by the data chain is as follows: there are file-specific random primers at both ends, followed by the address segment of the previous sequence increment; every five data chains generate an XOR redundant chain, and the redundant bases are after the address segment;

[0011] The address chain includes a multi-level address for determining the position of the DNA cluster in the entire storage system; both ends of the address chain are also specific random primers of the address chain, the address chain specific primer is followed by the specific primer of the file to which it belongs, followed by the multi-level address of the file in the entire file system, the head of the address chain is followed by the address content of the logically largest address in the DNA cluster, and finally the error correction redundant information.

[0012] The file information chain is used to store detailed file information, including file name, file type, modification date, file size, and file permissions. Two pairs of consecutive random primers are located at each end of the chain: one specific to the chain and the other specific to the file. Following the primers are file details and an error correction code.

[0013] As a preferred embodiment, the multi-level address in the address chain includes a TB segment, a GB segment, and a frag segment. As a preferred embodiment, the error correction code in the data chain is used to correct insertion, substitution, or deletion errors in the base sequence during data synthesis, storage, and sequencing.

[0014] As a preferred method, the data XOR redundant chain is mainly used for indirect loss of the chain caused by excessive data chain errors that cannot be restored, and the chain can be completely restored through XOR operation.

[0015] Furthermore, the cluster metadata in the system is used to record information about each DNA cluster. The information in the cluster metadata includes:

[0016] Cluster ID;

[0017] Cluster status, which is either idle or occupied;

[0018] The file ID corresponding to the cluster data.

[0019] As a preferred approach, when the cluster state is idle, the file ID corresponding to the cluster data is a default value of -1.

[0020] Furthermore, the information in the file metadata in the system includes:

[0021] Randomly generated file IDs are used to identify different files with the same name and create indexes;

[0022] The properties of the file itself;

[0023] The number of clusters occupied by the file;

[0024] The number of the cluster that the file occupies.

[0025] Based on the same inventive concept, the present application also provides a file encoding and decoding method running in the system suitable for PB-level data DNA storage, which is characterized in that when a save operation is performed on a file with a preset suffix, the user-state monitoring program receives an execution instruction and copies the file to the working directory, and encodes or decodes the file according to the file suffix. After the encoding or decoding operation is completed, the result is copied back to the original file directory.

[0026] As a preferred embodiment, the encoding and decoding method includes:

[0027] For each file, when the kernel processes the first save operation, the data storage system creates two variables, namely the first variable and the second variable. The first variables and the second variables of different files are independent of each other and are stored in a common data structure. At the same time, two variables of type spinlock_t are created to protect the data from read and write conflicts in the case of multiple threads.

[0028] In the initial state, both the first and second variables are 0. Only when the first variable is judged to be 0 will the kernel send an instruction to the monitoring program, and after sending, the first variable is set to 1. After that, the loop uses the spin lock to detect whether the second variable is 1. When the second variable is 1, the loop is jumped out and the whole process ends.

[0029] When the external monitoring program receives the instruction from the kernel, it copies the file to the working directory. During this process, it enters the kernel again. At this time, it detects that the first variable corresponding to the file is set to 1, sets the second variable to 1, clears the first variable and returns directly;

[0030] When the previous process in the loop detects that the second variable is 1, it jumps out of the loop, clears the second variable and returns. At this time, the task in the kernel of this encoding and decoding operation is completed, and the remaining encoding operations are implemented in the user layer. After completion, the encoded and decoded data will be copied back to the original directory.

[0031] As a preferred method, when an electronic file is input and the system receives the encoding instruction, a file number is created to record various attributes of the file;

[0032] When the file is larger than a set value, the file is divided into pieces of fixed size;

[0033] Based on the relevant information of the cluster metadata in the storage system, check the ID of the idle cluster, assign the ID of the idle cluster to each shard of the file, and update the relevant records in the cluster metadata;

[0034] The file number and the occupied cluster number are recorded in the file metadata of the storage system.

[0035] As a preferred method, when a specific electronic file needs to be restored from a base sequence, the cluster ID occupied by the file is first obtained from the file metadata in the file system, the DNA in the corresponding cluster is amplified and sequenced, and finally a decoding program is used to restore the base sequence to the original electronic file.

[0036] Compared with the prior art, this application has the following beneficial effects:

[0037] The present application proposes a system and encoding / decoding method suitable for DNA storage of PB-level data. By adopting a secondary indexing method, the upper limit of DNA storage capacity is within a relatively large scale, and the additional cost in the address chain is negligible, which can make the total capacity of the DNA database reach above the PB level. While breaking through the current storage size limit and raising the upper limit of the existing DNA data storage level, the proportion of indexes in the entire DNA storage system is reduced as much as possible, allowing DNA storage to realize the application potential of high data density. At the same time, the present application modifies the file system to realize automatic encoding of binary data and automatic decoding of base sequences, streamlines and automates the entire encoding and decoding process, simplifies the part that requires manual operation, and facilitates subsequent data management and archiving. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The storage format of the data link in the cluster.

[0039] Figure 2 This is the storage format of the address chain in the cluster.

[0040] Figure 3 This is the cluster metadata field in the storage system.

[0041] Figure 4 This is the file metadata field in the storage system.

[0042] Figure 5 Schematic diagram of the automatic encoding and decoding process of the file system.

[0043] Figure 6 Flowchart for coding electronic documents.

[0044] Figure 7 Decoding flow chart. DETAILED DESCRIPTION

[0045] In the process of using DNA to encode raw binary data, since the data storage size of each DNA chain is limited, it is necessary to distinguish them through indexes to ensure that the original data can be restored after decoding. If the scale of the data to be stored is as large as possible, the proportion of the index in the entire storage system will also be large, affecting the actual data storage density. To solve this problem, this application applies the idea of ​​multi-level indexing to DNA storage. For unused indexes, no additional space is required to store them. Only the used indexes are stored. For the repeated parts of the indexes in different DNA chains, this application takes them out and stores them separately, instead of storing the same data in multiple different DNA chains.

[0046] Overall, this application divides the storage system into multiple domains. For example, each domain can store up to 200KB of data. DNA chains within a domain are organized using intra-domain indexes. A parameter called Address_max is set. Within each domain, the DNA chain with the address Address_max stores the common index of all DNA chains within the domain within the entire storage system, such as the PB block or TB block it resides in, as well as the domain's own number. This storage method improves the utilization of payload data in the DNA storage system.

[0047] The encoding and decoding of the file system will be automatically triggered by the saving of binary files or base sequences. The system will first copy the file to be processed to the working area, and then determine whether it should be encoded or decoded based on the suffix name and perform related operations. For different files to be encoded and decoded at the same time, this application also designs a set of strategies to avoid the resource competition problem in the related code in the Linux kernel, so that multiple tasks can be carried out simultaneously without affecting each other and causing confusion.

[0048] The specific implementation of this application is further described in detail below with reference to the accompanying drawings.

[0049] 1. Storage system architecture

[0050] Architecturally, the data storage system of the DNA database is composed of different DNA clusters, which include DNA data chains formed after binary data encoding, XOR redundant chains generated by the data chains, address chains of the DNA clusters located in the storage system, basic information chains of the stored files, and redundant chains generated by XORing the address chains and file information chains. In terms of binary data that can be stored, preferably, the theoretical maximum capacity of the entire storage system is 1PB, and the theoretical maximum capacity of each DNA cluster is 200KB. In the DNA cluster, the format of the data chain and the address chain is as follows: Figure 1 and Figure 2 shown.

[0051] In the cluster, the two ends of each data chain are specific file random primers. After removing the primers, the beginning is the address segment, which can be used to determine the logical position of the data chain in the cluster; the middle segment of the data chain is the base sequence of the stored data; the end of the data chain is the error correction code, which is used to correct the insertion, replacement, and deletion errors of the base sequence during the synthesis, storage, and sequencing process.

[0052] The XOR redundant chain structure generated by the data chain is: there are specific file random primers at both ends, followed by the address segment of the previous sequence increment; every five data chains generate an XOR redundant chain, and the redundant base is after the address segment.

[0053] Since the total storage size of the storage system is 1 PB, a multi-level address can be set for each cluster to determine the location of the cluster in the entire storage system.

[0054] The multi-level address includes a TB segment, a GB segment and a frag segment. This information is stored in an address chain. Both ends of the address chain are also specific random primers of the address chain. After the address chain specific primer is a specific primer of the file to which it belongs, followed by the multi-level address of the file in the entire file system, followed by the address content that is the logically largest address in the cluster, and finally the error correction redundant information. This method can distinguish between data chains and address chains. The present application divides DNA chains with a common prefix in the address into the same cluster, wherein the address chain serves as a common prefix for the entire cluster, which can reduce the proportion of bases occupied by the address in the storage system and improve storage density. As the scale of the storage system expands, within a relatively large scale, the addresses required by the traditional indexing method are getting longer and longer, and this has little effect on the present application's method of using the address chain as a common prefix for the DNA chains in the cluster.

[0055] The file information chain is used to store detailed information of the file, including information such as file name, file type, modification date, file size and file permissions: there are two pairs of continuous random primers at both ends of the information chain, one is a random primer specific to the information chain, and the other is a random primer specific to the file to which it belongs; after the primers are file detailed information and error correction code.

[0056] At the computer level, the cluster metadata of the storage system records the information of each cluster, such as Figure 3 As shown, cluster metadata includes the following information:

[0057] 1) Cluster ID;

[0058] 2) The state of the cluster; the state is idle or occupied;

[0059] 3) File ID corresponding to the cluster data; when the cluster status is idle, this item has a default value of -1.

[0060] In addition to cluster metadata, the storage system also stores file metadata, which makes it easy to query detailed information about a single file, such as Figure 4 As shown, the file metadata contains the following information:

[0061] 1) File ID, randomly generated, used to identify different files with the same name and facilitate index creation;

[0062] 2) File attributes, such as file name, file size, creation time, etc.

[0063] 3) the number of clusters occupied by the file;

[0064] 4) The number of the cluster occupied by the file.

[0065] 2. Automatic encoding and decoding of the file system

[0066] The relevant code of the ext4 file system in the Linux kernel is modified so that when a user saves a file with a specific suffix, an instruction is sent to the user-mode monitoring program. The monitoring program then copies the file to the working directory (for example, the monitoring program uses the shell command cp in Linux to copy the file to the working directory) and encodes or decodes the file according to the file suffix. The result is copied back to the original directory after completion, and the user does not need to manually encode or decode the file.

[0067] The specific implementation is as follows: For each file, when the kernel first saves it, the system automatically creates two variables: netlink_state and receive_from_user. These variables are independent for different files and stored in a common data structure. Furthermore, two variables of type spinlock_t are used: netlink_lock and receive_lock. These variables are used to protect data from read and write conflicts in multi-threaded environments. Initially, both netlink_state and receive_from_user are 0. Only when netlink_state is 0 does the kernel send instructions to the monitoring program. After sending instructions, netlink_state is set to 1 to prevent repeated instructions. A spin lock is then used to detect whether receive_from_user is 1. When receive_from_user is 1, the loop is broken, ending the entire process.

[0068] When the external monitoring program receives the instruction from the kernel, it copies the file to the working directory. During this process, it enters the kernel again. At this time, since the netlink_state corresponding to the file is detected to be 1, the receive_from_user is set to 1, the netlink_state variable is cleared and the program returns directly. The previous process in the loop detects that receive_from_user is 1 and then jumps out of the loop, clears the receive_from_user variable and returns. At this point, the task in the kernel for this encoding and decoding operation is completed, and the remaining encoding operations are all implemented in the user layer. After completion, the encoded and decoded data will be copied back to the original directory (for example, the file can be copied back to the original directory through the shell command cp in Linux), so that the user can directly see the result.

[0069] 3. Encoded electronic files

[0070] When an electronic file is input and the system receives the encoding instruction, a file number is created to record the various attributes of the file. When the file is larger than 200KB, the file is divided into fixed 200KB fragments. When the file is less than 200KB, it is processed as the original size. Based on the cluster metadata related information in the storage system, the ID of the free cluster is checked, and the ID of the free cluster is assigned to each fragment of the file and the relevant records in the cluster metadata are updated. The file number and the occupied cluster number are recorded in the file metadata in the storage system.

[0071] 4. Restore the original electronic file from the base sequence

[0072] When a specific electronic file needs to be restored from a base sequence, the cluster ID occupied by the file is first obtained from the file metadata in the file system, the DNA in the corresponding cluster is amplified, sequenced, and other operations are performed, and finally a decoding program is used to restore the base sequence to the original electronic file.

[0073] In summary, this application divides the 1PB space into multiple clusters and stores data information and index information separately in the clusters. Compared with the traditional indexing method of directly attaching the address to the prefix of the DNA chain without using secondary index or multi-level index, the strategy of using secondary index greatly reduces the proportion of bases required for the address in the entire DNA data storage.

[0074] In addition, this application modifies the ext4 file system in the Linux kernel. When saving files with specific suffixes, it will use the netlink communication mechanism to communicate with the user-state monitoring program, copy the file to the working directory for encoding and decoding, and then copy the result back to the original directory. During this process, in order to prevent confusion caused by multiple processes running at the same time, the system sets the netlink_state parameter and receive_from_user parameter for each file separately for process control, and uses mutex locks to limit access to parameters in a multi-process environment.

[0075] Existing technologies generally use a primary index, which directly places the address at the head of the DNA chain to distinguish different DNA chains and file information. If this method is used to construct a petabyte-level index, the base sequence length occupied by the address in each DNA chain is at least 26.5. This application uses a secondary index, which stores DNA chains with the same address prefix in a cluster. Only one address chain in the cluster is required to identify the common prefix. Under this method, the base sequence length occupied by the address in each DNA chain is 8, and there is only one address chain in approximately every 10,000 DNA chains in the cluster, and the base sequence length of the address chain does not exceed 200. Therefore, the number of bases required for the address when storing petabyte-level data is greatly reduced, and data density is improved.

[0076] The advantage of this application is that the use of secondary indexes reduces the proportion of bases required for storage addresses in the entire DNA storage system, improves coding density, and stores cluster metadata and file metadata, making it convenient to query the correspondence and related information between clusters and files in the system. At the same time, modifying the file system automates the file encoding and decoding process, reducing manual operations, and preventing resource competition problems under multiple processes through mutex locks and state parameters, thereby improving system stability. After testing, it was verified that this application is feasible.

[0077] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which fall within the scope of protection of this application.

Claims

1. A system suitable for DNA storage of PB-level data, characterized in that: The system includes multiple DNA clusters, each of which includes a data chain formed by binary data encoding, an XOR redundant chain generated by the data chain, an address chain of the DNA cluster at the storage system location, a basic information chain of the stored file, and a redundant chain generated by XORing the address chain and the file information chain; wherein: The two ends of the data chain are specific file random primers. After removing the primers, the beginning is the address segment, which determines the logical position of the data chain in the DNA cluster; the middle segment of the data chain is the base sequence of the stored data; and the end of the data chain is the error correction code. The XOR redundant chain structure generated by the data chain is as follows: there are file-specific random primers at both ends, followed by the address segment of the previous sequence increment; every five data chains generate an XOR redundant chain, and the redundant bases are after the address segment; The address chain includes a multi-level address for determining the location of the DNA cluster in the entire storage system; both ends of the address chain are also specific random primers of the address chain, followed by a specific primer for the file to which it belongs, followed by the multi-level address of the file in the entire file system, followed by an address content that is the logically largest address in the DNA cluster, and finally error correction redundant information; The file information chain is used to store detailed information of the file, including information such as file name, file type, modification date, file size and file permissions: there are two pairs of continuous random primers at both ends of the information chain, one is a random primer specific to the information chain, and the other is a random primer specific to the file to which it belongs; after the primers are file detailed information and error correction code.

2. The system for storing PB-level data in DNA according to claim 1, characterized in that: The multi-level addresses in the address chain include a TB segment, a GB segment, and a frag segment.

3. The system for storing PB-level data in DNA according to claim 1, characterized in that: The error correction code in the data chain is used to correct insertion, substitution or deletion errors in the base sequence during data synthesis, storage and sequencing.

4. The system for storing PB-level data in DNA according to any one of claims 1 to 3, characterized in that: The cluster metadata in the system is used to record the information of each DNA cluster. The information in the cluster metadata includes: Cluster ID; Cluster status, which is either idle or occupied; The file ID corresponding to the cluster data.

5. The system for storing PB-level data in DNA according to claim 4, characterized in that: When the cluster status is idle, the file ID corresponding to the cluster data is the default value -1.

6. The system for storing PB-level data in DNA according to any one of claims 1 to 3, characterized in that: The information in the file metadata in the system includes: Randomly generated file IDs are used to identify different files with the same name and create indexes; The properties of the file itself; The number of clusters occupied by the file; The number of the cluster that the file occupies.

7. A coding and decoding method running on a system suitable for PB-level data DNA storage as described in any one of claims 1 to 6, characterized in that: When a save operation is performed on a file with a preset suffix, the user-mode monitoring program receives the execution instruction and copies the file to the working directory, and encodes or decodes the file according to the file suffix. After the encoding or decoding operation is completed, the result is copied back to the original file directory.

8. The encoding and decoding method according to claim 7, wherein: include: For each file, when the kernel processes the first save operation, the data storage system creates two variables, namely the first variable and the second variable. The first variables and the second variables of different files are independent of each other and are stored in a common data structure. At the same time, two variables of type spinlock_t are created to protect the data from read and write conflicts in the case of multiple threads. In the initial state, both the first variable and the second variable are 0. Only when the first variable is judged to be 0 will the kernel send an instruction to the monitoring program, and after sending, the first variable is set to 1; The loop uses a spin lock to detect whether the second variable is 1. When the second variable is 1, the loop is jumped out and the whole process ends. When the external monitoring program receives the instruction from the kernel, it copies the file to the working directory. During this process, it enters the kernel again and detects that the first variable corresponding to the file is set to 1. It then sets the second variable to 1, clears the first variable, and returns. When the previous process in the loop detects that the second variable is 1, it jumps out of the loop, clears the second variable and returns. The task in the kernel in this encoding and decoding operation is completed, and the remaining encoding operations are implemented in the user layer. After completion, the encoded and decoded data is copied back to the original directory.

9. The encoding and decoding method according to claim 8, wherein: When an electronic file is input and the system receives the coding instruction, a file number is created to record the various attributes of the file; When the file is larger than a set value, the file is divided into pieces of fixed size; Based on the relevant information of the cluster metadata in the storage system, check the ID of the idle cluster, assign the ID of the idle cluster to each shard of the file, and update the relevant records in the cluster metadata; The file number and the occupied cluster number are recorded in the file metadata of the storage system.

10. The encoding and decoding method according to claim 8, wherein: When a specific electronic file needs to be restored from a base sequence, the cluster ID occupied by the file is first obtained from the file metadata in the file system, the DNA in the corresponding cluster is amplified and sequenced, and finally a decoding program is used to restore the base sequence to the original electronic file.

Images