A DNA data storage dynamic compression method based on classification algorithm

Abstract

The application discloses a DNA data storage dynamic compression method based on a classification algorithm, and comprises the following steps: 1) selecting a file to be stored; 2) selecting a classifier to classify the selected file; 3) using a compression algorithm on each file according to the classification result; 4) separately performing base conversion on the compressed file data; 5) saving the name of the file and the number of converted bases into a separate base sequence as a file directory; 6) splicing the file base sequence and the file directory information into a long base sequence; 7) dividing the spliced long base sequence into a plurality of equal-length sequences, adding an address code and an error correction code; and 8) when restoring the file, reading the file directory information, restoring as required, and obtaining an input file. The method improves the data compression rate in DNA data storage, and can select a compression algorithm according to the nature of the file, thereby achieving the purpose of improving the file data compression rate.

Description

TECHNICAL FIELD

[0001] The present application relates to a DNA data storage compression method, in particular to a DNA data storage dynamic compression method based on a classification algorithm. BACKGROUND

[0002] With the exponential growth of information data, the data storage medium relying on modern optical and magnetic medium cannot meet the demand of future big data storage. Therefore, it is necessary to use a storage technology with larger information storage density and lower cost.

[0003] Deoxyribonucleic acid (DNA) is a natural information storage medium, and its storage density is times that of traditional magnetic medium storage , and it can save information for hundreds of years without loss. The writing process of DNA storage maps digital data into DNA nucleotide sequences (nucleotides are the basic components of DNA), synthesizes the corresponding DNA molecules, and stores them. Modern biotechnology has developed basic tools that can manipulate DNA, including DNA synthesis and reading. The emergence of these technologies has provided support for the development of DNA data storage. The main problem of current DNA data storage is high cost. By 2020, the cost of synthesizing DNA is about 530 million US dollars per terabyte, and the reading cost is about 100,000 US dollars per terabyte. The cost of traditional magnetic medium storage is about 16 US dollars per terabyte. However, data centers using magnetic media need to spend millions of dollars in maintenance every year, while DNA storage only needs one thousandth of the cost. With the development of DNA synthesis and sequencing technology, the corresponding cost is also rapidly declining.

[0004] Due to the high cost of current DNA data storage, this technology can be used to store cold data and big data backup. Cold data is a part of data that is not frequently active and will not be accessed frequently or even forever, but still needs to be saved. According to statistics, about 80% of the data in the data center belongs to cold data. By using the high density and long storage time of DNA storage, these data can be stored, which has an advantage over traditional magnetic medium storage.

[0005] Before storing data, the data is usually compressed. The compressed data can effectively reduce the storage space occupied. Therefore, when using DNA storage technology to save data, the data needs to be compressed, which can reduce the number of bases synthesized and indirectly reduce the cost of synthesis.

[0006] Most of the current research work of scholars mainly focuses on DNA coding and synthesis technology. By optimizing or innovating the coding method and novel synthesis technology, the synthesis cost in DNA storage is reduced.

[0007] Purpose of the invention

[0008] Purpose of the invention: This invention provides a dynamic compression method for DNA data storage based on a classification algorithm.

[0009] Technical solution: The dynamic compression method for DNA data storage based on classification algorithms includes the following steps:

[0010] I. Select a files to be stored. You can choose any type and size of file (such as txt, mp4, etc., with no upper limit on file size) and the number of files is unlimited.

[0011] II. Select a machine learning classifier (any classifier will do) and classify the selected files according to their file type and size.

[0012] The training process includes the following steps:

[0013] 1) Obtain t files for training, containing one or more types (e.g., different file types such as txt, mp4, etc.), and files of any size.

[0014] 2) Compress the acquired files using d compression algorithms (such as BZIP2, DEFLATE, etc.) to compress each file individually.

[0015] 3) Compare the results of compressing each file using d compression algorithms, and for each file, obtain d compressed data using d compression algorithms. ,choose This yields the algorithm with the highest compression ratio.

[0016] 4) Save the file type, file size, and the compression algorithm with the highest compression ratio used as a vector (file type, file size, compression algorithm), where the file type is in string format and the file size is an integer range. The compression algorithm outputs the data in string format, which is then used as the dataset.

[0017] 5) Use conversion encoding (such as hot encoding, target encoding, etc.) to convert the file type from string to vector in numerical form.

[0018] 6) Use data standardization methods (such as z-score standardization, linear scaling, etc.) to convert the file size to a certain range. The data.

[0019] 7) Divide the dataset into two parts, a training set and a test set, according to the m:n ratio.

[0020] .

[0021] 8) Train the training set using machine learning classification algorithm, and verify the effect of the classifier using test data.

[0022] 9) Save the trained model (e.g. save as.pkl model file).

[0023] Its classification process includes the following steps:

[0024] 1) Save the file information to be classified into vector form (file type, file size), where the file type is in string form and the file size is in integer range .

[0025] 2) Use conversion coding (such as hot coding, target coding, etc.) to convert file type from string to numerical form vector.

[0026] 3) Use data standardization method (such as z-score standardization, linear scale standardization, etc.) to convert file size into a certain range of data.

[0027] 4) Call the trained classifier file (e.g. call.pkl file).

[0028] 5) Get the classification result of the file, and the classification result has s kinds, where , where d is the number of compression algorithms used.

[0029] III. For the classified files, according to the classification result Use the compression algorithm corresponding to the category for the files in each category, Use corresponding compression algorithm, where at most contains a file.

[0030] IV. Convert the compressed data of each file into base sequence, i.e. convert the binary sequence of the file into base sequence, get e base sequences, where .

[0031] V. Save the basic information (file name, file type) of a file and the converted base number into a separate base sequence, and the base number is the base sequence converted from the file data contains the number of bases, forms a file information directory, and adds a mark sequence before and after the file directory.

[0032] VI. Concatenate all file base sequences and file directory information into a long base sequence.

[0033] VII. According to the length of DNA synthesis requirements, the spliced long base sequence is divided into L pieces of length h sequences, and the address code is added at the beginning of each sequence The address code is arranged in ascending order, and the range is , the sequence The r-bit error correction code is added at the end.

[0034] VIII. When restoring data, according to the results of base sequencing, sort the sequences according to the address code of each base sequence, and process the possible errors of the sequences according to the error correction code in each sequence.

[0035] IX. After sorting, remove the address code and error correction code according to the sequence, and splice the L sequences into a long base sequence according to the address code order.

[0036] X. According to the set flag sequence, read the file directory part of the base sequence, restore according to the base number information in the directory as needed, and get the input file.

[0037] In steps II, III, IV, a data processing flow is constructed for classifying and compressing the files to be stored first. The data is converted into base sequence according to the base conversion rule of DNA storage after being compressed separately between files.

[0038] The compression algorithm used in step II includes common compression algorithms such as BZIP2, DEFLATE, PPMD, etc. The classification algorithm includes K-neighbors, decision tree, random forest, gradient boosting, adaptive enhancement, naive Bayes or support vector machine.

[0039] Further, in steps V and VI, the base sequence structure is:

[0040] The file directory is composed of information of a files, arranged in the order of file reading:

[0041] (File name 1, file 1 base number) + (file name 2, file 2 base number) + … + (file name a, file a base number)

[0042] The base sequences of each file are arranged in the order of the files in the directory after the file directory:

[0043] File directory base sequence + file 1 base sequence + file 2 base sequence + … + file a base sequence

[0044] Further, in step VIII, the addressing mode of on-demand reduction is: first calculate the total number of bases contained before the required reduction file, get the starting position of the required reduction file, combine the base number of the file in the file directory, and get the position of the file in the long base sequence, and read the part of the base sequence to complete the on-demand reduction. In view of the above problems, the compression ratio of the data to be stored is improved to reduce the synthesis cost of DNA storage. In the existing compression algorithm, various compression algorithms have their advantages, but there is no compression algorithm that can have the optimal compression effect on any file type. A machine learning classification algorithm is used to classify the files to be stored, and a relatively optimal compression algorithm is used for the classified files. The advantages of different compression algorithms are fully utilized to save the storage space. Compared with directly compressing all files, the method of first classifying the files and then compressing them will take more time, but it can improve the compression ratio of the file data and reduce the size of the compressed data. Because this method is used in DNA data storage, the time cost spent in compressing the data can be ignored compared with the time of DNA synthesis.

[0045] The method of the present application aims to solve the problem of low data compression rate in DNA data storage, which increases the cost of DNA data storage. The method uses a machine learning classification algorithm to classify the files to be stored, and uses different compression algorithms for the classified files according to the categories. The compression algorithm used in each category is the optimal algorithm in that category. The method of first classification and then compression can improve the data compression ratio and reduce the number of bases after conversion, thereby reducing the synthesis cost in DNA data storage.

[0046] Advantages: Compared with the prior art, the present application has the following advantages:

[0047] 1. The method classifies the data files to be compressed according to the type and size characteristics of the files, and then compresses the classified files using a compression algorithm. This method can maximize the advantages of different compression algorithms, improve the data compression rate, and effectively reduce the cost of DNA data storage.

[0048] 2. When compressing data, each file is compressed separately, which can restore the file on demand and reduce the data restoration time. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 It is a method flow diagram;

[0050] Figure 2 It is a flowchart of the classification algorithm;

[0051] Figure 3 It is a base sequence splicing diagram. DETAILED DESCRIPTION

[0052] 5000 files are selected as the dataset for training the classifier, which are randomly downloaded from the internet (arbitrary size and type), and contain 10 file types, 500 files for each type. In addition, 100 files are additionally selected as test data, which contain 10 different file types.

[0053] I. 100 files containing 10 file types are stored as the files needed to be stored this time.

[0054] II. Train the machine learning classifier (arbitrary classifier) to classify the selected files according to file type and size, using K-Nearest Neighbor algorithm as an example.

[0055] The training process includes the following steps:

[0056] 1) 5000 files containing 10 file types are used as training files.

[0057] 2) The above five compression algorithms (BZIP2, DEFLATE, PPMD, BROTLI, LZMA2) are used to compress the 5000 file dataset.

[0058] 3) Compare the compression results of each file to get the highest compression ratio result.

[0059] 4) Save the file type, file size and the highest compression algorithm used as a vector (file type, file size, compression algorithm) as a dataset, and get a record containing 5000 data (file type, file size, compression algorithm). The file size is the size before compression, and the compression algorithm is the highest compression algorithm after using the five compression algorithms respectively.

[0060] 5) Convert the file type from string to numerical vector using hot encoding. Convert the file type in the obtained 5000 data to the following results:

[0061] Txt: (1, 0, ⋯, 0, 0)

[0062] Jpg: (0, 1, ⋯, 0, 0)

[0063] …

[0064] Mp3: (0, 0, ⋯, 1, 0)

[0065] Where txt, jpg, etc. are file types, and only part of the conversion results are shown.

[0066] A 11-dimensional vector is obtained, in which the first 10 dimensions represent one of the 10 file types, and the 11th dimension represents an unknown file type.

[0067] 6) The file size of the obtained 5000 data is standardized, and the data standardization calculation formula is:

[0068] wherein is the converted sample data value, is the sample data mean, is the standard deviation of all sample data.

[0069] After steps 5) and 6), the data is a 13-dimensional feature vector, in which the first 11 dimensions represent the type of file, the 12th dimension represents the size of the file, and the 13th dimension is the compression algorithm used, that is, the classification label result.

[0070] 7) The data set is divided into two parts according to 7:3, in which 3500 data are used as training data, and 1500 data are used as test data.

[0071] 8) The KNN classification algorithm is used to train the training set, and the test data is used to verify the effect of the classifier, and the classification accuracy obtained is 87%.

[0072] The classification process includes the following steps:

[0073] 1) Save the information of 100 files to be classified into a vector form (file type, file size).

[0074] 2) Convert the file type from a string to a numerical form vector using hot encoding.

[0075] 3) Convert the file size to data with a mean of 0 and a standard deviation of 1 using data standardization.

[0076] 4) Calculate the distance between the data and each data in the training set. The distance calculation formula is:

[0077]

[0078] 5) Find the 15 points closest to the data, and count the frequency of the categories to which the 15 data belong.

[0079] 6) Classify the file data into the category with the highest frequency.

[0080] III. For the classified files, use the optimal compression algorithm for each type of file according to the classification result.

[0081] IV. Each compressed file data is individually converted into base sequence, i.e. the binary sequence of the file is converted into base sequence. The conversion rule is ternary conversion rule, and the rule is as follows:

[0082] Table 1 Conversion rule

[0083]

[0084] V. The name of 100 files and the number of encoded bases are encoded, merged into a file directory, and AAAA is added before and after the file directory as a mark sequence.

[0085] VI. The base sequence of 100 files and the file directory information are spliced into a long base sequence.

[0086] VII. According to the requirement of DNA synthesis length, the spliced long base sequence is divided into several sequences with a length of 150 bp, and 20-bit address code and 10-bit RS error correction code are added.

[0087] VIII. When data is restored, according to the result of base sequencing, the sequence is sorted according to the address code of each base sequence, and the possible errors of the sequence are processed according to the error correction code in each sequence.

[0088] IX. After the sorted sequence, the address code and error correction code are removed according to the order to splice back into a long base sequence.

[0089] X. According to the set mark sequence, read the file directory part of the base sequence, restore according to the file information in the directory as needed, and get the input file.

[0090] Table 2 Comparison table of compression ratio and sequence number

[0091] Classification method Compression ratio Number of sequences Compression method used herein 16.9% 5,630,000 Traditional compression method 14.8% 5,750,000

Claims

1. A DNA data storage dynamic compression method based on classification algorithm, characterized in that: It comprises the following steps: I. Select a file to be stored, select any type and size of file as needed, the number is unlimited; II. Select the machine learning classifier, and classify the selected files according to the file type and size; Its training process includes the following steps: 1) Obtain t file data for training, including one or more types of files of any size; 2) Compress the obtained files, and use d compression algorithms to individually compress each file; 3) comparing the results of compressing each file with d compression algorithms, obtaining d compressed data for each file using d compression algorithms , selecting , obtaining the algorithm with the highest compression ratio; 4) Save the file type, file size, and the compression algorithm with the highest compression ratio used as a vector, where the file type is a string and the file size is an integer range. The compression algorithm outputs the data in string format, which is then used as the dataset. 5) Convert the file type from a string to a numerical form vector using conversion encoding; 6) Convert file size into a range of data using data standardization methods; 7) Divide the data set into training set and test set in m:n, wherein, ， 8) Train the training set using machine learning classification algorithm, and verify the effect of the classifier using test data; 9) Save the trained model, Its classification process includes the following steps: 1) Save the file information to be classified into a vector form containing file type, file size, where the file type is in the form of a string, and the file size is in the form of an integer range , 2) Convert the file type from a string to a numerical form vector using conversion encoding; 3) using data standardization, converting file size into a certain range of data ; 4) Call the trained classifier file; 5) get the classification result of the file, the classification result has s kinds in total, wherein wherein d is the kind of compression algorithm used, III. For the classified files, the classification results are applied For the files in each category, the compression algorithm corresponding to the category is applied, using the corresponding compression algorithm, wherein at most a files are included, IV. Compressing each file The data is individually base sequence converted, i.e. the binary sequence of the file is converted into a base sequence, resulting in e base sequences, wherein , V. The basic information of a file and the converted base number are saved as a single base sequence, and the base number is The base sequence converted from the file data The number of bases contained, forming a file information directory, and adding a mark sequence before and after the file directory, VI. Concatenate all file base sequences and file directory information into a long base sequence, VII. According to the requirement of DNA synthesis length, the spliced long base sequence is divided into L sequences with length h, and the sequence The address code is added in front, and the address code is arranged in ascending order, with a range of The sequence r-bit error correction code is added at the end. VIII. When restoring data, sort the sequences according to the address code of each base sequence according to the base sequencing result, and process the possible errors of the sequences according to the error correction code in each sequence; IX. After sorting, remove the address code and error correction code according to the sequence, and concatenate the L sequences into a long base sequence according to the address code sequence; X. According to the set flag sequence, read the file directory part of the base sequence, restore according to the base number information in the directory as needed, and obtain the input file.

2. The DNA data storage dynamic compression method based on classification algorithm according to claim 1, characterized in that, In steps II, III and IV, a data processing flow is constructed for classifying and then compressing the files to be stored. After individual compression of the files, the data is converted into base sequences according to the base conversion rules of DNA storage.

3. The DNA data storage dynamic compression method based on classification algorithm according to claim 1, characterized in that, In step II, the compression algorithms used include BZIP2, DEFLATE, and PPMD. The classification algorithms used include K-nearest neighbors, decision trees, random forests, gradient boosting, adaptive boosting, naive Bayes, or support vector machines.

4. The DNA data storage dynamic compression method based on classification algorithm according to claim 1, characterized in that, In steps V and VI, the base sequence structure is: The file directory is composed of information of a files, arranged in the order of file reading: (file name 1, file 1 base number) + (file name 2, file 2 base number) + … + (file name a, file a base number), After the file directory, arrange the base sequences of each file in the order of the directory: file directory base sequence + file 1 base sequence + file 2 base sequence + … + file a base sequence.

5. The DNA data storage dynamic compression method based on classification algorithm according to claim 1, characterized in that, In step VIII, the addressing method for restoration on demand is as follows: first, calculate the total number of bases contained before the file to be restored, get the starting position of the file to be restored, combine the file directory base number of the file, and get the position of the file in the long base sequence. Reading this part of the base sequence can complete the restoration on demand.

Images