Solid nanopore electric signal data compression method and device, and storage medium

Abstract

The invention discloses a solid-state nanopore electric signal data compression method and device and a storage medium, and the method comprises the steps: normalizing obtained N pieces of solid-state nanopore electric signal data, and obtaining N pieces of normalized data; the indication information associated with the address of a first data unit at a preset position in the N pieces of normalized data is used for replacing a second data unit, which is the same as the first data unit, in the N pieces of normalized data to obtain first compressed data, the first data unit comprises M pieces of normalized data, and the M pieces of normalized data comprise M pieces of normalized data; the indication information associated with the address is used for indicating the position of the first data unit in the N pieces of normalized data.

Description

Technical Field

[0001] The present invention relates to the fields of signal processing and data compression, and in particular to a solid-state nanopore electrical signal data compression method, device, and computer-readable storage medium. Background Art

[0002] Nanopore single-molecule detection technology is to fill both sides of the insulating film with holes with ion solution and apply voltage. When single molecules such as DNA or protein in the ion solution pass through the nanopore, the nanopore will be temporarily blocked, causing the current value in the nanopore to fluctuate. At the same time, because biological molecules pass through the nanopore very quickly, the equipment used to capture current changes usually uses a high sampling rate, which will generate a large amount of electrical signal data, posing a great challenge to data transmission and analysis, so it is necessary to compress the original data.

[0003] Therefore, it is necessary to provide a solid-state nanopore electrical signal data compression method to solve the problem in the prior art that the amount of solid-state nanopore electrical signal data is large and the data transmission and analysis are difficult. Summary of the invention

[0004] In view of this, the main purpose of the present invention is to provide a method, device and storage medium for solid-state nanopore electrical signal data compression.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a solid-state nanopore electrical signal data compression method is provided, the method comprising:

[0007] Normalizing the acquired N solid-state nanopore electrical signal data to obtain N normalized data;

[0008] The address-associated indication information of the first data unit at a predetermined position in the N normalized data is used to replace the second data unit in the N normalized data that is identical to the first data unit, so as to obtain first compressed data, wherein the first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data.

[0009] In some embodiments, the normalizing the acquired N solid-state nanopore electrical signal data to obtain N normalized data includes:

[0010] Numbering the N solid-state nanopore electrical signal data, wherein the same solid-state nanopore electrical signal data uses the same number, and the number corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data, wherein the number occupies less bits than the solid-state nanopore electrical signal data.

[0011] In some embodiments, the normalizing the acquired N solid-state nanopore electrical signal data to obtain N normalized data includes:

[0012] According to the hash table, determine the hash values ​​corresponding to the N solid-state nanopore electrical signal data, wherein the same solid-state nanopore electrical signal data uses the same hash value, and use the hash value corresponding to the solid-state nanopore electrical signal data as the normalized data of the solid-state nanopore electrical signal data, wherein the bits occupied by the hash value are smaller than the bits occupied by the solid-state nanopore electrical signal data.

[0013] In some embodiments, determining the hash values ​​corresponding to the N pieces of solid-state nanopore electrical signal data respectively according to the hash table includes:

[0014] Determine, according to the hash table, a hash value corresponding to i bits of data at a predetermined position of each solid-state nanopore electrical signal data;

[0015] The hash value corresponding to i bits of data at a predetermined position of the solid-state nanopore electrical signal data is used as the hash value corresponding to the solid-state nanopore electrical signal data; wherein the solid-state nanopore electrical signal data is I bits of data, I>i.

[0016] In some embodiments, the associating the address of the first data unit at a predetermined position in the N normalized data with the indication information to replace the second data unit in the N normalized data that is identical to the first data unit to obtain the first compressed data includes:

[0017] Determine a first data unit in the N normalized data that is the same as the second data unit;

[0018] Determining, according to the first data unit, indication information associated with an address of the first data unit;

[0019] The first compressed data is obtained by replacing the second data unit with indication information associated with the address of the first data unit;

[0020] The first data unit includes M normalized data, M is greater than or equal to 3 and less than N, and the length of the address-associated indication information is less than or equal to the length of 3 normalized data.

[0021] In some embodiments, the method further comprises:

[0022] determining the number of repetitions of different compressed data units in the first compressed data;

[0023] Sort the different compressed data units based on the number of repetitions of the different compressed data units to obtain a first data sequence;

[0024] Each compressed data unit in the first data sequence is encoded according to a preset rule to obtain second compressed data.

[0025] In some embodiments, the ordering of the different compressed data units based on the number of repetitions of the different compressed data units to obtain the first data sequence includes:

[0026] According to the number of repetitions of the different compressed data units, they are arranged in descending order to obtain a first data sequence.

[0027] In some embodiments, the preset rules include:

[0028] When the code length of the first compressed data unit in the first compressed data is the same as the code length of the second compressed data unit, the sum of the code of the second compressed data unit plus the first predetermined value is determined as the code of the first compressed data unit;

[0029] When the coding length of the first compressed data unit in the first compressed data is greater than the coding length of the second compressed data unit, the code of the second compressed data unit is added to the sum of the first predetermined value to obtain the first code, and the second predetermined value is added after the first code to determine it as the code of the first compressed data unit.

[0030] According to a second aspect of an embodiment of the present disclosure, a solid-state nanopore electrical signal data compression device is provided, the device comprising:

[0031] A first processing module is used to normalize the acquired N solid-state nanopore electrical signal data to obtain N normalized data corresponding to the N solid-state nanopore electrical signal data;

[0032] The second processing module is used to replace the second data unit in the N normalized data that is identical to the first data unit with the indication information associated with the address of the first data unit at a predetermined position in the N normalized data, so as to obtain the first compressed data, wherein the first data unit includes M normalized data, and the indication information associated with the address is used to indicate the position of the first data unit in the N normalized data.

[0033] According to a third aspect of an embodiment of the present disclosure, a solid-state nanopore electrical signal data compression device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.

[0034] According to the fourth aspect of the embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented. A solid-state nanopore electrical signal data compression method, device and storage medium provided in the embodiment of the present invention, the method comprises: normalizing the acquired N solid-state nanopore electrical signal data to obtain N normalized data; replacing the second data unit identical to the first data unit in the N normalized data with the address-associated indication information of the first data unit at a predetermined position in the N normalized data to obtain the first compressed data, wherein the first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data; in this way, by normalizing the solid-state nanopore electrical signal data and replacing the repeated data units, the data compression of the solid-state nanopore electrical signal data is realized, the storage space occupied by the solid-state nanopore electrical signal data is reduced, and the search time is greatly reduced, thereby ensuring that the massive data can be transmitted in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic flow chart of a solid-state nanopore electrical signal data compression method provided by an embodiment of the present invention;

[0036] Figure 2 A schematic flow chart of another solid-state nanopore electrical signal data compression method provided by an embodiment of the present invention;

[0037] Figure 3 A schematic structural diagram of a solid-state nanopore electrical signal data compression device provided by an embodiment of the present invention;

[0038] Figure 4 A schematic structural diagram of another solid-state nanopore electrical signal data compression device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described in detail below in conjunction with the embodiments.

[0040] Before further describing the present invention in detail in conjunction with the embodiments, the related technology of data compression is described first.

[0041] The existing data compression method is to perform linear scanning on the data, search in the scanned data, determine whether the current data has appeared before, and if it has appeared, directly compress it by position index. This search method is extremely inefficient and ignores the fact that the solid-state nanopore electrical signal data has a certain temporal regularity, that is, there are some scattered repeated data in the solid-state nanopore electrical signal data.

[0042] On the other hand, the existing data compression method uses Huffman coding for compression, and the Huffman coding table needs to be transmitted together with the compressed data, which introduces new transmission overhead.

[0043] Therefore, it is necessary to provide a solid-state nanopore electrical signal data compression method to solve the current problem that the amount of solid-state nanopore electrical signal data is large and the data transmission and analysis are difficult.

[0044] Figure 1 A schematic flow chart of a solid-state nanopore electrical signal data compression method provided by an embodiment of the present invention; Figure 1 As shown, the method includes:

[0045] Step 101, normalizing the acquired N solid-state nanopore electrical signal data to obtain N normalized data;

[0046] Here, both sides of the insulating film with holes are filled with ion solution and voltage is applied. When a single molecule in the ion solution passes through the nanopore, the nanopore is temporarily blocked and the current value in the nanopore fluctuates. The electrical signal data generated when the device captures the current change is solid-state nanopore electrical signal data.

[0047] Specifically, normalization is to convert data into numerical values ​​within a certain range, for example, to convert solid-state nanopore electrical signal data into numbers, wherein the number occupies less bits than the solid-state nanopore electrical signal data.

[0048] Step 102: Replace a second data unit in the N normalized data that is identical to the first data unit with the address-associated indication information of the first data unit at a predetermined position in the N normalized data to obtain first compressed data, wherein the first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data.

[0049] Specifically, the predetermined position may be the position of the first data unit identical to the second data unit in the N normalized data; the indication information associated with the address of the first data unit may include at least one of the following: the address of the first data unit; identification information that uniquely identifies the address of the first data unit. For example, the indication information associated with the address of the first data unit may be the address of the first normalized data in the first data unit and the length of the first data unit.

[0050] Here, N normalized data may constitute a normalized data sequence, and the first data unit may be M adjacent normalized data at a predetermined position in the normalized data sequence. For example, the first data unit may be the first M adjacent normalized data in the normalized data sequence. The second data unit may be a data unit after the first data unit. The second data unit may include M adjacent normalized data, so that the normalized data sequence may include the first data unit and multiple second data units, and the multiple second data units may be different or the same.

[0051] If the second data unit is the same as the first data unit, the second data unit is replaced with the indication information associated with the address of the first data unit, so that the bits of the second data unit can be reduced, achieving the effect of compressing data.

[0052] Exemplarily, the normalized data of the solid-state nanopore electrical signal data may be a hash value corresponding to the solid-state nanopore electrical signal data. In this way, N normalized data may constitute a hash table, wherein the hash table contains N hash values. M hash values ​​constitute a first data unit. For example, the first M hash values ​​in the hash table may be used as the first data unit, and the remaining hash values ​​in the hash table may be composed of one or more second data units. The second data unit may contain M adjacent hash values. If the second data unit is the same as the first data unit, the indication information associated with the address of the first data unit is used to replace the second data unit. In this way, the bits of the second data unit may be reduced, thereby achieving the effect of compressing the data.

[0053] In actual application, when decompressing the first compressed data, when scanning the data unit represented by the indication information associated with the address of the first data unit, the corresponding data unit can be found at the corresponding position of the decompressed data for decompression.

[0054] In this way, in view of the fact that the solid-state nanopore electrical signal data has many repeated data units, the compression rate is improved by compressing the repeated data units.

[0055] In some embodiments, the normalizing the acquired N solid-state nanopore electrical signal data to obtain N normalized data includes:

[0056] Numbering the N solid-state nanopore electrical signal data, wherein the same solid-state nanopore electrical signal data uses the same number, and the number corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data, wherein the number occupies less bits than the solid-state nanopore electrical signal data.

[0057] In a possible implementation, each solid-state nanopore electrical signal data occupies 16 bits, and each number occupies 8 bits.

[0058] Specifically, the initial number can be set to 0, the solid-state nanopore electrical signal data can be scanned, and N solid-state nanopore electrical signal data can be numbered in turn to determine whether the scanned solid-state nanopore electrical signal data is the same as the numbered solid-state nanopore electrical signal data; if not, the current number is used to represent the solid-state nanopore electrical signal data, and the number is increased by 1; if the same, the known solid-state nanopore electrical signal data is represented by the same number as the numbered solid-state nanopore electrical signal data, and the number corresponding to each solid-state nanopore electrical signal data is used as the normalized data.

[0059] In some embodiments, the normalizing the acquired N solid-state nanopore electrical signal data to obtain N normalized data includes:

[0060] According to the hash table, determine the hash values ​​corresponding to the N solid-state nanopore electrical signal data, wherein the same solid-state nanopore electrical signal data uses the same hash value, and use the hash value corresponding to the solid-state nanopore electrical signal data as the normalized data of the solid-state nanopore electrical signal data, wherein the bits occupied by the hash value are smaller than the bits occupied by the solid-state nanopore electrical signal data.

[0061] In a possible implementation, each solid-state nanopore electrical signal data occupies 16 bits, and each hash value occupies 8 bits.

[0062] Here, a hash table may be pre-established, and the hash table includes different hash values ​​corresponding to different solid-state nanopore electrical signal data, wherein the hash value may be a serial number.

[0063] Specifically, the solid-state nanopore electrical signal data is scanned, and the hash value corresponding to each solid-state nanopore electrical signal data is determined according to the hash table, and it is judged whether the scanned solid-state nanopore electrical signal data is the same as the scanned solid-state nanopore electrical signal data. If they are different, the hash value corresponding to the solid-state nanopore electrical signal data is obtained; if they are the same, the hash value of the scanned solid-state nanopore electrical signal data is used as the hash value corresponding to the solid-state nanopore electrical signal data, and the hash value corresponding to each solid-state nanopore electrical signal data is used as the normalized data.

[0064] In some embodiments, determining the hash values ​​corresponding to the N pieces of solid-state nanopore electrical signal data respectively according to the hash table includes:

[0065] Determine, according to the hash table, a hash value corresponding to i bits of data at a predetermined position of each solid-state nanopore electrical signal data;

[0066] The hash value corresponding to i bits of data at a predetermined position of the solid-state nanopore electrical signal data is used as the hash value corresponding to the solid-state nanopore electrical signal data; wherein the solid-state nanopore electrical signal data is I bits of data, I>i.

[0067] Here, based on the characteristic that the i-bit data of most solid-state nanopore electrical signal data are the same, the i-bit data of the predetermined position of each solid-state nanopore electrical signal data is taken, and a hash table can be pre-established. The hash table includes different hash values ​​corresponding to the i-bit data of the predetermined position of different solid-state nanopore electrical signal data, and the hash value corresponding to the i-bit data of the predetermined position of the solid-state nanopore electrical signal data is used as the hash value corresponding to the solid-state nanopore electrical signal data.

[0068] Specifically, the solid-state nanopore electrical signal data is 16-bit data. Since the first 8 bits of most solid-state nanopore electrical signal data are the same, the last 8 bits of each solid-state nanopore electrical signal data can be taken to determine the different hash values ​​corresponding to the last 8 bits of different solid-state nanopore electrical signal data; the solid-state nanopore electrical signal data is scanned to determine whether the scanned solid-state nanopore electrical signal data is the same as the scanned solid-state nanopore electrical signal data. If they are not the same, the hash value corresponding to the last 8 bits of the solid-state nanopore electrical signal data is used as the hash value corresponding to the solid-state nanopore electrical signal data; if they are the same, the hash value corresponding to the scanned solid-state nanopore electrical signal data is used as the hash value corresponding to the solid-state nanopore electrical signal data.

[0069] In this way, by utilizing the fact that the solid-state nanopore electrical signal data has a lot of repetitive data, the original solid-state nanopore electrical signal data is represented by numbers, thereby realizing the conversion of two-byte data into one-byte data.

[0070] In practical applications, among N solid-state nanopore electrical signal data, the number of different solid-state nanopore electrical signal data is greater than X. Then the N solid-state nanopore electrical signal data can be divided into Y data blocks, wherein each data block contains less than or equal to X different solid-state nanopore electrical signal data. Here, X is a positive integer. For example, X can be 256. Y is a positive integer greater than or equal to 2. In this way, it can be ensured that the normalized data corresponding to each solid-state nanopore electrical signal data only occupies one byte.

[0071] In some embodiments, the associating the address of the first data unit at a predetermined position in the N normalized data with the indication information to replace the second data unit in the N normalized data that is identical to the first data unit to obtain the first compressed data includes:

[0072] Determine a first data unit in the N normalized data that is the same as the second data unit;

[0073] Determining, according to the first data unit, indication information associated with an address of the first data unit;

[0074] The first compressed data is obtained by replacing the second data unit with indication information associated with the address of the first data unit;

[0075] The first data unit includes M normalized data, M is greater than or equal to 3 and less than N, and the length of the address-associated indication information is less than or equal to the length of 3 normalized data.

[0076] In one possible implementation, the first data unit includes M normalized data, M is greater than or equal to 3 and less than N, that is, the length of the second data unit and the first data unit is greater than or equal to the length of 3 normalized data. At the same time, the length of the indication information associated with the address of the first data unit is usually less than or equal to the length of 3 normalized data. In this way, by replacing the second data unit that is identical to the first data unit with the indication information associated with the address of the first data unit, the problem of increasing storage due to the second data unit being too short being replaced by the indication information associated with the address of the first data unit can be avoided.

[0077] Specifically, assuming that the indication information associated with the address of the first data unit is the address of the first normalized data in the first data unit and the length of the first data unit, when a first data unit identical to the second data unit is found among N normalized data, the address of the first normalized data in the first data unit and the length of the first data unit are determined, and the address of the first normalized data in the first data unit and the length of the first data unit are used to replace the second data unit to obtain first compressed data.

[0078] In this way, by stipulating that the same data unit is greater than or equal to three bytes, it is possible to avoid the problem that the second data unit is replaced by the indication information associated with the address of the first data unit, which increases the storage due to the second data unit being too short.

[0079] In some embodiments, the method further comprises:

[0080] determining the number of repetitions of different compressed data units in the first compressed data;

[0081] Sort the different compressed data units based on the number of repetitions of the different compressed data units to obtain a first data sequence;

[0082] Each compressed data unit in the first data sequence is encoded according to a preset rule to obtain second compressed data.

[0083] In a possible implementation, the encoding lengths of different compressed data units are obtained according to the number of repetitions of different compressed data units in the first compressed data.

[0084] Specifically, according to the number of repetitions of different compressed data units in the first compressed data, the depth of each different compressed data unit in the traditional Huffman coding tree is calculated, thereby obtaining the coding length of the different compressed data units.

[0085] In some embodiments, the ordering of the different compressed data units based on the number of repetitions of the different compressed data units to obtain the first data sequence includes:

[0086] According to the number of repetitions of the different compressed data units, they are arranged in descending order to obtain a first data sequence.

[0087] Here, the more times the compressed data unit is repeated, the shorter the corresponding code length is.

[0088] Specifically, the different compressed data units are arranged in descending order according to the number of repetitions of the different compressed data units, and the corresponding encoding lengths of the different compressed data units are also arranged in descending order.

[0089] In some embodiments, the preset rules include:

[0090] When the code length of the first compressed data unit in the first compressed data is the same as the code length of the second compressed data unit, the sum of the code of the second compressed data unit plus the first predetermined value is determined as the code of the first compressed data unit;

[0091] When the coding length of the first compressed data unit in the first compressed data is greater than the coding length of the second compressed data unit, the code of the second compressed data unit is added to the sum of the first predetermined value to obtain the first code, and the second predetermined value is added after the first code to determine it as the code of the first compressed data unit.

[0092] Here, the first compressed data unit and the second compressed data unit are compressed data units arranged according to the number of repetitions in the first data sequence, wherein the number of repetitions of the first compressed data unit is greater than or equal to the number of repetitions of the second compressed data unit, that is, the encoding length of the first compressed data unit is less than or equal to the encoding length of the second compressed data unit.

[0093] Here, the first predetermined value may be 1, and the second predetermined value may be 0.

[0094] Specifically, the preset rule may be that, during the encoding process, if the encoding lengths of the first compressed data unit and the second compressed data unit are the same, then the encoding of the second compressed data unit is the sum of the encoding of the first compressed data unit plus 1; if the encoding length of the second compressed data unit is greater than the encoding length of the first compressed data unit, then the encoding of the second compressed data unit is the sum of the encoding of the first compressed data unit plus 1, and 0 is added after the encoding.

[0095] Illustratively, Table 1 is a schematic table of compressed data unit encoding lengths provided in an embodiment of the present invention.

[0096] Compressed Data Unit a b c d e f g h i j k l m n Code length 3 3 4 4 4 5 5 5 6 6 7 7 7 7

[0097] Table 1

[0098] Specifically, different compressed data units include a, b, c, d, e, f, g, h, i, j, k, l, m, n, and the encoding length of a is 3, the encoding length of b is 3, the encoding length of c is 4, the encoding length of d is 4, the encoding length of e is 4, the encoding length of f is 5, the encoding length of g is 5, the encoding length of h is 5, the encoding length of i is 6, the encoding length of j is 6, the encoding length of k is 7, the encoding length of l is 7, the encoding length of m is 7, and the encoding length of n is 7.

[0099] Illustratively, Table 2 is a schematic table of the number of coding lengths provided in an embodiment of the present invention.

[0100] Code length 1 2 3 4 5 6 7 Number 0 0 2 3 3 2 4

[0101] Table 2

[0102] Specifically, the number of different compressed data units with a code length of 1 is 0, the number of different compressed data units with a code length of 2 is 0, the number of different compressed data units with a code length of 3 is 2, the number of different compressed data units with a code length of 4 is 3, the number of different compressed data units with a code length of 5 is 3, the number of different compressed data units with a code length of 6 is 2, and the number of different compressed data units with a code length of 7 is 4. The number of different compressed data units with a code length of i can be represented by count[i], then count[1]=0, count[2]=0, count[3]=2, count[4]=3, count[5]=3, count[6]=2, count[7]=4.

[0103] Illustratively, Table 3 is a schematic table of compressed data unit encoding provided by an embodiment of the present invention.

[0104] Compressed Data Unit Code length coding a 3 000 b 3 001 c 4 0100 d 4 0101 e 4 0110 f 5 01110 g 5 01111 h 5 10000 i 6 100010 j 6 100011 k 7 1001000 l 7 1001001 m 7 1001010 n 7 1001011

[0105] Table 3

[0106] Specifically, let the code of the first compressed data unit a be 000; according to the coding rule, the code of the compressed data unit b is a code plus 1, that is, 001; the code of the compressed data unit c is b code plus 1, and then add a 0 at the end, that is, 0100; the code of the compressed data unit d is c code plus 1, that is, 0101; the code of the compressed data unit e is d code plus 1, that is, 0110; the code of the compressed data unit f is e code plus 1, and then add a 0 at the end, that is, 01110; the code of the compressed data unit g is f code plus 1, that is, 01111; The encoding of unit h is g code plus 1, that is, 10000; the encoding of compressed data unit i is h code plus 1, followed by a 0, that is, 100010; the encoding of compressed data unit j is i code plus 1, that is, 100011; the encoding of compressed data unit k is j code plus 1, followed by a 0, that is, 1001000; the encoding of compressed data unit l is k code plus 1, that is, 1001001; the encoding of compressed data unit m is l code plus 1, that is, 1001010; the encoding of compressed data unit n is m code plus 1, that is, 1001011.

[0107] In actual application, when decompressing compressed data, it is only necessary to save the compressed data unit table and the code length number table in the compressed information header, and there is no need to attach the coding information to the compressed information header. During the decompression process, the decoding information is constructed according to the preset rules based on the compressed data unit table and the code length number table for decoding.

[0108] Exemplarily, the compressed data unit table can be [a, b, c, d, e, f, g, h, i, j, k, l, m, n], and the coding length number table can be [0, 0, 2, 3, 3, 2, 4].

[0109] In this way, by improving Huffman coding, it is no longer necessary to transmit the entire coding table during data transmission. Instead, only the coding length of each different compressed data unit needs to be known for encoding and decoding, thereby reducing the storage amount of header information generated by compression and improving the compression rate.

[0110] The following provides multiple specific examples in combination with any of the above embodiments:

[0111] The steps of the solid-state nanopore electrical signal data compression method are as follows: Figure 2As shown in the figure, it mainly includes the following three steps: (1) In view of the fact that solid-state pore electrical signal data (i.e., solid-state nanopore electrical signal data) has a lot of repeated data, the hash algorithm is used for normalization processing to greatly improve the compression ratio, and the original 16-bit data is compressed into 8 bits (i.e., normalized data); (2) In view of the fact that solid-state pore electrical signal data (i.e., solid-state nanopore electrical signal data) has a lot of repeated fragments, weak regularity data is used to compress the repeated fragments, and the normalized 8-bit data is converted into data with three bytes as a unit; (3) From the byte level, based on replacing the traditional Huffman algorithm with the paradigm-improved Huffman algorithm, the storage amount of header information generated by compression is reduced, thereby improving the compression rate.

[0112] 1. Data normalization:

[0113] Through statistical analysis of multiple groups of sequencing data obtained under different molecular sizes and nanopore sizes, it is found that although the amount of gene sequencing data is large, the number of different values ​​is very limited. Therefore, the gene sequencing data can be normalized. The purpose of the normalization operation is to limit the value of the nanopore electrical signal data (i.e., solid-state nanopore electrical signal data) to a range of 256 different values, so that each value can be represented by only one byte, which is convenient for subsequent processing operations. Due to the data range limitation, it is necessary to re-iterate compression every 256 different values. This range is the optimal range obtained by analyzing the nanopore electrical signal data.

[0114] The specific method of data normalization is to set the initial representation id to 0, scan the solid-state nanopore data (i.e., solid-state nanopore electrical signal data), and as long as an unknown value appears, it is represented by id, and the id is increased by 1. In this way, after scanning a section of sequencing data, the original 2-byte value can be represented by a shorter 1-byte value (i.e., normalized data). Determine whether the scanned value is an unknown value. Construct a hash table with a length of 2 to the 8th power, i.e., 256, traverse all data points (i.e., solid-state nanopore electrical signal data), take the last eight bits of the data point, if the last eight bits of the data point have not appeared, put the value of the data point into the hash table relative to its last eight bits; if the last eight bits of the data point have appeared and the value of the data point is different from the previous value, then find the nearest free position behind the hash table position corresponding to the last eight bits of the data point and put it in. In general, the first eight bits of most data points are the same, so constructing a hash table with a length of 256 is sufficient to put the values ​​of all data points. If one hash table is not big enough to hold the data, the processed data block will be further compressed after the first hash table is full, and then a new hash table will be rebuilt to process the new data block. The hash table method is used to speed up the judgment and greatly reduce the time used to compress the data. After this compression, the data is converted from 16 bits to 8 bits.

[0115] 2. Weak regularity data compression:

[0116] The traditional compression method is to perform a linear scan on the data (i.e., solid-state nanopore electrical signal data) to determine whether the current data has appeared in the scanned data. If it has appeared, it can be compressed directly by position indexing. However, this search method is extremely inefficient and ignores the characteristics of nanopore current data (i.e., solid-state nanopore electrical signal data), that is, it has a certain temporal regularity and there are some scattered repeated data in the data.

[0117] In the application of nanopore electrical signal data transmission, the data processing delay constraint is extremely strict. Therefore, the repeated structure information of the data is used to compress the data. The method of exchanging space for time is used to greatly reduce the search time, thereby ensuring that massive data can be transmitted in a timely manner.

[0118] First, define the length of string S (i.e., solid-state nanopore electrical signal data) as N, and the substring S(i,j) of string S, 1≤i,j≤N. For the prefix substring S(1,j), L(i,j) is the first character S i The substring (i.e., the first data unit) and the first character S j The maximum matching length of the substring (i.e., the second data unit) is:

[0119] L(i,j)=max{l|S(i,i+l-1)=S(j,j+l-1)}, where l≤Nj

[0120] It is said that the string S(j,j+l-1) (ie, the second data unit) matches the string S(i,i+l-1) (ie, the first data unit), and the matching length is l.

[0121] Define p(j) as the longest matching i value in all cases, that is,

[0122] p(j)=argmax i {L(i,j)}, where 1≤i≤j

[0123] In the specific implementation, a sliding window dictionary is used to store historical characters, and p and L values ​​are found in the sliding window, and (p, L) is used to represent the current data segment (i.e., the second data unit). The decompression process is the opposite of the compression process. When encountering a segment of data represented by (p, L), a segment of data with a length of L is found at the corresponding position p of the historical data for decompression.

[0124] In order to avoid the problem that the repeated data fragment is too short and the address is used to replace the data, the normalized 8-bit data is converted into data with three bytes as a unit, stored with integer data, and a hash table is constructed to find whether the three-byte data has appeared before. If the search distance exceeds the sliding window size, the search is stopped. After the repeated data point is found, the subsequent data points are compared to see if they are continuous and the comparison is stopped until the compared data is different or the continuous identical data fragment is too long. The address of the same data fragment that appeared before (i.e., the indication information associated with the address of the first data unit) is used to replace the repeated fragment data (the second data unit) for storage.

[0125] 3. Secondary compression:

[0126] Secondary compression compresses the data compressed in step 2 (i.e., the first compressed data) again at the byte level to achieve a better compression effect. The existing compression method uses Huffman coding. However, the Huffman coding information needs to be transmitted along with the compressed data before it can be decoded, which introduces new transmission overhead.

[0127] By improving it, it is not necessary to transmit the entire encoding table, but only needs to know the encoding length of each symbol (i.e. different compressed data units) to encode and decode. The special design of 256 different characters (i.e. different compressed data units) is also to improve the efficiency of encoding, so that the extra information required for transmission and decoding is as little as possible.

[0128] The specific compression steps are as follows:

[0129] 1) Count the frequency (i.e. number of repetitions) of each character that needs to be encoded.

[0130] 2) Based on these frequency information, the depth of the symbol in the traditional Huffman coding tree (that is, the number of bits required to represent the symbol, i.e., the coding length) is calculated.

[0131] 3) Count the number of symbols corresponding to each code length. Based on this information, assign codes to each symbol in increasing order of depth, starting from the character with the shortest depth. The code assignment process is always performed by adding 1 to the previous code. If the depth of the current character is deeper, 0 is directly added after the code.

[0132] 4) Encode and output the compressed information, save the symbol table arranged in frequency order, and then save the first code in each group of codes of the same length and the number of codes in the group as shown in Table 4.

[0133] Table 4 Symbol code length diagram

[0134] symbol a b c d e f g h i j k l m n Code length 3 3 4 4 4 5 5 5 6 6 7 7 7 7

[0135] Table 5 count table

[0136] length 1 2 3 4 5 6 7 Number 0 0 2 3 3 2 4

[0137] Use count[i] to represent the number of codewords of length i, then count[1] = 0, count[2] = 0, count[3] = 2, count[4] = 3, count[5] = 3, count[6] = 2, count[7] = 4, as shown in Table 5. Initially, let the first symbol a be encoded as 000. The encoding of symbol b is a code plus 1, that is, 001. The encoding of symbol c is b code plus 1 followed by a 0, that is, 0100. The encoding of symbol d is c code plus 1, that is, 0101. The encoding of symbol e is d code plus 1, that is, 0110. The encoding of symbol f is e code plus 1 followed by a 0, that is, 01110. By analogy, the entire codeword space can be constructed as shown in Table 6.

[0138] Table 6 Symbol encoding examples

[0139]

[0140]

[0141] With the encoding table, the entire data can be compressed. In order to decompress, the encoding information needs to be attached to the compressed data header. The header information only needs to save the symbol table and the count table, which can greatly reduce the header information. In the example, the symbol table is [a,b,c,d,e,f,g,h,i,j,kl,m,n], and the count table is [0,0,2,3,3,2,4], and the amount of header data is very small. During the decompression process, the decoding information can be constructed and decoded according to the symbol table and the count table through step 3. Finally, the encoding result is output to the final compressed file.

[0142] The test results of this compression method are shown in Table 7.

[0143] Table 7 Compression results

[0144]

[0145] Figure 3 A schematic diagram of the structure of a solid-state nanopore electrical signal data compression device provided by an embodiment of the present invention; Figure 3 As shown, the device comprises: a first processing module and a second processing module, wherein:

[0146] The first processing module is used to normalize the acquired N solid-state nanopore electrical signal data to obtain N normalized data corresponding to the N solid-state nanopore electrical signal data;

[0147] The second processing module is used to replace the second data unit in the N normalized data that is identical to the first data unit with the indication information associated with the address of the first data unit at a predetermined position in the N normalized data, so as to obtain first compressed data, wherein the first data unit includes M normalized data, and the indication information associated with the address is used to indicate the position of the first data unit in the N normalized data.

[0148] Specifically, the first processing module is also used to number the N solid-state nanopore electrical signal data, wherein the same solid-state nanopore electrical signal data uses the same number, and the number corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data, wherein the bits occupied by the number are smaller than the bits occupied by the solid-state nanopore electrical signal data.

[0149] Specifically, the first processing module is also used to determine the hash values ​​corresponding to the N solid-state nanopore electrical signal data respectively according to the hash table, wherein the same solid-state nanopore electrical signal data uses the same hash value, and the hash value corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data, wherein the bits occupied by the hash value are smaller than the bits occupied by the solid-state nanopore electrical signal data.

[0150] Specifically, the first processing module is also used to determine the hash value corresponding to the i-bit data at the predetermined position of each solid-state nanopore electrical signal data according to the hash table; and use the hash value corresponding to the i-bit data at the predetermined position of the solid-state nanopore electrical signal data as the hash value corresponding to the solid-state nanopore electrical signal data; wherein the solid-state nanopore electrical signal data is I-bit data, I>i.

[0151] Specifically, the second processing module is further used to determine a first data unit in the N normalized data that is the same as the second data unit;

[0152] Determining, according to the first data unit, indication information associated with an address of the first data unit;

[0153] The first compressed data is obtained by replacing the second data unit with indication information associated with the address of the first data unit;

[0154] The first data unit includes M normalized data, M is greater than or equal to 3 and less than N, and the length of the address-associated indication information is less than or equal to the length of 3 normalized data.

[0155] Specifically, the device further includes: a third processing module; the third processing module is used to determine the number of repetitions of different compressed data units in the first compressed data;

[0156] Sort the different compressed data units based on the number of repetitions of the different compressed data units to obtain a first data sequence;

[0157] Each compressed data unit in the first data sequence is encoded according to a preset rule to obtain second compressed data.

[0158] Specifically, the third processing module is further used to arrange the different compressed data units in descending order according to the number of repetitions of the different compressed data units to obtain a first data sequence.

[0159] Specifically, the third processing module is further configured to, when the encoding length of the first compressed data unit in the first compressed data is the same as the encoding length of the second compressed data unit, determine the sum of the encoding of the second compressed data unit plus the first predetermined value as the encoding of the first compressed data unit;

[0160] When the coding length of the first compressed data unit in the first compressed data is greater than the coding length of the second compressed data unit, the code of the second compressed data unit is added to the sum of the first predetermined value to obtain the first code, and the second predetermined value is added after the first code to determine it as the code of the first compressed data unit.

[0161] It should be noted that: the solid-state nanopore electrical signal data compression device provided in the above embodiment only uses the division of the above program modules as an example when implementing the corresponding solid-state nanopore electrical signal data compression method. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the processing device is divided into different program modules to complete all or part of the processing described above. In addition, the device provided in the above embodiment and the embodiment of the corresponding method belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0162] Figure 4 A schematic diagram of the structure of another solid-state nanopore electrical signal data compression device provided by an embodiment of the present invention; Figure 4As shown, the device 40 includes: a processor 401 and a memory 402 for storing a computer program that can be run on the processor; when the processor 401 is used to run the computer program, it executes: normalizing the acquired N solid-state nanopore electrical signal data to obtain N normalized data; replacing the second data unit in the N normalized data that is the same as the first data unit with the address-associated indication information of the first data unit at a predetermined position in the N normalized data to obtain the first compressed data, wherein the first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data. Wherein, when the processor runs the computer program, the corresponding processes implemented by the processing device in each method of the embodiment of the present invention are implemented, and for the sake of brevity, they are not repeated here.

[0163] In practical applications, the device 40 may further include: at least one network interface 403. The various components in the solid-state nanopore electrical signal data compression device 40 are coupled together via a bus system 404. It is understood that the bus system 404 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 4 In the figure, various buses are labeled as bus system 404. There may be at least one processor 401. The network interface 403 is used for wired or wireless communication between the solid-state nanopore electrical signal data compression device 40 and other devices.

[0164] The memory 402 in the embodiment of the present invention is used to store various types of data to support the operation of the solid-state nanopore electrical signal data compression device 40 .

[0165] The method disclosed in the above embodiment of the present invention can be applied to the processor 401, or implemented by the processor 401. The processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 401 or the instruction in the form of software. The above processor 401 can be a general processor, a digital signal processor (DSP, DiGital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 401 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined to execute. The software module can be located in a storage medium, which is located in the memory 402. The processor 401 reads the information in the memory 402 and completes the steps of the above method in combination with its hardware.

[0166] In an exemplary embodiment, the solid-state nanopore electrical signal data compression device 40 can be implemented by one or more application-specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field-programmable gate array (FPGA), general-purpose processor, controller, microcontroller (MCU), microprocessor, or other electronic components to perform the aforementioned method.

[0167] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon; when the computer program is executed by a processor, the following steps are performed: normalizing N acquired solid-state nanopore electrical signal data to obtain N normalized data; replacing a second data unit in the N normalized data that is identical to the first data unit with the indication information associated with the address of the first data unit at a predetermined position in the N normalized data to obtain first compressed data, wherein the first data unit includes M normalized data, and the indication information associated with the address is used to indicate the position of the first data unit in the N normalized data.

[0168] Among them, when the computer program is executed by the processor, the corresponding processes implemented by the processing device in each method of the embodiment of the present invention are implemented, and for the sake of brevity, they are not repeated here.

[0169] In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0170] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0171] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0172] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0173] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0174] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A solid-state nanopore electrical signal data compression method, characterized in that: The method comprises: Normalizing the acquired N solid-state nanopore electrical signal data to obtain N normalized data; The address-associated indication information of the first data unit at a predetermined position in the N normalized data is used to replace the second data unit in the N normalized data that is identical to the first data unit, so as to obtain first compressed data, wherein the first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data.

2. The method according to claim 1, characterized in that The normalized acquired N solid-state nanopore electrical signal data are obtained to obtain N normalized data, including: Numbering the N solid-state nanopore electrical signal data, wherein the same solid-state nanopore electrical signal data uses the same number, and the number corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data, wherein the number occupies less bits than the solid-state nanopore electrical signal data.

3. The method according to claim 1, characterized in that The normalized acquired N solid-state nanopore electrical signal data are obtained to obtain N normalized data, including: According to the hash table, determine the hash values ​​corresponding to the N solid-state nanopore electrical signal data, wherein the same solid-state nanopore electrical signal data uses the same hash value, and use the hash value corresponding to the solid-state nanopore electrical signal data as the normalized data of the solid-state nanopore electrical signal data, wherein the bits occupied by the hash value are smaller than the bits occupied by the solid-state nanopore electrical signal data.

4. The method according to claim 3, characterized in that Determining the hash values ​​corresponding to the N pieces of solid-state nanopore electrical signal data respectively according to the hash table includes: Determine, according to the hash table, a hash value corresponding to i bits of data at a predetermined position of each solid-state nanopore electrical signal data; The hash value corresponding to i bits of data at a predetermined position of the solid-state nanopore electrical signal data is used as the hash value corresponding to the solid-state nanopore electrical signal data; wherein the solid-state nanopore electrical signal data is I bits of data, I>i.

5. The method according to claim 1, characterized in that The step of associating the address of the first data unit at the predetermined position in the N normalized data with the indication information to replace the second data unit in the N normalized data that is identical to the first data unit to obtain the first compressed data includes: Determine a first data unit in the N normalized data that is the same as the second data unit; Determining, according to the first data unit, indication information associated with an address of the first data unit; The first compressed data is obtained by replacing the second data unit with indication information associated with the address of the first data unit; The first data unit includes M normalized data, M is greater than or equal to 3 and less than N, and the length of the address-associated indication information is less than or equal to the length of 3 normalized data.

6. The method according to claim 1, characterized in that The method further comprises: determining the number of repetitions of different compressed data units in the first compressed data; Sort the different compressed data units based on the number of repetitions of the different compressed data units to obtain a first data sequence; Each compressed data unit in the first data sequence is encoded according to a preset rule to obtain second compressed data.

7. The method according to claim 6, characterized in that The step of sorting the different compressed data units based on the number of repetitions of the different compressed data units to obtain a first data sequence includes: According to the number of repetitions of the different compressed data units, they are arranged in descending order to obtain a first data sequence.

8. The method according to claim 6, characterized in that The preset rules include: When the code length of the first compressed data unit in the first compressed data is the same as the code length of the second compressed data unit, the sum of the code of the second compressed data unit plus the first predetermined value is determined as the code of the first compressed data unit; When the coding length of the first compressed data unit in the first compressed data is greater than the coding length of the second compressed data unit, the code of the second compressed data unit is added to the sum of the first predetermined value to obtain the first code, and the second predetermined value is added after the first code to determine it as the code of the first compressed data unit.

9. A solid-state nanopore electrical signal data compression device, characterized in that: The device comprises: A first processing module is used to normalize the acquired N solid-state nanopore electrical signal data to obtain N normalized data corresponding to the N solid-state nanopore electrical signal data; The second processing module is used to replace the second data unit in the N normalized data that is identical to the first data unit with the indication information associated with the address of the first data unit at a predetermined position in the N normalized data, so as to obtain the first compressed data, wherein the first data unit includes M normalized data, and the indication information associated with the address is used to indicate the position of the first data unit in the N normalized data.

10. A solid-state nanopore electrical signal data compression device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images