Solid-state nanopore electrical signal data compression method, device and storage medium

Abstract

The application discloses a solid-state nanopore electric signal data compression method, device and storage medium, and the method comprises the following steps: normalizing obtained N solid-state nanopore electric signal data to obtain N normalized data; replacing a second data unit same as a first data unit of a predetermined position in the N normalized data with address-associated indication information of the first data unit, to obtain first compression data, wherein the first data unit comprises M normalized data, and the address-associated indication information is used for indicating the position of the first data unit in the N normalized data.

Description

Technical Field

[0001] This invention relates to the fields of signal processing and data compression, and in particular to a method, apparatus, and computer-readable storage medium for compressing solid-state nanopore electrical signal data. Background Technology

[0002] Nanopore single-molecule detection technology involves filling both sides of a porous insulating film with an ionic solution and applying a voltage. When single molecules such as DNA or proteins in the ionic solution pass through the nanopore, the nanopore is temporarily blocked, causing fluctuations in the current value within the nanopore. Furthermore, because biomolecules pass through the nanopore at extremely high speeds, devices used to capture these current changes typically employ high sampling rates, generating a large amount of electrical signal data. This poses a significant challenge to data transmission and analysis, making raw data compression essential.

[0003] Therefore, there is a need to provide a method for compressing solid-state nanopore electrical signal data to solve the problem that the amount of solid-state nanopore electrical signal data is large and the data transmission and analysis are difficult in the existing technology. Summary of the Invention

[0004] In view of this, the main objective of the present invention is to provide a method, apparatus and storage medium for compressing solid-state nanopore electrical signal data.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0006] According to a first aspect of the present disclosure, a method for compressing electrical signal data in solid-state nanopores is provided, the method comprising:

[0007] The N solid-state nanopore electrical signal data were obtained by normalization, resulting in N normalized data.

[0008] The address-associated indication information of the first data unit at a predetermined position in the N normalized data is used to replace the second data unit that is the same as the first data unit in the N normalized data to obtain the first compressed data. The first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data.

[0009] In some embodiments, the normalized acquisition of N solid-state nanopore electrical signal data yields N normalized data, including:

[0010] The N solid-state nanopore electrical signal data are numbered, wherein the same solid-state nanopore electrical signal data are assigned the same number. The number corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data, wherein the number occupies fewer bits than the solid-state nanopore electrical signal data.

[0011] In some embodiments, the normalized acquisition of N solid-state nanopore electrical signal data yields N normalized data, including:

[0012] According to the hash table, the hash values ​​corresponding to the N solid-state nanopore electrical signal data are determined respectively. The same solid-state nanopore electrical signal data uses the same hash value. The hash value corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data. The number of bits occupied by the hash value is less than the number of bits occupied by the solid-state nanopore electrical signal data.

[0013] In some embodiments, determining the hash values ​​corresponding to the N solid-state nanopore electrical signal data according to the hash table includes:

[0014] Based on the hash table, determine the hash value corresponding to the i bits of the predetermined position of each solid nanopore electrical signal data;

[0015] The hash value corresponding to the i bits of the solid nanopore electrical signal data at a predetermined position is used as the hash value corresponding to the solid nanopore electrical signal data; wherein, the solid nanopore electrical signal data consists of I bits, I>i.

[0016] In some embodiments, replacing the indication information associated with the address of the first data unit at a predetermined position in the N normalized data with the second data unit that is identical to the first data unit in the N normalized data to obtain the first compressed data includes:

[0017] Identify the first data unit among the N normalized data that is identical to the second data unit;

[0018] Based on the first data unit, determine the address association indication information of the first data unit;

[0019] The first compressed data is obtained by replacing the indication information associated with the address of the first data unit with that of the second data unit;

[0020] The first data unit includes M normalized data, where M is greater than or equal to 3 and less than N, and the length of the address-associated indication information is less than or equal to the length of 3 normalized data.

[0021] In some embodiments, the method further includes:

[0022] Determine the number of repetitions of different compressed data units in the first compressed data;

[0023] Based on the number of repetitions of the different compressed data units, the different compressed data units are sorted to obtain a first data sequence;

[0024] Each compressed data unit in the first data sequence is encoded according to a preset rule to obtain the second compressed data.

[0025] In some embodiments, sorting the different compressed data units based on the number of repetitions of the different compressed data units to obtain a first data sequence includes:

[0026] Based on the number of repetitions of the different compressed data units, they are arranged in descending order to obtain the first data sequence.

[0027] In some embodiments, the preset rules include:

[0028] When the encoding length of the first compressed data unit in the first compressed data is the same as the encoding length of the second compressed data unit, the sum of the encoding of the second compressed data unit and the first predetermined value is determined as the encoding of the first compressed data unit.

[0029] When the encoding length of the first compressed data unit in the first compressed data is greater than the encoding length of the second compressed data unit, the encoding of the second compressed data unit is added to the first predetermined value to obtain the first encoding. Then, a second predetermined value is added to the first encoding to determine the encoding of the first compressed data unit.

[0030] According to a second aspect of the present disclosure, a solid-state nanopore electrical signal data compression device is provided, the device comprising:

[0031] The first processing module is used to normalize the acquired N solid nanopore electrical signal data to obtain N normalized data corresponding to the N solid nanopore electrical signal data.

[0032] The second processing module is used to replace the address-associated indication information of the first data unit at a predetermined position in the N normalized data with the second data unit that is the same as the first data unit in the N normalized data to obtain the first compressed data. The first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data.

[0033] According to a third aspect of the present disclosure, a solid-state nanopore electrical signal data compression device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect.

[0034] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in the first aspect. The present invention provides a method, apparatus, and storage medium for compressing solid-state nanopore electrical signal data. The method includes: normalizing N acquired solid-state nanopore electrical signal data to obtain N normalized data; replacing a second data unit identical to the first data unit in the N normalized data with address-associated indication information of a first data unit at a predetermined position in the N normalized data to obtain first compressed data, wherein the first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data; thus, by normalizing the solid-state nanopore electrical signal data and replacing duplicate data units, data compression of the solid-state nanopore electrical signal data is achieved, reducing the storage space occupied by the solid-state nanopore electrical signal data, greatly reducing search time, and thus ensuring timely transmission of massive amounts of data. Attached Figure Description

[0035] Figure 1 A schematic flowchart of a solid-state nanopore electrical signal data compression method provided in an embodiment of the present invention;

[0036] Figure 2 A schematic flowchart of another solid-state nanopore electrical signal data compression method provided in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the structure of a solid-state nanopore electrical signal data compression device provided in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of another solid-state nanopore electrical signal data compression device provided in an embodiment of the present invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the embodiments.

[0040] Before providing a more detailed description of the present invention in conjunction with the embodiments, the relevant technologies for data compression will be explained first.

[0041] Existing data compression methods involve linearly scanning the data, searching within the already scanned data to determine if the current data has appeared before. If it has, compression can be performed directly using position indexing. This search method is extremely inefficient and ignores the temporal regularity of solid-state nanopore electrical signal data, meaning that there are some scattered and repetitive data points within the solid-state nanopore electrical signal data.

[0042] On the other hand, existing data compression methods use Huffman coding for compression, which requires transmitting the Huffman coding table along with the compressed data, introducing new transmission overhead.

[0043] Therefore, there is a need to provide a method for compressing solid-state nanopore electrical signal data to solve the problem of large data volume and difficult data transmission and analysis of solid-state nanopore electrical signal data.

[0044] Figure 1 This is a flowchart illustrating a solid-state nanopore electrical signal data compression method provided in an embodiment of the present invention; as shown below. Figure 1 As shown, the method includes:

[0045] Step 101: Normalize the N solid-state nanopore electrical signal data to obtain N normalized data.

[0046] Here, the two sides of the porous insulating film are filled with ionic solution and a voltage is applied. When a single molecule in the ionic solution passes through the nanopore, the nanopore is temporarily blocked, and the current value inside the nanopore fluctuates. The device captures the electrical signal data generated when the current changes, which is the solid nanopore electrical signal data.

[0047] Specifically, normalization transforms data into values ​​within a certain range. For example, it transforms solid-state nanopore electrical signal data into numbers, where the number of bits used is less than the number of bits used in the solid-state nanopore electrical signal data.

[0048] Step 102: Replace the address-associated indication information of the first data unit at a predetermined position in the N normalized data with the second data unit that is the same as the first data unit in the N normalized data to obtain the first compressed data. The first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data.

[0049] Specifically, the predetermined location can be the position of the first data unit, which is the same as the second data unit, among N normalized data; the indication information associated with the address of the first data unit can include at least one of the following: the address of the first data unit; and identification information that uniquely identifies the address of the first data unit. For example, the indication information associated with the address of the first data unit can be the address of the first normalized data in the first data unit and the length of the first data unit.

[0050] Here, N normalized data points can form a normalized data sequence. The first data unit can be M adjacent normalized data points at a predetermined position in the normalized data sequence. For example, the first data unit can be the first M adjacent normalized data points in the normalized data sequence. The second data unit can be any data unit following the first data unit. The second data unit can contain M adjacent normalized data points. Thus, the normalized data sequence can contain the first data unit and multiple second data units, which can be different or the same.

[0051] If the second data unit is identical to the first data unit, then the second data unit is replaced with the address-associated indication information of the first data unit. This reduces the number of bits in the second data unit, effectively compressing the data.

[0052] For example, the normalized data of solid-state nanopore electrical signal data can be the hash values ​​corresponding to the solid-state nanopore electrical signal data. Thus, N normalized data can form a hash table, where the hash table contains N hash values. M hash values ​​constitute a first data unit. For instance, the first M hash values ​​of the hash table can be used as the first data unit, and the remaining hash values ​​in the hash table can be used to form one or more second data units. The second data unit can contain M adjacent hash values. If the second data unit is the same as the first data unit, then the address-associated indication information of the first data unit is used to replace the second data unit. This reduces the number of bits in the second data unit, achieving the effect of data compression.

[0053] In practical applications, when decompressing the first compressed data, the corresponding data unit can be found at the corresponding position of the already decompressed data when the data unit indicated by the address associated with the first data unit is scanned.

[0054] Thus, taking into account the large number of repetitive data units in solid-state nanopore electrical signal data, the compression rate is improved by compressing the repetitive data units.

[0055] In some embodiments, the normalized acquisition of N solid-state nanopore electrical signal data yields N normalized data, including:

[0056] The N solid-state nanopore electrical signal data are numbered, wherein the same solid-state nanopore electrical signal data are assigned the same number. The number corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data, wherein the number occupies fewer bits than the solid-state nanopore electrical signal data.

[0057] In one possible implementation, each solid-state nanopore electrical signal data occupies 16 bits, and each number occupies 8 bits.

[0058] Specifically, the initial number can be set to 0. The solid-state nanopore electrical signal data is scanned, and N solid-state nanopore electrical signal data are numbered sequentially. It is determined whether the scanned solid-state nanopore electrical signal data is the same as the numbered solid-state nanopore electrical signal data. If they are different, the current number is used to represent the solid-state nanopore electrical signal data, and the number is incremented by 1. If they are the same, the number that is the same as the numbered solid-state nanopore electrical signal data is used to represent the known solid-state nanopore electrical signal data. The number corresponding to each solid-state nanopore electrical signal data is used as normalized data.

[0059] In some embodiments, the normalized acquisition of N solid-state nanopore electrical signal data yields N normalized data, including:

[0060] According to the hash table, the hash values ​​corresponding to the N solid-state nanopore electrical signal data are determined respectively. The same solid-state nanopore electrical signal data uses the same hash value. The hash value corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data. The number of bits occupied by the hash value is less than the number of bits occupied by the solid-state nanopore electrical signal data.

[0061] In one possible implementation, each solid-state nanopore electrical signal data occupies 16 bits, and each hash value occupies 8 bits.

[0062] Here, a hash table can be pre-built, which includes different hash values ​​corresponding to different solid-state nanopore electrical signal data. The hash value can be a number.

[0063] Specifically, the solid-state nanopore electrical signal data is scanned. Based on the hash table, the hash value corresponding to each solid-state nanopore electrical signal data is determined. It is then determined whether the scanned solid-state nanopore electrical signal data is the same as the previously scanned solid-state nanopore electrical signal data. If they are different, the hash value corresponding to the solid-state nanopore electrical signal data is obtained. If they are the same, the hash value of the previously scanned solid-state nanopore electrical signal data is used as the hash value corresponding to the solid-state nanopore electrical signal data. The hash value corresponding to each solid-state nanopore electrical signal data is used as normalized data.

[0064] In some embodiments, determining the hash values ​​corresponding to the N solid-state nanopore electrical signal data according to the hash table includes:

[0065] Based on the hash table, determine the hash value corresponding to the i bits of the predetermined position of each solid nanopore electrical signal data;

[0066] The hash value corresponding to the i bits of the solid nanopore electrical signal data at a predetermined position is used as the hash value corresponding to the solid nanopore electrical signal data; wherein, the solid nanopore electrical signal data consists of I bits, I>i.

[0067] Here, based on the characteristic that the i bits of most solid-state nanopore electrical signal data are the same, a hash table can be pre-built by taking the i bits of the predetermined position of each solid-state nanopore electrical signal data. The hash table includes different hash values ​​corresponding to the i bits of the predetermined position of different solid-state nanopore electrical signal data. The hash value corresponding to the i bits of the predetermined position of the solid-state nanopore electrical signal data is used as the hash value corresponding to the solid-state nanopore electrical signal data.

[0068] Specifically, the solid-state nanopore electrical signal data is 16 bits. Since the first 8 bits of most solid-state nanopore electrical signal data are the same, the last 8 bits of each solid-state nanopore electrical signal data can be taken to determine different hash values ​​corresponding to the last 8 bits of different solid-state nanopore electrical signal data. The solid-state nanopore electrical signal data is scanned to determine whether the scanned solid-state nanopore electrical signal data is the same as the scanned solid-state nanopore electrical signal data. If they are not the same, the hash value corresponding to the last 8 bits of the solid-state nanopore electrical signal data is used as the hash value corresponding to the solid-state nanopore electrical signal data. If they are the same, the hash value corresponding to the scanned solid-state nanopore electrical signal data is used as the hash value corresponding to the solid-state nanopore electrical signal data.

[0069] Thus, by taking advantage of the high repetition rate of solid-state nanopore electrical signal data, the original solid-state nanopore electrical signal data is represented by numbers, thereby converting two bytes of data into one byte of data.

[0070] In practical applications, if there are more than X distinct solid-state nanopore electrical signal data points out of N data points, then the N data points can be divided into Y data blocks. Each data block contains X distinct solid-state nanopore electrical signal data points, where X is a positive integer (e.g., 256). Y is a positive integer greater than or equal to 2. This ensures that the normalized data corresponding to each solid-state nanopore electrical signal data point occupies only one byte.

[0071] In some embodiments, replacing the indication information associated with the address of the first data unit at a predetermined position in the N normalized data with the second data unit that is identical to the first data unit in the N normalized data to obtain the first compressed data includes:

[0072] Identify the first data unit among the N normalized data that is identical to the second data unit;

[0073] Based on the first data unit, determine the address association indication information of the first data unit;

[0074] The first compressed data is obtained by replacing the indication information associated with the address of the first data unit with that of the second data unit;

[0075] The first data unit includes M normalized data, where M is greater than or equal to 3 and less than N, and the length of the address-associated indication information is less than or equal to the length of 3 normalized data.

[0076] In one possible implementation, the first data unit includes M normalized data, where M is greater than or equal to 3 and less than N. That is, the length of the second data unit and the first data unit is greater than or equal to the length of 3 normalized data. Meanwhile, the length of the address-associated indication information of the first data unit is usually less than or equal to the length of 3 normalized data. In this way, by replacing the second data unit that is the same as the first data unit with the address-associated indication information of the first data unit, the problem of increasing storage due to the second data unit being too short can be avoided.

[0077] Specifically, assuming that the address associated with the first data unit is the address of the first normalized data in the first data unit and the length of the first data unit, when a first data unit identical to the second data unit is found in N normalized data, the address of the first normalized data in the first data unit and the length of the first data unit are determined, and the address of the first normalized data in the first data unit and the length of the first data unit are used to replace the second data unit to obtain the first compressed data.

[0078] Thus, by specifying that the same data unit is greater than or equal to three bytes, the problem of replacing the second data unit with the address associated with the first data unit due to the second data unit being too short can be avoided, which would increase storage.

[0079] In some embodiments, the method further includes:

[0080] Determine the number of repetitions of different compressed data units in the first compressed data;

[0081] Based on the number of repetitions of the different compressed data units, the different compressed data units are sorted to obtain a first data sequence;

[0082] Each compressed data unit in the first data sequence is encoded according to a preset rule to obtain the second compressed data.

[0083] In one possible implementation, the encoding length of different compressed data units is obtained based on the number of repetitions of different compressed data units in the first compressed data.

[0084] Specifically, based on the number of repetitions of different compressed data units in the first compressed data, the depth of each different compressed data unit in the traditional Huffman coding tree is calculated, thereby obtaining the coding length of different compressed data units.

[0085] In some embodiments, sorting the different compressed data units based on the number of repetitions of the different compressed data units to obtain a first data sequence includes:

[0086] Based on the number of repetitions of the different compressed data units, they are arranged in descending order to obtain the first data sequence.

[0087] Here, the more times a compressed data unit is repeated, the shorter the corresponding encoding length.

[0088] Specifically, the data units are arranged in descending order of the number of repetitions of different compressed data units, and the encoding lengths of the corresponding compressed data units are also arranged in ascending order of the number of repetitions.

[0089] In some embodiments, the preset rules include:

[0090] When the encoding length of the first compressed data unit in the first compressed data is the same as the encoding length of the second compressed data unit, the sum of the encoding of the second compressed data unit and the first predetermined value is determined as the encoding of the first compressed data unit.

[0091] When the encoding length of the first compressed data unit in the first compressed data is greater than the encoding length of the second compressed data unit, the encoding of the second compressed data unit is added to the first predetermined value to obtain the first encoding. Then, a second predetermined value is added to the first encoding to determine the encoding of the first compressed data unit.

[0092] Here, the first compressed data unit and the second compressed data unit are compressed data units arranged in the first data sequence according to the number of repetitions. The number of repetitions of the first compressed data unit is greater than or equal to the number of repetitions of the second compressed data unit, that is, the encoding length of the first compressed data unit is less than or equal to the encoding length of the second compressed data unit.

[0093] Here, the first predetermined value can be 1, and the second predetermined value can be 0.

[0094] Specifically, the preset rule can be that, during the encoding process, if the encoding lengths of the first compressed data unit and the second compressed data unit are the same, then the encoding of the second compressed data unit is the sum of the encoding of the first compressed data unit plus 1; if the encoding length of the second compressed data unit is greater than the encoding length of the first compressed data unit, then the encoding of the second compressed data unit is the sum of the encoding of the first compressed data unit plus 1, and 0 is added after the encoding.

[0095] For example, Table 1 is a schematic table of the encoding length of a compressed data unit provided in an embodiment of the present invention.

[0096] Encoding length 3 3 4 4 4 5 5 5 6 6 7 7 7 7

[0097] Table 1

[0098] Specifically, the different compressed data units include a, b, c, d, e, f, g, h, i, j, k, l, m, n, and the encoding length of a is 3, the encoding length of b is 3, the encoding length of c is 4, the encoding length of d is 4, the encoding length of e is 4, the encoding length of f is 5, the encoding length of g is 5, the encoding length of h is 5, the encoding length of i is 6, the encoding length of j is 6, the encoding length of k is 7, the encoding length of l is 7, the encoding length of m is 7, and the encoding length of n is 7.

[0099] For example, Table 2 is a schematic table of the number of encoding lengths provided in an embodiment of the present invention.

[0100] Number 0 0 2 3 3 2 4

[0101] Table 2

[0102] Specifically, the number of different compressed data units with a code length of 1 is 0, the number of different compressed data units with a code length of 2 is 0, the number of different compressed data units with a code length of 3 is 2, the number of different compressed data units with a code length of 4 is 3, the number of different compressed data units with a code length of 5 is 3, the number of different compressed data units with a code length of 6 is 2, and the number of different compressed data units with a code length of 7 is 4. The number of different compressed data units with a code length of i can be represented by count[i], then count[1] = 0, count[2] = 0, count[3] = 2, count[4] = 3, count[5] = 3, count[6] = 2, count[7] = 4.

[0103] For example, Table 3 is a schematic table of compressed data unit encoding provided in an embodiment of the present invention.

[0104] a 3 000 b 3 001 c 4 0100 d 4 0101 e 4 0110 f 5 01110 g 5 01111 h 5 10000 i 6 100010 j 6 100011 k 7 1001000 l 7 1001001 m 7 1001010 n 7 1001011

[0105] Table 3

[0106] Specifically, let the first compressed data unit 'a' be encoded as 000; according to the encoding rules, the encoded data unit 'b' is 'a' plus 1, i.e., 001; the encoded data unit 'c' is 'b' plus 1, then appended with a 0, i.e., 0100; the encoded data unit 'd' is 'c' plus 1, i.e., 0101; the encoded data unit 'e' is 'd' plus 1, i.e., 0110; the encoded data unit 'f' is 'e' plus 1, then appended with a 0, i.e., 01110; the encoded data unit 'g' is 'f' plus 1, i.e., 01111; the compressed data... The encoding of unit h is g encoding plus 1, i.e., 10000; the encoding of compressed data unit i is h encoding plus 1, and then a 0 is added to the end, i.e., 100010; the encoding of compressed data unit j is i encoding plus 1, i.e., 100011; the encoding of compressed data unit k is j encoding plus 1, and then a 0 is added to the end, i.e., 1001000; the encoding of compressed data unit l is k encoding plus 1, i.e., 1001001; the encoding of compressed data unit m is l encoding plus 1, i.e., 1001010; the encoding of compressed data unit n is m encoding plus 1, i.e., 1001011.

[0107] In practical applications, when decompressing compressed data, it is only necessary to save the compressed data unit table and the code length count table in the header of the compressed information. There is no need to attach the code information to the header of the compressed information. During the decompression process, the decoding information is constructed according to the compressed data unit table and the code length count table according to the preset rules, and then the decoding is performed.

[0108] For example, the compressed data unit table can be [a, b, c, d, e, f, g, h, i, j, k, l, m, n], and the encoding length number table can be [0, 0, 2, 3, 3, 2, 4].

[0109] Thus, by improving Huffman coding, the entire coding table does not need to be transmitted during data transmission. Instead, only the coding length of each different compressed data unit needs to be known for encoding and decoding, reducing the amount of header information stored during compression and improving the compression ratio.

[0110] The following provides several specific examples in conjunction with any of the above embodiments:

[0111] The steps of the solid-state nanopore electrical signal data compression method are as follows: Figure 2As shown, the main steps are as follows: (1) In view of the characteristic of the solid pore electrical signal data (i.e., solid nanopore electrical signal data) having a lot of repetitive data, the normalization process using the hash algorithm greatly improves the compression ratio, compressing the original 16-bit data into 8 bits (i.e., normalized data); (2) In view of the characteristic of the solid pore electrical signal data (i.e., solid nanopore electrical signal data) having a lot of repetitive segments, the repetitive segments are compressed using weak regularity data, and the normalized 8-bit data is converted into data with three bytes as a unit; (3) At the byte level, based on the use of the paradigm-improved Huffman algorithm to replace the traditional Huffman algorithm, the amount of header information stored during compression is reduced, and the compression ratio is improved.

[0112] 1. Data normalization:

[0113] Statistical analysis of sequencing data obtained at different molecular and nanopore sizes revealed that while the volume of gene sequencing data is large, the number of distinct values ​​is very limited. Therefore, gene sequencing data can be normalized. The purpose of normalization is to limit the values ​​of nanopore electrical signal data (i.e., solid-state nanopore electrical signal data) to a range of 256 distinct values. This ensures that each value is represented by only one byte, facilitating subsequent processing. Due to the data range limitation, iterative compression needs to be performed every 256 distinct values. This range is the optimal range determined through analysis of the nanopore electrical signal data.

[0114] The specific method for data normalization is as follows: An initial representation `id` is set to 0. The solid-state nanopore data (i.e., solid-state nanopore electrical signal data) is scanned. Whenever an unknown value appears, it is represented by `id`, and `id` is incremented by 1. This way, after scanning a segment of sequencing data, the original 2-byte value can be represented by a shorter 1-byte value (i.e., normalized data). The method then determines whether a scanned value is unknown. A hash table of length 2^8 (256) is constructed. All data points (i.e., solid-state nanopore electrical signal data) are traversed. The last eight bits of each data point are taken. If the last eight bits of a data point have not appeared before, the value of that data point is placed in the hash table at the position corresponding to its last eight bits. If the last eight bits of a data point have appeared before and the value of that data point is different from previously seen values, the nearest available position in the hash table after the position corresponding to the last eight bits of that data point is found and the value is placed there. Generally, the first eight bits of most data points are the same, so a hash table of length 256 is sufficient to hold the values ​​of all data points. If a single hash table is insufficient, the processed data blocks are further compressed after the first hash table is full, and then a new hash table is built to process new data blocks. Using hash tables speeds up the judgment process and significantly reduces the time spent compressing data. After this compression, the data is reduced from 16 bits to 8 bits.

[0115] 2. Compression of weakly regular data:

[0116] Traditional compression methods involve linearly scanning the data (i.e., solid-state nanopore electrical signal data) to determine if the current data has appeared in the previously scanned data. If it has, compression can be performed directly using position indexing. However, this search method is extremely inefficient, ignoring the characteristics of nanopore current data (i.e., solid-state nanopore electrical signal data), namely, its temporal regularity and the presence of scattered duplicate data.

[0117] For applications involving nanopore electrical signal data transmission where data processing latency is extremely constrained, data compression is performed using the repetitive structural information of the data. This space-for-time tradeoff method greatly reduces search time, thereby ensuring that massive amounts of data can be transmitted in a timely manner.

[0118] First, define the length of the string S (i.e., the electrical signal data of solid-state nanopores) as N, and the substrings S(i,j) of string S, where 1≤i,j≤N. For the prefix substring S(1,j), let L(i,j) be the first character S. i The substring (i.e., the first data unit) and the first character S j The length of the longest matching substring (i.e., the second data unit) is:

[0119] L(i,j)=max{l|S(i,i+l-1)=S(j,j+l-1)}, where l≤Nj

[0120] The string S(j,j+l-1) (i.e., the second data unit) is said to match the string S(i,i+l-1) (i.e., the first data unit), and the matching length is l.

[0121] Define p(j) as the value of i for the longest matching in all cases, i.e.

[0122] p(j) = argmax i {L(i,j)}, where 1≤i≤j

[0123] In the specific implementation, a sliding window dictionary is used to store historical characters. The values ​​of p and L are found in the sliding window, and (p, L) is used to represent the current segment of data (i.e., the second data unit). The decompression process is the reverse of the compression process. When a segment of data represented by (p, L) is encountered, a segment of data of length L is found at the corresponding position p in the historical data and decompressed.

[0124] To avoid the problem of excessive storage space caused by replacing data with addresses due to excessively short duplicate data segments, the normalized 8-bit data is converted into data in units of three bytes, stored as integers, and a hash table is constructed. The system searches for whether the three-byte data has appeared before; if the search distance exceeds the sliding window size, the search stops. After finding a duplicate data point, it compares subsequent data points for consecutive identical data, continuing until different data points are found or the consecutive identical data segments become too long, at which point the comparison stops. The address of the previously occurring identical data segment (i.e., the indication information associated with the address of the first data unit) replaces the duplicate data segment (the second data unit) for storage.

[0125] 3. Secondary compression:

[0126] Secondary compression further compresses the data already compressed in step 2 (i.e., the first compressed data) at the byte level to achieve better compression results. Existing compression methods use Huffman coding. However, Huffman-coded information needs to be transmitted along with the compressed data for decoding, introducing new transmission overhead.

[0127] By improving upon this design, it eliminates the need to transmit the entire encoding table; instead, it only requires knowing the encoding length of each symbol (i.e., different compressed data units) to perform encoding and decoding. The special design of 256 different characters (i.e., different compressed data units) is also intended to improve encoding efficiency, minimizing the amount of additional information needed for transmission and decoding.

[0128] The specific compression steps are as follows:

[0129] 1) Count the frequency (i.e. the number of repetitions) of each character that needs to be encoded.

[0130] 2) Calculate the depth of the symbol in the traditional Huffman coding tree based on this frequency information (that is, the number of bits required to represent the symbol, i.e., the coding length).

[0131] 3) Count how many symbols correspond to each code length. Based on this information, assign codes to each symbol starting from the character with the shortest depth and proceeding in ascending order of depth. The code assignment process always involves incrementing the previous code by 1. If the current character has a deeper depth, simply append a 0 to the end of the code.

[0132] 4) Encode and output compressed information, save the symbol table arranged in frequency order, and then save the first code in each group of codes of the same length and the number of codes in that group, as shown in Table 4.

[0133] Table 4. Symbol Code Length Diagram

[0134] Code length 3 3 4 4 4 5 5 5 6 6 7 7 7 7

[0135] Table 5 count table

[0136] Number 0 0 2 3 3 2 4

[0137] Let count[i] represent the number of codewords of length i. Then count[1] = 0, count[2] = 0, count[3] = 2, count[4] = 3, count[5] = 3, count[6] = 2, count[7] = 4, as shown in Table 5. Initially, let the code of the first symbol a be 000. The code of symbol b is the code of a plus 1, i.e., 001. The code of symbol c is the code of b plus 1 followed by a 0, i.e., 0100. The code of symbol d is the code of c plus 1, i.e., 0101. The code of symbol e is the code of d plus 1, i.e., 0110. The code of symbol f is the code of e plus 1 followed by a 0, i.e., 01110. And so on, the entire codeword space can be constructed as shown in Table 6.

[0138] Table 6. Examples of Symbol Encoding

[0139]

[0140]

[0141] With the encoding table, the entire data can be compressed. For decompression, the encoded information needs to be appended to the header of the compressed data. The header only needs to store the symbol table and the count table, greatly reducing the amount of header data. In the example, the symbol table is [a,b,c,d,e,f,g,h,i,j,kl,m,n], and the count table is [0,0,2,3,3,2,4], resulting in a very small header. During decompression, the decoding information is constructed based on the symbol table and the count table in step 3. Finally, the encoded result is output to the final compressed file.

[0142] The test results of this compression method are shown in Table 7.

[0143] Table 7 Compression Results

[0144]

[0145] Figure 3 This is a schematic diagram of the structure of a solid-state nanopore electrical signal data compression device provided in an embodiment of the present invention; as shown below. Figure 3 As shown, the device includes: a first processing module and a second processing module, wherein,

[0146] The first processing module is used to normalize the acquired N solid nanopore electrical signal data to obtain N normalized data corresponding to the N solid nanopore electrical signal data.

[0147] The second processing module is used to replace the address-associated indication information of the first data unit at a predetermined position in the N normalized data with the second data unit that is the same as the first data unit in the N normalized data to obtain the first compressed data. The first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data.

[0148] Specifically, the first processing module is further configured to number the N solid-state nanopore electrical signal data, wherein the same solid-state nanopore electrical signal data uses the same number, and the number corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data, wherein the number occupies fewer bits than the number of bits occupied by the solid-state nanopore electrical signal data.

[0149] Specifically, the first processing module is further configured to determine the hash values ​​corresponding to the N solid-state nanopore electrical signal data according to the hash table, wherein the same solid-state nanopore electrical signal data uses the same hash value, and the hash value corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data, wherein the number of bits occupied by the hash value is less than the number of bits occupied by the solid-state nanopore electrical signal data.

[0150] Specifically, the first processing module is further configured to determine, according to the hash table, the hash value corresponding to the i bits of data at a predetermined position of each solid-state nanopore electrical signal data; and to use the hash value corresponding to the i bits of data at the predetermined position of the solid-state nanopore electrical signal data as the hash value corresponding to the solid-state nanopore electrical signal data; wherein, the solid-state nanopore electrical signal data consists of I bits, where I > i.

[0151] Specifically, the second processing module is further configured to determine a first data unit that is identical to the second data unit among the N normalized data;

[0152] Based on the first data unit, determine the address association indication information of the first data unit;

[0153] The first compressed data is obtained by replacing the indication information associated with the address of the first data unit with that of the second data unit;

[0154] The first data unit includes M normalized data, where M is greater than or equal to 3 and less than N, and the length of the address-associated indication information is less than or equal to the length of 3 normalized data.

[0155] Specifically, the device further includes: a third processing module; the third processing module is used to determine the number of repetitions of different compressed data units in the first compressed data;

[0156] Based on the number of repetitions of the different compressed data units, the different compressed data units are sorted to obtain a first data sequence;

[0157] Each compressed data unit in the first data sequence is encoded according to a preset rule to obtain the second compressed data.

[0158] Specifically, the third processing module is further configured to arrange the different compressed data units in descending order of repetition frequency to obtain a first data sequence.

[0159] Specifically, the third processing module is further configured to, when the encoding length of the first compressed data unit in the first compressed data is the same as the encoding length of the second compressed data unit, add the encoding of the second compressed data unit to a first predetermined value and then determine the encoding of the first compressed data unit as the sum of the two values.

[0160] When the encoding length of the first compressed data unit in the first compressed data is greater than the encoding length of the second compressed data unit, the encoding of the second compressed data unit is added to the first predetermined value to obtain the first encoding. Then, a second predetermined value is added to the first encoding to determine the encoding of the first compressed data unit.

[0161] It should be noted that the solid-state nanopore electrical signal data compression device provided in the above embodiments is only illustrated by the division of the above-described program modules when implementing the corresponding solid-state nanopore electrical signal data compression method. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the processing device can be divided into different program modules to complete all or part of the processing described above. In addition, the device and the corresponding method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0162] Figure 4 This is a schematic diagram of another solid-state nanopore electrical signal data compression device provided in an embodiment of the present invention; as shown below. Figure 4As shown, the device 40 includes a processor 401 and a memory 402 for storing a computer program capable of running on the processor. When the processor 401 runs the computer program, it performs the following: normalizing N solid-state nanopore electrical signal data to obtain N normalized data; replacing a second data unit in the N normalized data that is identical to the first data unit with an address-associated indication information of a first data unit at a predetermined position in the N normalized data to obtain first compressed data, wherein the first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data. The processor, when running the computer program, implements the corresponding processes implemented by the processing device in the various methods of this embodiment of the invention; for simplicity, these will not be elaborated further here.

[0163] In practical applications, the device 40 may further include at least one network interface 403. The various components in the solid-state nanopore electrical signal data compression device 40 are coupled together via a bus system 404. It is understood that the bus system 404 is used to achieve communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 4 All buses are labeled as bus system 404. The number of processors 401 can be at least one. Network interface 403 is used for wired or wireless communication between the solid-state nanopore electrical signal data compression device 40 and other devices.

[0164] The memory 402 in this embodiment of the invention is used to store various types of data to support the operation of the solid-state nanopore electrical signal data compression device 40.

[0165] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 402. Processor 401 reads the information in memory 402 and combines its hardware to complete the steps of the aforementioned method.

[0166] In an exemplary embodiment, the solid-state nanopore electrical signal data compression device 40 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0167] This invention also provides a computer-readable storage medium storing a computer program thereon; when the computer program is run by a processor, it performs the following: normalizing the acquired N solid-state nanopore electrical signal data to obtain N normalized data; replacing the address-associated indication information of the first data unit at a predetermined position in the N normalized data with the address-associated indication information of the second data unit that is the same as the first data unit in the N normalized data to obtain first compressed data, wherein the first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data.

[0168] The computer program, when run by the processor, implements the corresponding processes implemented by the processing device in the various methods of the embodiments of the present invention. For the sake of brevity, these will not be described in detail here.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0170] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0172] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0174] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for compressing electrical signal data in solid-state nanopores, characterized in that, The method includes: The N solid-state nanopore electrical signal data obtained by normalization are used to obtain N normalized data; the normalization is to convert the solid-state nanopore electrical signal data into numbers, wherein the number of bits occupied by the number of numbers is less than the number of bits occupied by the solid-state nanopore electrical signal data. The address-associated indication information of the first data unit at a predetermined position in the N normalized data is used to replace the second data unit that is identical to the first data unit in the N normalized data to obtain the first compressed data. The first data unit comprises M normalized data units, and the address-associated indication information indicates the position of the first data unit in the N normalized data units. The address-associated indication information of the first data unit is the address of the first normalized data unit in the first data unit and the length of the first data unit. The step of replacing the address-associated indication information of the first data unit at a predetermined position in the N normalized data with the second data unit that is identical to the first data unit in the N normalized data to obtain the first compressed data includes: Identify the first data unit among the N normalized data that is identical to the second data unit; Based on the first data unit, determine the address association indication information of the first data unit; The first compressed data is obtained by replacing the instruction information associated with the address of the first data unit with that of the second data unit; The first data unit includes M normalized data, where M is greater than or equal to 3 and less than N, and the length of the address-associated indication information is less than or equal to the length of 3 normalized data.

2. The method according to claim 1, characterized in that, The normalized acquisition of N solid-state nanopore electrical signal data yields N normalized data, including: The N solid-state nanopore electrical signal data are numbered, wherein the same solid-state nanopore electrical signal data are assigned the same number. The number corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data, wherein the number occupies fewer bits than the solid-state nanopore electrical signal data.

3. The method according to claim 1, characterized in that, The normalized acquisition of N solid-state nanopore electrical signal data yields N normalized data, including: According to the hash table, the hash values ​​corresponding to the N solid-state nanopore electrical signal data are determined respectively. The same solid-state nanopore electrical signal data uses the same hash value. The hash value corresponding to the solid-state nanopore electrical signal data is used as the normalized data of the solid-state nanopore electrical signal data. The number of bits occupied by the hash value is less than the number of bits occupied by the solid-state nanopore electrical signal data.

4. The method according to claim 3, characterized in that, The step of determining the hash values ​​corresponding to the N solid-state nanopore electrical signal data according to the hash table includes: Based on the hash table, determine the hash value corresponding to the i bits of the predetermined position of each solid nanopore electrical signal data; The hash value corresponding to the i bits of the solid nanopore electrical signal data at a predetermined position is used as the hash value corresponding to the solid nanopore electrical signal data; wherein, the solid nanopore electrical signal data consists of I bits, I>i.

5. The method according to claim 1, characterized in that, The method further includes: Determine the number of repetitions of different compressed data units in the first compressed data; Based on the number of repetitions of the different compressed data units, the different compressed data units are sorted to obtain a first data sequence; Each compressed data unit in the first data sequence is encoded according to a preset rule to obtain the second compressed data.

6. The method according to claim 5, characterized in that, The first data sequence is obtained by sorting the different compressed data units based on the number of repetitions of the different compressed data units, including: Based on the number of repetitions of the different compressed data units, they are arranged in descending order to obtain the first data sequence.

7. The method according to claim 5, characterized in that, The preset rules include: When the encoding length of the first compressed data unit in the first compressed data is the same as the encoding length of the second compressed data unit, the sum of the encoding of the second compressed data unit and the first predetermined value is determined as the encoding of the first compressed data unit. When the encoding length of the first compressed data unit in the first compressed data is greater than the encoding length of the second compressed data unit, the encoding of the second compressed data unit is added to the first predetermined value to obtain the first encoding. Then, a second predetermined value is added to the first encoding to determine the encoding of the first compressed data unit.

8. A solid-state nanopore electrical signal data compression device, characterized in that, The device includes: The first processing module is used to normalize the acquired N solid nanopore electrical signal data to obtain N normalized data corresponding to the N solid nanopore electrical signal data; the normalization is to convert the solid nanopore electrical signal data into a number, wherein the number occupies fewer bits than the solid nanopore electrical signal data. The second processing module is used to replace the address-associated indication information of the first data unit at a predetermined position in the N normalized data with the second data unit that is the same as the first data unit in the N normalized data to obtain the first compressed data. The first data unit includes M normalized data, and the address-associated indication information is used to indicate the position of the first data unit in the N normalized data. The address-associated indication information of the first data unit is the address of the first normalized data in the first data unit and the length of the first data unit. The second processing module is further configured to determine a first data unit that is identical to the second data unit among the N normalized data; determine the address association indication information of the first data unit based on the first data unit; and obtain the first compressed data by replacing the address association indication information of the first data unit with that of the second data unit. The first data unit includes M normalized data, where M is greater than or equal to 3 and less than N, and the length of the address-associated indication information is less than or equal to the length of 3 normalized data.

9. A solid-state nanopore electrical signal data compression device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images