Methods, apparatus, electronic devices and storage media for processing FASTQ data
By filtering and compressing the row sequence identifiers of FASTQ data, the problem of excessive storage space occupied by FASTQ data is solved, and the data restoration process is simplified and the space usage is reduced.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BGI TECH SOLUTIONS CO LTD
- Filing Date
- 2022-07-25
- Publication Date
- 2026-05-26
AI Technical Summary
FASTQ data occupies a large amount of storage space, and the filtered data is retained for a long time, which further increases the storage space usage.
The original FASTQ file is filtered to obtain the FASTQ file to be processed. The row sequence identifier of each sequence unit is compressed, the FASTQ file to be processed is deleted, and the corresponding sequence unit is found from the original FASTQ file through the compressed sequence identifier to achieve restoration.
It reduces the space occupied by the FASTQ files to be processed and simplifies the restoration process, making the data usage process simple and easy to operate.
Smart Images
Figure CN117795605B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for processing FASTQ data. Background Technology
[0002] In recent years, with the development of sequencing technology, the price of sequencing has become increasingly lower, leading to a surge in the amount of sequencing data produced. How to effectively reduce the storage space of FASTQ sequencing data has become an urgent problem to be solved.
[0003] To reduce the storage space occupied by FASTQ data, after sequencing is completed by the sequencing equipment, the raw FASTQ data is obtained. This raw FASTQ data contains problems such as sequencing errors and low sequencing quality values. After data filtering, filtered FASTQ data is obtained. Finally, the filtered FASTQ data is compressed based on a preset compression algorithm to reduce the storage space occupied by the FASTQ data.
[0004] However, in practical applications, the filtered FASTQ data is analyzed and used for a short time, but is retained for a long time. If there is too much FASTQ data, it may further lead to the occupation of storage space. Summary of the Invention
[0005] This application provides a method, apparatus, electronic device, and storage medium for processing FASTQ data. Its main objective is to address the problem of FASTQ data consuming excessive storage space.
[0006] According to a first aspect of this application, a method for processing FASTQ data is provided, comprising:
[0007] The original FASTQ file is filtered to obtain a FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data.
[0008] Obtain the row sequence identifier of each sequence unit in the FASTQ file to be processed, compress the row sequence identifier, and delete the FASTQ file to be processed.
[0009] Optionally, the sequence unit corresponding to the compressed sequence identifier can be found in the original FASTQ file to restore the FASTQ file to be processed.
[0010] Optionally, obtaining the row sequence identifier of each sequence unit in the FASTQ file to be processed and compressing the row sequence identifier includes:
[0011] Obtain the sequence identifier of the first line of each sequence unit in the FASTQ file to be processed, and compress the sequence identifier of the first line.
[0012] Optionally, the step of searching for the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file includes:
[0013] The compressed sequence identifiers are read sequentially and stored in a preset data set;
[0014] The compressed sequence identifiers and corresponding sequence units contained in the preset data set in the original FASTQ file are output to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored.
[0015] Optionally, the step of searching for the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file includes:
[0016] The sequence identifiers of the compressed first row are read sequentially, and the sequence identifiers of the first row are stored as the primary key in a preset data set;
[0017] The sequence units corresponding to the sequence identifiers of the first row in the preset data set in the original FASTQ file are output to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored. The restored FASTQ file to be processed stores sequence units containing four rows of sequences.
[0018] Optionally, after obtaining the row sequence identifier of each sequence unit in the FASTQ file to be processed and compressing the row sequence identifier, the method further includes:
[0019] Clear the filtered sequencing data in the FASTQ file to be processed.
[0020] Optionally, after filtering the original FASTQ file to obtain the FASTQ file to be processed, the method further includes:
[0021] The FASTQ file to be processed is subjected to hash encryption calculation to obtain the encrypted hash value.
[0022] Optionally, after searching for the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file to restore the FASTQ file to be processed, the method further includes:
[0023] The pre-set encryption algorithm is used to encrypt the restored FASTQ file to be processed to obtain a second encryption key;
[0024] Based on whether the first encryption key and the second encryption key are consistent, it is determined whether there is any loss in the restoration process of the FASTQ data to be processed.
[0025] According to a second aspect of this application, a FASTQ data processing apparatus is provided, comprising:
[0026] The processing unit is used to filter the original FASTQ file to obtain a FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data.
[0027] A compression unit is used to obtain the row sequence identifier of each sequence unit in the FASTQ file to be processed, and to compress the row sequence identifier;
[0028] The deletion unit is used to delete the FASTQ file to be processed.
[0029] Optionally, the device further includes:
[0030] The restoration unit is used to find the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file, so as to restore the FASTQ file to be processed.
[0031] Optionally, the compression unit includes:
[0032] The acquisition module is used to acquire the sequence identifier of the first line of each sequence unit in the FASTQ file to be processed;
[0033] The compression module is used to compress the sequence identifier of the first line.
[0034] Optionally, the restoration unit includes:
[0035] The first storage module is used to read the compressed sequence identifiers sequentially and store them in a preset data set;
[0036] The second output module is used to output the compressed sequence identifiers and corresponding sequence units contained in the preset data set in the original FASTQ file to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored.
[0037] Optionally, the restoration unit includes:
[0038] The second storage module is used to read the sequence identifier of the compressed first row in sequence and store the sequence identifier of the first row as the primary key in a preset data set;
[0039] The second output module is used to output the corresponding sequence units containing the sequence identifier of the first row in the preset data set in the original FASTQ file to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored. The restored FASTQ file to be processed stores sequence units containing four rows of sequences.
[0040] Optionally, the device further includes:
[0041] The deletion unit is used to obtain the row sequence identifier of each sequence unit in the FASTQ file to be processed by the compression unit, and after compressing the row sequence identifier, clear the filtered sequencing data in the FASTQ file to be processed.
[0042] Optionally, the device further includes:
[0043] The first encryption unit is used to perform hash encryption calculation on the FASTQ file to be processed after filtering the original FASTQ file to obtain the FASTQ file to be processed, so as to obtain the encrypted hash value.
[0044] Optionally, the device may further include:
[0045] The second encryption unit is used to perform encryption calculations on the restored FASTQ file to be processed using the preset encryption algorithm to obtain the second encryption key.
[0046] The determining unit is used to determine whether there is any loss in the restoration process of the FASTQ data to be processed based on whether the first encryption key and the second encryption key are consistent.
[0047] According to a third aspect of this application, an electronic device is provided, comprising:
[0048] At least one processor; and
[0049] A memory communicatively connected to the at least one processor; wherein,
[0050] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.
[0051] According to a fourth aspect of this application, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect above.
[0052] According to a fifth aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect above.
[0053] The method, apparatus, electronic device, and storage medium for processing FASTQ data provided in this application filter the original FASTQ file to obtain a FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data. The method obtains the row sequence identifier of each sequence unit in the FASTQ file to be processed, compresses the row sequence identifier, and deletes the FASTQ file to be processed. Compared with related technologies, the embodiments of this application only compress the row sequence identifier of each sequence unit in the filtered FASTQ file to be processed, rather than compressing the entire sequence unit, thus reducing the space occupied by the FASTQ file to be processed. When using the compressed FASTQ file to be processed, it can be restored and used by means of the compressed row sequence identifier and the original sequencing data. The process is simple and easy to operate.
[0054] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description
[0055] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein:
[0056] Figure 1 A flowchart illustrating a method for processing FASTQ data provided in an embodiment of this application;
[0057] Figure 2 A schematic diagram illustrating the format of a FASTQ file to be processed, provided for an embodiment of this application;
[0058] Figure 3 A flowchart illustrating another method for processing FASTQ data provided in an embodiment of this application;
[0059] Figure 4 A flowchart illustrating another method for processing FASTQ data provided in an embodiment of this application;
[0060] Figure 5 A flowchart illustrating another method for processing FASTQ data provided in an embodiment of this application;
[0061] Figure 6 A flowchart illustrating another method for processing FASTQ data provided in an embodiment of this application;
[0062] Figure 7 A schematic diagram illustrating a row sequence identifier for the first row of a sequence unit provided in an embodiment of this application;
[0063] Figure 8 A schematic diagram of a restored FASTQ file to be processed provided in an embodiment of this application;
[0064] Figure 9 A flowchart illustrating another method for processing FASTQ data provided in an embodiment of this application;
[0065] Figure 10 A schematic diagram of a FASTQ data processing apparatus provided in an embodiment of this application;
[0066] Figure 11 A schematic diagram of another FASTQ data processing apparatus provided in this application embodiment;
[0067] Figure 12 This is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of this application. Detailed Implementation
[0068] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0069] The following description, with reference to the accompanying drawings, outlines a method, apparatus, electronic device, and storage medium for processing FASTQ data according to embodiments of this application.
[0070] Figure 1 This is a flowchart illustrating a method for processing FASTQ data provided in an embodiment of this application.
[0071] like Figure 1 As shown, the method includes the following steps:
[0072] Step 101: Filter the original FASTQ file to obtain the FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data.
[0073] In practical applications, raw FASTQ data can be obtained after sequencing. In this embodiment, the raw FASTQ data is referred to as raw data. In order to initially reduce the storage space occupied by the raw FASTQ data, the raw FASTQ data that may have sequencing errors or low sequencing quality values during the sequencing process is filtered to obtain high-quality FASTQ files to be processed. In this embodiment, the FASTQ files to be processed are referred to as clean data, which is a subset of raw data.
[0074] To facilitate better understanding, the format of the FASTQ file to be processed is described in the embodiments of this application. Figure 2 This is a schematic diagram illustrating the format of a FASTQ file to be processed, provided in an embodiment of this application. The FASTQ file to be processed is a standard file storing raw sequencing data, with each four lines forming an independent sequence unit (or sequence storage unit), such as... Figure 2 As shown, each four rows represent one sequence unit, with a total of four sequence units. The preset data format for each sequence unit is as follows:
[0075] The first line contains the sequence identifier and related descriptive information, starting with '@', which is a unique identifier for each sequence unit;
[0076] The second line is a sequence consisting of A, C, G, T and N, where A, C, G and T are base information and N is a filler code used to replace the sequence in case of sequencing failure.
[0077] The third line begins with '+', followed by a line sequence identifier, related descriptive information, or nothing at all. In the example shown in this embodiment of the invention, the data in this line only contains "+".
[0078] The fourth line contains the sequence quality information, which corresponds one-to-one with the bases in the second line of the sequence. Each base corresponds to a quality value, which is represented by ASCII code and is used to measure the reliability of the sequenced base. The higher the quality value, the more reliable it is.
[0079] It should be noted that, Figure 2 This is merely an illustrative example; the embodiments of this application do not limit the number of sequence units in the FASTQ file to be processed, nor the content of each line in each sequence unit. Figure 2 This explanation is provided solely to facilitate a better understanding of the format.
[0080] Step 102: Obtain the row sequence identifier of each sequence unit in the FASTQ file to be processed, compress the row sequence identifier, and delete the FASTQ file to be processed.
[0081] The detailed description of the sequence unit in step 101 is as follows: During compression, the line sequence identifier in the sequence unit of the FASTQ file to be processed is read and compression is performed. The compressed file contains only the identifier information of the sequence unit.
[0082] As one feasible approach in this application embodiment, at least one row of sequence unit identifiers in the sequence unit can be read for compression. For example, the row sequence identifiers of four rows of the sequence can be read for compression, or the row sequence identifiers of the first three rows of the sequence can be read for compression. This application embodiment does not limit the number of row sequence identifiers selected.
[0083] As one feasible approach in this application embodiment, compression can be performed by reading one row of sequence unit identifiers from the sequence unit. For example, compression can be performed by reading the first row of row sequence identifiers, or compression can be performed by reading the second row of row sequence identifiers. It should be noted that if the third row only contains '+' without any descriptive information, it cannot be read and compressed. This application embodiment does not limit the specific row sequence selected for the row sequence identifiers.
[0084] In this embodiment of the application, after compressing the FASTQ file to be processed, the FASTQ file to be processed is deleted to save storage space.
[0085] The method for processing FASTQ data provided in this application filters the original FASTQ file to obtain a FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data. The method obtains the row sequence identifier of each sequence unit in the FASTQ file to be processed, compresses the row sequence identifier, and then deletes the FASTQ file to be processed. Compared with related technologies, this embodiment only compresses the row sequence identifier of each sequence unit in the filtered FASTQ file to be processed, rather than compressing the entire sequence unit, thus reducing the space occupied by the FASTQ file to be processed. When using the compressed FASTQ file to be processed, it can be restored and used by means of the compressed row sequence identifier and the original sequencing data. The process is simple and easy to operate.
[0086] In one possible implementation of this embodiment, such as Figure 3 As shown, another method for processing FASTQ data is also provided, including:
[0087] Step 201: Filter the original FASTQ file to obtain the FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data.
[0088] Step 202: Obtain the row sequence identifier of each sequence unit in the FASTQ file to be processed, compress the row sequence identifier, and delete the FASTQ file to be processed.
[0089] For a description of steps 201 and 202, please refer to the detailed description of steps 101 and 102 above. The embodiments of this application will not be repeated here.
[0090] Step 203: Locate the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file to restore the FASTQ file to be processed.
[0091] When a FASTQ file to be processed is needed, the compressed line sequence identifiers are read, and the original sequence units corresponding to the compressed line sequence identifiers are output from the original FASTQ file and recorded in a blank FASTQ file to be processed, thus restoring the FASTQ file to be processed. When using the FASTQ file to be processed, the compressed sequence identifiers can be retrieved from the original FASTQ file to find the corresponding sequence units. This not only reduces the space occupied by the FASTQ file to be processed, but also simplifies the restoration process.
[0092] The method for processing FASTQ data provided in this application filters the original FASTQ file to obtain a FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data. The method obtains the row sequence identifier of each sequence unit in the FASTQ file to be processed, compresses the row sequence identifier, deletes the FASTQ file to be processed, and searches for the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file to restore the FASTQ file to be processed. Compared with related technologies, the embodiments of this application only compress the row sequence identifier of each sequence unit in the filtered FASTQ file to be processed, rather than compressing the entire sequence unit, thus reducing the space occupied by the FASTQ file to be processed. When using the FASTQ file to be processed, the compressed sequence identifier can be found in the original FASTQ file to complete the process. This not only reduces the space occupied by the FASTQ file to be processed, but also makes the restoration process of the FASTQ file to be processed simple and easy to operate.
[0093] In one possible implementation of this embodiment, such as Figure 4 As shown, another method for processing FASTQ data is also provided, including:
[0094] Step 301: Filter the original FASTQ file to obtain the FASTQ file to be processed. The original FASTQ file is used to store the sequence units of the original sequencing data.
[0095] After sequencing, raw FASTQ data can be obtained. In this embodiment, the raw FASTQ data is referred to as Faw data. In order to initially reduce the storage space occupied by the raw FASTQ data, the raw FASTQ data that may have sequencing errors or low sequencing quality values during the sequencing process is filtered to obtain high-quality FASTQ files to be processed. In this embodiment, the FASTQ files to be processed are referred to as clean data, which is a subset of raw data.
[0096] Step 302: Obtain the sequence identifier of the first line of each sequence unit in the FASTQ file to be processed, and compress the sequence identifier of the first line.
[0097] During compression, the sequence identifier of the first line of each sequence unit in the FASTQ file to be processed is read and compression is performed. In this embodiment, the compressed file is called Row1_ID.gz, and the compressed file contains only the identifier information of the first line of sequence.
[0098] In this embodiment of the application, after compressing the FASTQ file to be processed, the FASTQ file to be processed is deleted to save storage space.
[0099] Step 303: Locate the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file to restore the FASTQ file to be processed.
[0100] When the FASTQ file to be processed needs to be used, the compressed line sequence identifier is read, and the original sequence unit corresponding to the compressed line sequence identifier is output from the original FASTQ file and recorded in the blank FASTQ file to be processed, thereby realizing the restoration of the FASTQ file to be processed.
[0101] This application provides another method for processing FASTQ data. Figure 5 This is a flowchart illustrating another method for processing FASTQ data provided in an embodiment of this application. This method for processing FASTQ data can be executed alone, or it can be executed together with any embodiment of this application or any possible implementation in the embodiment, or it can be executed together with any technical solution in related technologies.
[0102] Step 401: Filter the original FASTQ file to obtain the FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data.
[0103] In practical applications, raw FASTQ data can be obtained after sequencing. In this embodiment, the raw FASTQ data is referred to as raw data. In order to initially reduce the storage space occupied by the raw FASTQ data, the raw FASTQ data that may have sequencing errors or low sequencing quality values during the sequencing process is filtered to obtain high-quality FASTQ files to be processed. In this embodiment, the FASTQ files to be processed are referred to as clean data, which is a subset of raw data.
[0104] Step 402: Obtain the row sequence identifier of each sequence unit in the FASTQ file to be processed, compress the row sequence identifier, and delete the FASTQ file to be processed.
[0105] During compression, the line sequence identifiers in the sequence units of the FASTQ file to be processed are read and compression is performed. The compressed file contains only the identifier information of the sequence units.
[0106] As one feasible approach in this application embodiment, at least one row of sequence unit identifiers in the sequence unit can be read for compression. For example, the row sequence identifiers of four rows of the sequence can be read for compression, or the row sequence identifiers of the first three rows of the sequence can be read for compression. This application embodiment does not limit the number of row sequence identifiers selected.
[0107] As one feasible approach in this application embodiment, compression can be performed by reading one row of sequence unit identifiers from the sequence unit. For example, compression can be performed by reading the first row of row sequence identifiers, or compression can be performed by reading the second row of row sequence identifiers. It should be noted that if the third row only contains '+' without any descriptive information, it cannot be read and compressed. This application embodiment does not limit the specific row sequence selected for the row sequence identifiers.
[0108] In this embodiment of the application, after compressing the FASTQ file to be processed, the FASTQ file to be processed is deleted to save storage space.
[0109] Step 403: Read the compressed sequence identifiers sequentially and store them in a preset data set.
[0110] Read the sequence identifier of the FASTQ file to be processed into a preset dataset. For example, the preset dataset is illustrated by using a hash table. The compressed sequence identifier can be read and stored in the hash table as the primary key.
[0111] Step 404: Output the compressed sequence identifiers and corresponding sequence units contained in the preset data set in the original FASTQ file to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored.
[0112] The sequence units containing the preset data set (hash table) in the original FASTQ file are output to obtain the restored FASTQ file to be processed.
[0113] This application provides another method for processing FASTQ data. Figure 6 This is a flowchart illustrating another method for processing FASTQ data provided in an embodiment of this application. This method can be executed alone, or in combination with any embodiment of this application or possible implementations within those embodiments, or in combination with any technical solution in related technologies. It includes:
[0114] Step 501: Filter the original FASTQ file to obtain the FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data.
[0115] In practical applications, raw FASTQ data can be obtained after sequencing. In this embodiment, the raw FASTQ data is referred to as raw data. In order to initially reduce the storage space occupied by the raw FASTQ data, the raw FASTQ data that may have sequencing errors or low sequencing quality values during the sequencing process is filtered to obtain high-quality FASTQ files to be processed. In this embodiment, the FASTQ files to be processed are referred to as clean data, which is a subset of raw data.
[0116] Step 502: Obtain the sequence identifier of the first line of each sequence unit in the FASTQ file to be processed, compress the sequence identifier of the first line, and delete the FASTQ file to be processed.
[0117] During compression, the line sequence identifiers in the sequence units of the FASTQ file to be processed are read and compression is performed. The compressed file contains only the identifier information of the sequence units.
[0118] As one feasible approach in this application embodiment, compression can be performed by reading one row of sequence unit identifiers from the sequence unit. For example, compression can be performed by reading the first row of row sequence identifiers, or compression can be performed by reading the second row of row sequence identifiers. It should be noted that if the third row only contains '+' without any descriptive information, it cannot be read and compressed. This application embodiment does not limit the specific row sequence selected for the row sequence identifiers.
[0119] In this embodiment of the application, after compressing the FASTQ file to be processed, the FASTQ file to be processed is deleted to save storage space.
[0120] Step 503: Read the sequence identifier of the compressed first row in sequence, and store the sequence identifier of the first row as the primary key in a preset data set.
[0121] The sequence identifier of the first line of the FASTQ file to be processed is read into a preset dataset. For example, the preset dataset is illustrated by a hash table. The read compressed sequence identifier can be stored in the hash table as the primary key.
[0122] Step 504: Output the sequence units corresponding to the sequence identifiers of the first row in the preset data set in the original FASTQ file to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored. The restored FASTQ file to be processed stores sequence units containing four rows of sequences.
[0123] For ease of understanding, exemplified as follows: Figure 7 As shown, Figure 7 This is a schematic diagram of a row sequence identifier for the first row of a sequence unit provided in an embodiment of this application, combined with... Figure 2 After compressing the first line of sequence identifiers, we get Figure 7 The two first-row sequence identifiers shown are stored as primary keys in a hash table. If the raw data of the original FASTQ file contains sequence units from the hash table, it is output, thus obtaining the first 8 rows of information from the original FASTQ file. This yields the restored FASTQ file to be processed, as shown in the figure. Figure 8 As shown.
[0124] This application provides another method for processing FASTQ data. Figure 9 This is a flowchart illustrating another method for processing FASTQ data provided in this application. This method can be executed alone, or in conjunction with any embodiment of this application or possible implementations within those embodiments, or in conjunction with any technical solution in related technologies. The method includes:
[0125] Step 601: Filter the original FASTQ file to obtain the FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data.
[0126] Step 602: Calculate the first encryption key by performing a preset encryption algorithm on the FASTQ file to be processed.
[0127] To verify the reduction in storage space of the FASTQ data clean data to be processed brought about by the processing of FASTQ data described in the embodiments of this application, an example of storing a 5.25Gb clean data sequence in FASTQ format is provided below.
[0128] Example 1: Sample W was sequenced using a sequencer to obtain a raw FASTQ sequence of 5.54 Gb size, i.e., raw data, named "W.fq.gz". W.fq.gz was filtered using preset filtering software, and sequences with low quality values were removed, resulting in a 5.25 Gb FASTQ file to be processed, named "W.clean.fq.gz". The MD5 value of the decompressed file "W.clean.fq" is "747d1e0ee01dd591c0f4b08f06bf52ce". This MD5 value can be used to determine whether the storage method of this embodiment of the invention is used. After restoration using Example 2 of the invention, the MD5 value of the clean data file of the FASTQ file to be processed is consistent, thereby verifying the reliability of the method described in this application.
[0129] The first line sequence identifier of the sequence unit in the file "W.clean.fq" is output to the file Row1_ID and compressed to obtain the final storage file "Row1_ID.gz" of the clean data of the FASTQ file to be processed. The original FASTQ file "W.clean.fq.gz" is then deleted. The size of "Row1_ID.gz" is 0.12Gb, which is 2.29% of the size of the file W.clean.fq.gz, achieving a significant reduction in the storage space of the clean data of the FASTQ file to be processed. In this embodiment, the reading and reduction of the clean data of the FASTQ file to be processed is implemented using the Per1 programming language, and the specific implementation code is as follows:
[0130]
[0131]
[0132] Step 603: Obtain the row sequence identifier of each sequence unit in the FASTQ file to be processed, compress the row sequence identifier, and delete the FASTQ file to be processed;
[0133] Step 604: Locate the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file to restore the FASTQ file to be processed.
[0134] Step 605: The restored FASTQ file to be processed is encrypted using the preset encryption algorithm to obtain the second encryption key.
[0135] Step 606: Determine whether there is any loss in the restoration process of the FASTQ data to be processed based on whether the first encryption key and the second encryption key are consistent.
[0136] Example 2 presents the 0.12Gb "Row1_ID.gz" obtained in Example 1 above, combined with the "W.fq.gz" obtained in Example 1. This step is used to verify the correctness of restoring the FASTQ file to be processed.
[0137] Read the sequence identifier file "Row1_ID.gz" into a preset dataset (hash table), store each line of information in "Row1_ID.gz" as the primary key in the hash table, and output the sequence units in "W.fq.gz" that contain the primary key of the hash table to obtain the restored FASTQ file clean data, i.e., "W.clean.fq.gz".
[0138] In this embodiment, the data restoration of the clean data in the FASTQ file to be processed is implemented using the Per1 programming language, and the specific implementation code is as follows:
[0139]
[0140]
[0141] Furthermore, after decompressing the file "W.clean.fq.gz", we obtained the file "W.clean.fq" and used the command "md5sum W.clean.fq" to obtain its MD5 value as "747d1e0ee01dd591c0f4b08f06bf52ce", which is the same as the MD5 value in Example 1. Therefore, it is proven that the method described in the embodiments of this application is effective and feasible.
[0142] In summary, the original FASTQ file is filtered to obtain a FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data. The row sequence identifier of each sequence unit in the FASTQ file to be processed is obtained, and the row sequence identifier is compressed. The FASTQ file to be processed is then deleted. The sequence unit corresponding to the compressed sequence identifier is found in the original FASTQ file to restore the FASTQ file to be processed. Compared with related technologies, this embodiment only compresses the row sequence identifier of each sequence unit in the filtered FASTQ file to be processed, rather than compressing the entire sequence unit. This reduces the space occupied by the FASTQ file to be processed. When using the FASTQ file to be processed, the compressed sequence identifier can be found in the original FASTQ file to complete the process. This not only reduces the space occupied by the FASTQ file to be processed, but also makes the restoration process of the FASTQ file simple and easy to operate.
[0143] Corresponding to the above-described method for processing FASTQ data, this application also proposes a device for processing FASTQ data. Since the device embodiment of this application corresponds to the method embodiment described above, details not disclosed in the device embodiment can be referred to the method embodiment described above, and will not be repeated in this application.
[0144] Figure 10 A schematic diagram of a FASTQ data processing apparatus provided in an embodiment of this application is shown below. Figure 10 As shown, it includes:
[0145] Processing unit 71 is used to filter the original FASTQ file to obtain a FASTQ file to be processed, wherein the original FASTQ file is used to store sequence units of the original sequencing data;
[0146] Compression unit 72 is used to obtain the row sequence identifier of each sequence unit in the FASTQ file to be processed, and to compress the row sequence identifier;
[0147] Deletion unit 73 is used to delete the FASTQ file to be processed;
[0148] The restoration unit 74 is used to find the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file, so as to restore the FASTQ file to be processed.
[0149] The FASTQ data processing apparatus provided in this application filters the original FASTQ file to obtain a FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data. The apparatus obtains the row sequence identifier of each sequence unit in the FASTQ file to be processed, compresses the row sequence identifier, and deletes the FASTQ file to be processed. Compared with related technologies, the embodiments of this application only compress the row sequence identifier of each sequence unit in the filtered FASTQ file to be processed, rather than compressing the entire sequence unit, thus reducing the space occupied by the FASTQ file to be processed.
[0150] Furthermore, in one possible implementation of this embodiment, such as Figure 11 As shown, the compression unit 72 includes:
[0151] The acquisition module 721 is used to acquire the sequence identifier of the first line of each sequence unit in the FASTQ file to be processed;
[0152] Compression module 722 is used to compress the sequence identifier of the first line.
[0153] Furthermore, in one possible implementation of this embodiment, such as Figure 11 As shown, the restoration unit 74 includes:
[0154] The first storage module 741 is used to read the compressed sequence identifiers sequentially and store them in a preset data set;
[0155] The second output module 742 is used to output the compressed sequence identifier and the corresponding sequence unit contained in the preset data set in the original FASTQ file to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored.
[0156] Furthermore, in one possible implementation of this embodiment, such as Figure 11 As shown, the restoration unit 74 includes:
[0157] The second storage module 743 is used to read the sequence identifier of the compressed first row in sequence and store the sequence identifier of the first row as the primary key in a preset data set;
[0158] The second output module 744 is used to output the corresponding sequence units containing the sequence identifier of the first row in the preset data set in the original FASTQ file to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored. The restored FASTQ file to be processed stores sequence units containing four rows of sequences.
[0159] Furthermore, in one possible implementation of this embodiment, such as Figure 11 As shown, the device further includes:
[0160] The first encryption unit 75 is used to perform hash encryption calculation on the FASTQ file to be processed after filtering the original FASTQ file to obtain the FASTQ file to be processed, so as to obtain the encrypted hash value.
[0161] Furthermore, in one possible implementation of this embodiment, such as Figure 11 As shown, the device further includes:
[0162] The second encryption unit 76 is used to perform encryption calculations on the restored FASTQ file to be processed using the preset encryption algorithm to obtain the second encryption key;
[0163] The determining unit 77 is used to determine whether there is any loss in the restoration process of the FASTQ data to be processed based on whether the first encryption key and the second encryption key are consistent.
[0164] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.
[0165] Figure 12 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.
[0166] like Figure 12 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 802 or a computer program loaded from storage unit 808 into RAM (Random Access Memory) 803. RAM 803 can also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. I / O (Input / Output) interface 805 is also connected to bus 804.
[0167] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0168] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as methods for processing FASTQ data. For example, in some embodiments, the method for processing FASTQ data may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the aforementioned method of processing FASTQ data by any other suitable means (e.g., by means of firmware).
[0169] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0170] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0171] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0172] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0173] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.
[0174] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.
[0175] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.
[0176] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.
[0177] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for processing FASTQ data, characterized in that, include: The original FASTQ file is filtered to obtain a FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data. Obtain the row sequence identifier of each sequence unit in the FASTQ file to be processed, compress the row sequence identifier, and delete the FASTQ file to be processed.
2. The method according to claim 1, characterized in that, The method further includes: The sequence unit corresponding to the compressed sequence identifier is located in the original FASTQ file to restore the FASTQ file to be processed.
3. The method according to claim 1 or 2, characterized in that, The step of obtaining the row sequence identifier of each sequence unit in the FASTQ file to be processed and compressing the row sequence identifier includes: Obtain the sequence identifier of the first line of each sequence unit in the FASTQ file to be processed, and compress the sequence identifier of the first line.
4. The method according to claim 2, characterized in that, The step of searching for the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file includes: The compressed sequence identifiers are read sequentially and stored in a preset data set; The compressed sequence identifiers and corresponding sequence units contained in the preset data set in the original FASTQ file are output to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored.
5. The method according to claim 3, characterized in that, The step of searching for the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file includes: The sequence identifiers of the compressed first row are read sequentially, and the sequence identifiers of the first row are stored as the primary key in a preset data set; The sequence units corresponding to the sequence identifiers of the first row in the preset data set in the original FASTQ file are output to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored. The restored FASTQ file to be processed stores sequence units containing four rows of sequences.
6. The method according to any one of claims 1-5, characterized in that, After filtering the original FASTQ file to obtain the FASTQ file to be processed, the method further includes: The first encryption key is obtained by calculating the FASTQ file to be processed according to a preset encryption algorithm.
7. The method according to claim 6, characterized in that, After retrieving the sequence unit corresponding to the compressed sequence identifier from the original FASTQ file to restore the FASTQ file to be processed, the method further includes: The pre-set encryption algorithm is used to encrypt the restored FASTQ file to be processed to obtain a second encryption key; Based on whether the first encryption key and the second encryption key are consistent, it is determined whether there is any loss in the restoration process of the FASTQ data to be processed.
8. A FASTQ data processing apparatus, characterized in that, include: The processing unit is used to filter the original FASTQ file to obtain a FASTQ file to be processed. The original FASTQ file is used to store sequence units of the original sequencing data. A compression unit is used to obtain the row sequence identifier of each sequence unit in the FASTQ file to be processed, and to compress the row sequence identifier; The deletion unit is used to delete the FASTQ file to be processed.
9. The apparatus according to claim 8, characterized in that, The device further includes: The restoration unit is used to find the sequence unit corresponding to the compressed sequence identifier in the original FASTQ file, so as to restore the FASTQ file to be processed.
10. The apparatus according to claim 8 or 9, characterized in that, The compression unit includes: The acquisition module is used to acquire the sequence identifier of the first line of each sequence unit in the FASTQ file to be processed; The compression module is used to compress the sequence identifier of the first line.
11. The apparatus according to claim 9, characterized in that, The reduction unit includes: The first storage module is used to read the compressed sequence identifiers sequentially and store them in a preset data set; The second output module is used to output the compressed sequence identifiers and corresponding sequence units contained in the preset data set in the original FASTQ file to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored.
12. The apparatus according to claim 10, characterized in that, The reduction unit includes: The second storage module is used to read the sequence identifier of the compressed first row in sequence and store the sequence identifier of the first row as the primary key in a preset data set; The second output module is used to output the corresponding sequence units containing the sequence identifier of the first row in the preset data set in the original FASTQ file to the FASTQ file to be processed, so that the FASTQ file to be processed can be restored. The restored FASTQ file to be processed stores sequence units containing four rows of sequences.
13. The apparatus according to any one of claims 8-12, characterized in that, The device further includes: The first encryption unit is used to perform hash encryption calculation on the FASTQ file to be processed after filtering the original FASTQ file to obtain the FASTQ file to be processed, so as to obtain the encrypted hash value.
14. The apparatus according to claim 13, characterized in that, The device also includes: The second encryption unit is used to perform encryption calculations on the restored FASTQ file to be processed using the preset encryption algorithm to obtain the second encryption key. The determining unit is used to determine whether there is any loss in the restoration process of the FASTQ data to be processed based on whether the first encryption key and the second encryption key are consistent.
15. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.
16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.
17. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.