A method for sparsifying a quality score sequence of a sequencing FASTQ file

Abstract

The application discloses a kind of sequencing FASTQ file quality score sequence's sparse processing method, comprising: S1, extracting quality score data stream from FASTQ file;S2, the quality score data stream to be processed is divided into multiple segments, each segment contains n characters;S3, set the sliding window size of similarity calculation to m characters, every m characters is divided into a segment, and similarity measurement between the segments is carried out;S4, set threshold e, when the average similarity result after similarity measurement is greater than or equal to e, two segments are replaced with each other, and the replaced quality score data stream is obtained.The application can reduce the complexity of quality score data stream by sparse preprocessing, thereby reducing the compression rate of FASTQ file.

Description

Technical Field

[0001] This invention relates to the field of lossy data compression technology, and more specifically, to a method for sparsification of quality fraction sequences in FASTQ sequencing files. Background Technology

[0002] Research on genes is a core area of ​​study in biomedicine and bioinformatics, encompassing fields such as biomedicine, food safety, and genetics. Gene sequences contain crucial information carrying genetic material, making the complete preservation of gene sequence structures and the extraction of key information for scientific research of paramount importance.

[0003] In recent years, with the continuous development of high-throughput sequencing technology, more genetic information has been revealed. While the technology is constantly being updated and iterated, sequencing prices are also continuously decreasing. The decline in sequencing costs has long exceeded the rate of decline predicted by Moore's Law. This reduction in sequencing costs and the development of sequencing technology have led to a surge in sequencing data. However, general-purpose compressors such as Gzip cannot reduce the redundant information in FASTQ files to a reasonable level, thus requiring specialized compressors to compress FASTQ data.

[0004] Raw high-throughput sequencing data is primarily stored in FASTQ format, consisting of three data streams (ID stream, base stream, and quality fraction stream). The quality fraction stream suffers from poor lossless compression due to its large alphabet and lack of clear character patterns. However, research indicates that lossy compression of the quality fraction stream does not affect the downstream analysis and use of FASTQ data. Therefore, developing a lossy compression method for the quality fraction stream aims to compress FASTQ data as much as possible without compromising its usability, thereby reducing storage and transmission costs.

[0005] Specifically, FASTQ files consist of short reads, in the following format:

[0006]

[0007] The first line is a unique identifier for each short read, starting with the character "@", and contains sequencing status information such as the sequencer model;

[0008] The second line is the DNA sequence obtained, consisting of five letters (A, G, C, T, N) representing different bases, where the letter N represents a base that cannot be recognized by the sequencer.

[0009] The third line begins with the symbol "+" and usually contains no other information in order to save space;

[0010] The fourth line is the quality score, the same length as the second line. Each character corresponds to the reliability of the bases measured in the second line, also known as the confidence level, and is represented by ASCII code. It is generally represented by 40 similar characters on the ASCII table, and the quality score values ​​in the same region tend to be similar. Summary of the Invention

[0011] The purpose of this invention is to provide a sparsity processing method for the quality score sequence of FASTQ sequencing files, so as to overcome the defects of the existing technology.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0013] A method for sparsification of FASTQ sequencing file quality score sequences, comprising the following steps:

[0014] S1. Extract the quality score data stream from the FASTQ file;

[0015] S2. Divide the quality score data stream to be processed into multiple segments, each segment containing n characters;

[0016] S3. Set the sliding window size for similarity calculation to m characters, and divide each m characters into... Each segment, and perform A measure of similarity between segments;

[0017] S4. Set a threshold e, and replace two segments whose average similarity result after similarity measurement is greater than or equal to e with each other to obtain the replacement quality score data stream.

[0018] Further, step S1 specifically involves: extracting the fourth line of data from each short read in the FASTQ file to obtain a quality score data stream, and removing meaningless characters from the quality score data stream.

[0019] Furthermore, the meaningless characters in the quality score data stream are the quality score values ​​corresponding to bases that cannot be identified during the sequencing process.

[0020] Furthermore, in step S2, if the last segment has less than n characters remaining, it is not included in the traversal range of the sliding window.

[0021] Furthermore, in step S3, between every two segments, the absolute value of the difference between the two characters is calculated, starting from the first character of each other, to calculate the average similarity of each character, up to the nth character.

[0022] Furthermore, in step S3, a similarity measurement method based on cosine similarity is used, and its formula is:

[0023]

[0024]

[0025] In the formula, and These are the ASCII codes for the i-th quality score in the two segments, respectively. The denominator AreaQ is the maximum possible difference in quality scores. The parameter... Used to amplify difference signals.

[0026] Furthermore, the formula for calculating the average similarity in step S4 is as follows:

[0027]

[0028] Furthermore, the threshold e > 0.95.

[0029] Compared with the prior art, the advantages of the present invention are as follows: The present invention provides a sparsification processing method for the quality score sequence of sequencing FASTQ files. By using a similarity measure based on cosine similarity and calculating the average similarity to replace quality score fragments, the present invention can reduce the complexity of the quality score data stream through sparsification preprocessing, thereby reducing the compression rate of FASTQ files. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart of the sparsification method for the quality score sequence of FASTQ sequencing files according to the present invention.

[0032] Figure 2 This is a schematic diagram illustrating the effect of the present invention. Detailed Implementation

[0033] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0034] See Figure 1 As shown in the figure, this embodiment discloses a method for sparsification of the quality score sequence of a sequencing FASTQ file, including the following steps:

[0035] Step S1: Extract the quality score data stream from the FASTQ file.

[0036] Specifically, the quality score data stream is obtained by extracting the fourth line of each short read from the FASTQ file, and meaningless characters in the quality score data stream are removed. Meaningless characters are the quality score values ​​corresponding to bases that cannot be identified during sequencing, and are represented by "#".

[0037] In this embodiment, for example, the quality score data stream is:

[0038]

[0039] The mass fraction data stream after removing the mass fraction corresponding to the meaningless base N "#" is as follows:

[0040]

[0041] Step S2: Divide the quality score data stream to be processed into multiple segments, each segment containing n characters.

[0042] If the last segment has fewer than n characters remaining, it is not included in the traversal range of the sliding window.

[0043] In this embodiment, taking n=5 as an example, the remaining quality score data streams are divided into groups of five.

[0044]

[0045] Step S3: Set the sliding window size for similarity calculation to m characters, and divide each m characters into... Each segment, and perform A measure of similarity between segments.

[0046] This invention employs a similarity measurement method based on cosine similarity, the formula of which is:

[0047]

[0048] in:

[0049]

[0050] and These are the ASCII codes for the i-th quality score in the two segments, respectively, and AreaQ in the denominator is the maximum possible difference in quality scores.

[0051] In this embodiment, m = 20, n = 5, AreaQ = 40. For example, a similarity matrix is ​​constructed between every four segments, and the average similarity between the segments is calculated:

[0052]

[0053] The results are shown in the table below:

[0054] CCCF2(1) ADHHH(2) HHJJJ(3) IJJJJ(4) CCCF2(1) \ 0.5898 0.5060 0.5060 ADHHH(2) 0.5898 \ 0.9240 0.8900 HHJJJ(3) 0.5060 0.9240 \ 0.9940 IJJJJ(4) 0.5060 0.8900 0.9940 \

[0055] Step S4: Set a threshold e, and replace two segments whose average similarity result after similarity measurement is greater than or equal to e with each other to obtain the replaced quality score data stream.

[0056] In this embodiment, the replacement threshold e is set to 0.985, when the mean cosine similarity It can be replaced when needed.

[0057] Therefore, in the first four segments, that is, the first 20 character combinations, the segment with the quality score number 3 replaces the segment with the quality score number 4:

[0058]

[0059] Similarly, sequences

[0060]

[0061] It can be replaced with:

[0062]

[0063] To verify the effectiveness of the present invention, Figure 2 The compression results of the quality score data stream before and after processing are given. The quality score stream before processing is as follows:

[0064]

[0065] and the processed mass fraction stream

[0066]

[0067] Create two files named OriginalFile and test respectively, then compress them using the general-purpose compressor Gzip, and compare the size difference between the two compressed files.

[0068] Because the test file has more characters than the original file, the original test file was larger in size. Figure 2It can be seen that after processing, the compressed file test.zip of test is smaller than the compressed file OriginalFile.zip of OriginalFile, with the former being about 87.10% of the latter. This is only the difference before and after processing about 100 quality score data streams, indicating that the present invention can significantly reduce the compression rate of quality score data streams.

[0069] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, the patent owner may make various modifications or alterations within the scope of the appended claims, as long as they do not exceed the protection scope described in the claims of the present invention, they shall be within the protection scope of the present invention.

Claims

1. A method for sparsification of FASTQ sequencing file quality score sequences, characterized in that, Includes the following steps: S1. Extract the quality score data stream from the FASTQ file; S2, divide the mass fraction data stream to be processed into multiple segments, each segment containing n characters; S3, set the size of the sliding window for similarity computation to characters, each of which is divided into segments, and similarity measure between segments is performed; segments is performed; S4, Set threshold e The average similarity result after similarity measurement is greater than or equal to e The two segments are replaced with each other to obtain the replaced quality score data stream; In step S3, a similarity measurement method based on cosine similarity is used, and its formula is as follows: In the formula, and These are the first two segments. The ASCII encoded value of each quality score, denominator It is the maximum difference that can be obtained in the quality fraction, parameter Used to amplify difference signals. .

2. The method of claim 1, wherein the method further comprises: Step S1 specifically involves: extracting the fourth line of data from each short read in the FASTQ file to obtain a quality score data stream, and removing meaningless characters from the quality score data stream.

3. The method of claim 2, wherein the method further comprises: The meaningless characters in the quality score data stream are the quality score values ​​corresponding to bases that cannot be recognized during the sequencing process.

4. The method of claim 1, wherein, If the last segment has less than n characters, it is not counted in the sliding window traversal.

5. The method of claim 1, wherein, The absolute value of the difference between two characters is calculated from the first character of each of the two fragments, between each other, to calculate the average similarity of each character, up to the n first character.

6. The method of claim 1, wherein, The formula for calculating the average similarity in step S4 is as follows: 。 7. The method of claim 1, wherein, the threshold value e＞0.95 .

Images