Construction method and construction device of large data BWT and terminal equipment

Abstract

The invention discloses a large data BWT construction method and device and terminal equipment, and relates to the technical field of computers, and the method comprises the steps: scanning a character string T, calculating L type and S type substring precursor sequences LPre and SPre according to the ranking of S * type characters in the T, calculating the ranking of the L type substring of the T by adopting a small top heap H1, calculating the ranking of the S type substring of the T by adopting a large top heap H2, and calculating the ranking of the S type substring of the T according to the ranking of the S * type characters in the T; sorting and naming the S * type substring of the T to generate a new contracted character string T1; calculating a suffix array SA (T1) of the T1; calculating the ranking of S * type suffixes of the T according to the SA (T1), calculating L type and S type suffix precursor tuple sequences LPre and SPre according to the ranking of the S * type suffixes of the T, calculating an L type BWT sequence Lbwt by using a small top heap H1, and calculating an S type BWT sequence Sbwt by using a large top heap H2; and combining the two types of BWT sequences to generate a final BWT sequence of the character string T.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, a device and a terminal device for constructing a large-scale data BWT. Background Art

[0002] The Burrows-Wheeler Transform (BWT) is a block-ordering-based lossless data compression method proposed by Michael Burrows and David Wheeler, primarily used in the field of data compression. The BWT of a string T is a string of the same length that only changes the order of the characters in T, without changing the characters themselves. This string contains a large number of consecutively repeated characters, making it more conducive to data compression. The BWT is a reversible transform that can restore the string T itself based on its BWT. Therefore, large-scale data compression can be achieved by calculating the BWT of a large string. The efficiency of constructing the BWT for large data determines the computational efficiency of the compression process and is the key to achieving large-scale data compression.

[0003] Existing BWT construction methods first calculate the suffix array (SA), then calculate the BWT of the data based on the SA. For small data that does not exceed the device's memory capacity, the BWT can be quickly calculated based on the SA. However, for large data that exceeds the memory capacity, an external suffix array construction algorithm is used to calculate the SA. Then, a priority queue is used to perform two rounds of integer sorting on the SA and the string to calculate the BWT. Calculating the SA and integer sorting require a large number of I / O operations, which is time-consuming and a performance bottleneck for large data compression. Summary of the Invention

[0004] In response to the problems of the prior art, the present invention provides a method, apparatus and terminal device for constructing a large-scale data BWT, which can directly calculate the BWT from large-scale data without pre-calculating the SA, thereby improving the computational efficiency of large-scale data compression.

[0005] In order to achieve the above objectives, the present invention provides the following technical solutions: On the one hand, the present invention provides a method for constructing a large-scale data BWT, comprising:

[0006] Scan the input string T, calculate its L-type substring predecessor tuple sequence LPre and S-type substring predecessor tuple sequence SPre, and save them to external memory respectively;

[0007] The L-type substring predecessor tuple is a tuple consisting of the L-type predecessor character and position of the substring of T; the S-type substring predecessor tuple is a tuple consisting of the S-type predecessor character and position of the substring of T;

[0008] Scan the string T, retrieve the S* type character, and push the S* type substring tuple into the top heap H1;

[0009] The S* type substring tuple is a two-tuple consisting of the first character and position of the S* type substring of T; the S* type substring is a string intercepted by two adjacent S* type characters in T, including the starting and ending S* type characters; the small top heap H1 is a minimum priority queue, the elements of which are triples consisting of the first character, position and rank of the suffix / substring, and the sort key is the suffix, the first character and rank of the substring;

[0010] Pop out the top tuple of the small top heap H1 in sequence, read its predecessor tuple e from the external memory sequence LPre, and push e into the small top heap H1 until the LPre sequence is empty;

[0011] Pop out the top elements of the small heap H1 in sequence and push them into the large heap H2 until the small heap H1 is empty;

[0012] The max-heap H2 is a maximum priority queue, wherein the elements are triples consisting of the first character of the suffix / substring, the position and the rank, and the sorting key is the suffix, the first character of the substring and the rank;

[0013] Pop out the top tuple of the heap H2 in sequence, read its predecessor tuple e from the external memory sequence SPre, and push e into H2 until the SPre sequence is empty;

[0014] Pop out the S* type substring tuples at the top of the max-heap H2 in sequence, name the tuples, and generate the contracted string T1 according to their positions in T;

[0015] Calculate the suffix array SA(T1) of string T1, calculate the S* type suffix ranking of string T based on SA(T1), and push the S* type suffix tuple into the small top heap H1;

[0016] Scan the string T and calculate the L-type suffix predecessor tuple and S-type suffix predecessor tuple sequences LPre and SPre respectively according to the ranking of the S* type suffixes, and save them to external memory respectively;

[0017] The L-type suffix predecessor tuple is a tuple consisting of the L-type predecessor character and position of the suffix of T; the S-type suffix predecessor tuple is a tuple consisting of the S-type predecessor character and position of the suffix of T;

[0018] Pop out the top tuple of the small top heap H1 in sequence, read its predecessor tuple e from the sequence LPre, and push e into H1. At the same time, write the predecessor character e.char to the end of the L-type BWT sequence Lbwt until the sequence LPre is empty;

[0019] Pop out the top elements of the small heap H1 in sequence and push them into the large heap H2 until the small heap H1 is empty;

[0020] Pop the top tuple of the big heap H2 in sequence, read its predecessor tuple e from the sequence SPre, and push e into H2. At the same time, write the predecessor character e.char to the end of the S type BWT sequence Sbwt until the sequence SPre is empty;

[0021] Merge the L-type and S-type BWT sequences Lbwt and Sbwt to generate the final BWT sequence of the string T.

[0022] On the other hand, the present invention also provides a device for constructing a large-scale data BWT, comprising:

[0023] The substring predecessor sequence calculation module scans the string T and first calculates the L-type substring predecessor tuple sequence LPre according to the inductive sorting rule. Then, based on the L-type substring ranking, it calculates the S-type substring predecessor tuple sequence SPre and saves the LPre and SPre sequences to external memory.

[0024] The substring sorting and naming module sorts and names the S* type substrings of the string T to generate a new contracted string T1;

[0025] The suffix array calculation module calculates SA(T1) directly based on the string T1 or recursively;

[0026] S* type suffix calculation module, calculates the ranking of the S* type suffix of string T according to SA(T1);

[0027] The suffix-precursor sequence calculation module calculates the L-type suffix-precursor tuple sequence LPre and the S-type suffix-precursor tuple sequence SPre according to the ranking of the S*-type suffixes of the string T;

[0028] The L-type BWT calculation module scans the L-type suffix predecessor tuple sequence LPre and uses the small top heap H1 to calculate the L-type BWT sequence Lbwt until the sequence LPre is empty;

[0029] The S-type BWT calculation module scans the S-type suffix predecessor tuple sequence SPre and uses the max-heap H2 to calculate the S-type BWT sequence Sbwt until the sequence SPre is empty.

[0030] The merging module merges the L-type BWT sequence Lbwt and the S-type BWT sequence Sbwt to generate the final BWT sequence of the string T.

[0031] On the other hand, the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a method for constructing a large data BWT as described in the first aspect above are implemented.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The terminal device proposed in the present invention uses an external memory block inductive sorting method to sort and name the S* type substrings of the string T, generate a contracted string T1, calculate the suffix array SA(T1) of T1, and then calculate the ranking of the S* type suffix of the string T based on SA(T1). Based on the ranking of the S* type suffix of T, a small top heap H1 is used to calculate the L type BWT sequence, and a large top heap H2 is used to calculate the S type BWT sequence. Finally, the two types of BWT sequences are merged to generate the final BWT sequence. In the application scenario of large-scale data compression, the large-scale data BWT construction method proposed in this embodiment can be used to directly construct the BWT from the string without calculating the suffix array of the large data, thereby improving the computational efficiency of large-scale data compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0035] Figure 1 Schematic diagram of the storage structure of the character type, suffix array and BWT sequence of T provided in an embodiment of the present application;

[0036] Figure 2 This is a schematic diagram of the storage structure of the T suffix bucket, L type and S type BWT provided in the embodiment of the present application;

[0037] Figure 3 This is a flowchart of the steps of a method for constructing a large-scale data BWT provided in an embodiment of the present application;

[0038] Figure 4 This is a schematic diagram of the structure of a large-scale data BWT construction device provided in an embodiment of the present application;

[0039] Figure 5 This is a structural diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0041] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0042] First, the technical terms that may be used in the embodiments of the present application are unified and described here.

[0043] String T: A sequence or array T[0, n - 1] composed of n characters; in particular, the character T[n - 1] = '$' is the only character that appears in T and has the smallest lexicographical order.

[0044] L - type character: If T[i] > T[i + 1], or T[i] is the same as T[i + 1] and T[i + 1] is of the L - type, 0 ≤ i ≤ n - 2, then T[i] is an L - type character.

[0045] S - type character: A character that is not of the L - type is an S - type character; in particular, T[n - 1] is defined as an S - type character.

[0046] S* - type character: If T[i] (1 ≤ i ≤ n - 1) is of the S - type and T[i - 1] is of the L - type, then T[i] is an S* - type character; in particular, T[n - 1] is defined as an S* - type character.

[0047] L* - type character: If T[i] (1 ≤ i ≤ n - 1) is of the L - type and T[i - 1] is of the S - type, then T[i] is an L* - type character. <00,00105>Substring and its type: In the string T[i, j] (0 ≤ i < j ≤ n - 1), if T[j] is of the S* - type and there is no S* - type character in T[i + 1, j - 1], then the string T[i, j] is called a substring of T, and its type is the same as the type of T[i].

[0049] Suffix and its type: The string T[i, n - 1] is called a suffix of T, and its type is the same as the type of T[i].

[0050] Predecessor character: If the predecessor character of the suffix exists for the substring starting with the character T[i], then its predecessor character is T[i - 1].

[0051] Suffix array SA: Sort the suffixes of T in lexicographical order and save the starting positions of the suffixes into an integer array. This array is the suffix array of T, denoted as SA(T).

[0052] Suffix buckets and their types: Suffixes with the same first character in SA(T) are placed together to form a suffix bucket; in a suffix bucket, L-type suffixes form an L-type bucket, and S-type suffixes form an S-type bucket, and the L-type bucket is located to the left of the S-type bucket.

[0053] BWT sequence: The BWT sequence of a string T is a reversible transformation of T, represented by bwt(T). If SA(T) is known, the calculation method of bwt(T) is: if SA[i]≠0, then bwt[i]=T[SA[i]-1], otherwise bwt[i]=T[n-1].

[0054] L-type and S-type BWT sequences: Sequences consisting of ordered precursor characters of L-type and S-type suffixes are called L-type and S-type BWT sequences, respectively, and are represented by Lbwt and Sbwt, respectively.

[0055] Taking the string T[0,12] = "bbccbbabbabb$" as an example, the specific content of each of the above terms is explained:

[0056] See also Figure 1 , shows a schematic diagram of the storage structure of the character type of T, the suffix array SA(T) and the BWT sequence bwt(t). The third row lists the types of each character of T. For example, since T

[03] >T

[04] , T

[03] is of type L; and since T

[02] =T

[03] and T

[03] is of type L, T

[02] is of type L; and since T

[01] is of type S, T

[02] is also of type L*. The fourth row lists the suffix ranking of T. The ranking of the suffix on the left is less than the ranking of the suffix on the right. The fifth row lists the BWT sequence bwt(T) of T. For example, bwt

[02] =T[SA

[02] -1]=T[(SA

[02] =06)-1]=T105]='b'; since SA

[09] =0, bwt

[09] ='$'.

[0057] See also Figure 2, shows a schematic diagram of the storage structure of the suffix buckets, bucket types, and L-type and S-type BWT sequences of SA(T). The third row lists the start and end ranges of the suffix buckets. From left to right, they are the '$' suffix bucket formed by SA

[00] , the 'a' suffix bucket formed by SA[01,02], the 'b' suffix bucket formed by SA[03,10], and the 'c' suffix bucket formed by SA[11,12]. The 4th line lists the types of suffix buckets, for example, SA[01, 02] is an S-type 'a' suffix bucket; in the 'b' suffix bucket composed of SA[03, 10], SA[03, 08] is an L-type 'b' suffix bucket composed of L-type suffixes, and SA[09, 10] is an S-type 'b' suffix bucket composed of S-type suffixes; the 5th and 6th lines list the L-type and S-type BWT sequences respectively, for example, the L-type BWT sequence Lbwt = 'bbbaaccb', which is composed of the predecessor characters of the L-type suffix, and the S-type BWT sequence Sbwt = 'bbbSb', which is composed of the predecessor characters of the S-type suffix.

[0058] The calculation framework of the embodiment of the present application: first, sort and name the S* type substrings of the string T to generate the contracted string T1, then calculate the suffix array SA(T1) of T1, and then use SA(T1) to calculate the ranking of the S* type suffixes of T, and finally, calculate the L type BWT sequence Lbwt and the S type BWT sequence Sbwt according to the ranking of the S* type suffixes of the string T, and merge Lbwt and Sbwt to generate the BWT sequence of the string T.

[0059] The technical solution of this application is described below through specific embodiments.

[0060] Reference Figure 3 , shows a schematic flow chart of the steps of a method for constructing a large-scale data BWT provided in an embodiment of the present application, which may specifically include the following steps:

[0061] S101, scan the input string T, calculate the L-type substring predecessor tuple sequence LPre and the S-type substring predecessor tuple sequence SPre, and save them to external memory;

[0062] In an embodiment of the present application, the process of calculating the L-type substring predecessor sequence LPre of the string T is as follows: (1) scanning the string T from right to left, retrieving the S*-type characters therein, and traversing the S*-type characters and L-type substrings of T in ascending order according to the rule of inductive sorting; (2) if the predecessor character of the currently traversed S*-type character or L-type substring is of L type, then constructing a tuple of the L-type predecessor character and position, and writing the tuple into the sequence LPre; (3) repeating the process until all the S*-type characters and L-type substrings of T are traversed.

[0063] In an embodiment of the present application, the process of calculating the L-type substring predecessor sequence SPre of a string T is as follows: (1) according to the rule of inductive sorting, traverse the L-type substrings and S-type substrings of T in descending order; (2) if the predecessor character of the currently traversed L-type substring or S-type substring is of type S, then construct a tuple of the S-type predecessor character and position, and write the tuple into the sequence SPre; (3) repeat the process until all L-type substrings and S-type substrings are traversed.

[0064] S102, scan the string T, retrieve the S* type character of T, and push the S* type substring into the top heap H1;

[0065] In the embodiment of the present application, the process of pushing the S* type substring into the mini-heap H1 is as follows: (1) setting the rank rank = 0, scanning the string T from right to left, and retrieving the S* type character; (2) constructing a triplet <char, rank++, index> using the retrieved S* type character char, position index, and rank rank, and pushing the triplet into the mini-heap H1; (3) continuing the scanning process until the scanning is completed.

[0066] S103 , pop the top tuple of the small top heap H1 in sequence, read its predecessor tuple e from the external memory sequence LPre, and push the predecessor tuple e into the small top heap H1 until the sequence LPre is empty;

[0067] In an embodiment of the present application, the elements in the mini-heap H1 are S* type substring tuples and L type substring tuples. The element popped out of the mini-heap H1 is the substring tuple with the smallest lexicographical order. The predecessor tuple e of the tuple is read from the external memory sequence LPre, and the predecessor tuple e = <e.char, ++rank, e.index> is constructed. Then, the predecessor tuple e is pushed into the mini-heap H1; wherein, e.char is the predecessor character in the predecessor tuple, rank is the ranking, the initial value is 0, and e.index is the position of the predecessor character in the predecessor tuple; the sorting key of the mini-heap H1 is <e.char, rank>.

[0068] S104, pop out the top elements of the small top heap H1 in sequence and push them into the large top heap H2 until the small top heap H1 is empty;

[0069] In an embodiment of the present application, the small heap H1 stores the L-type substring tuple of the string T, and the process of popping out the top tuple of the small heap H1 in sequence is to take out the L-type substrings of the string T in ascending order of the substrings; after pushing the top element of the small heap H1 into the big heap H2, the big heap H2 stores the L-type substring tuple of the string T, and the L-type substrings of the string T can be taken out in descending order of the substrings.

[0070] S105 , popping out the top tuples of the max heap H2 in sequence, reading their predecessor tuples e from the external memory sequence SPre, and pushing e into the max heap H2 until the sequence SPre is empty;

[0071] In the embodiment of the present application, the elements in the max heap H2 are L-type substring tuples and S-type substring tuples. The element popped out of the max heap H2 is the substring tuple with the largest lexicographic order. The predecessor tuple e of the tuple is read from the external memory sequence SPre to construct the predecessor tuple e=<e.char,--rank,e.index> , and then push the predecessor tuple e into the max heap H2; where e.char is the predecessor character in the predecessor tuple, rank is the ranking, the initial value is n, and e.index is the position of the predecessor character in the predecessor tuple; the sort key of the max heap H2 is<e.char,rank> .

[0072] S106. Pop out the S* type substring tuples at the top of the max heap H2 in sequence, name the tuples, and generate a new contracted string T1 based on the position of the tuples in the string T.

[0073] In an embodiment of the present application, after S105 is executed, the elements in the max heap H2 are S* type substring tuples of the string T. The S* type substrings are named according to the rankings of the S* type substrings popped out from the max heap H2 in sequence, and according to the positions of the S* type substrings in the string T, the newly named S* type substrings are used to generate a new contracted string T1 according to their positions in the string T.

[0074] S107. Calculate the suffix array SA(T1) of the string T1, calculate the ranking of the S* type suffixes in the string T according to SA(T1), and push the S* type suffix tuple of the string T into the mini-heap H1;

[0075] In an embodiment of the present application, the method for calculating the suffix array SA(T1) of the string T1 is: if the length of the character set of the string T1 is equal to the length of the string T1, then SA(T1) can be calculated directly according to the lexicographical order of the characters in the string T1; otherwise, steps S101-S106 are recursively executed until the length of the character set of the string T1 is equal to the length of the string T1.

[0076] In an embodiment of the present application, the process of calculating the ranking of the S* type suffix in the string T according to SA(T1) is: (1) calculating the position of the S* type suffix in the string T according to the string T1; (2) calculating the ranking of the S* type suffix in the string T according to SA(T1).

[0077] In an embodiment of the present application, the process of pushing the S* type suffix tuple of the string T into the mini-heap H1 is to set the ranking rank = 0, traverse the S* type suffixes of the string T in order from small to large, construct the triple of the S* type suffix <char, rank++, index> in turn, and push the tuple into H1 until all S* type suffixes of the string T are traversed.

[0078] S108. Scan the string T and calculate the L-type suffix precursor sequence LPre and the S-type suffix precursor sequence SPre according to the ranking of the S* type suffixes, and save them to external memory;

[0079] In an embodiment of the present application, the process of calculating the L-type suffix predecessor sequence LPre of a string T is as follows: (1) according to the rule of inductive sorting, traverse the S*-type suffixes and L-type suffixes of T in ascending order; (2) if the predecessor character of the currently traversed S*-type suffix or L-type suffix is ​​of type L, then construct a tuple of the L-type predecessor character and position, and write the tuple into the sequence LPre; (3) repeat the process until all the S*-type suffixes and L-type suffixes of T are traversed.

[0080] In an embodiment of the present application, the process of calculating the S-type suffix predecessor sequence SPre of a string T is as follows: (1) according to the rule of inductive sorting, traverse the L-type suffixes and S-type suffixes of T in descending order; (2) if the predecessor character of the currently traversed L-type suffix or S-type suffix is ​​of S type, then construct a tuple of the S-type predecessor character and position, and write the tuple into the sequence SPre; (3) repeat the process until all the L-type suffixes and S-type suffixes of T are traversed.

[0081] S109. Pop the top tuple of the small top heap H1 in sequence, read its predecessor tuple e from the sequence LPre, push e into the small top heap H1, and write e.char to the end of the L-type BWT sequence Lbwt until the sequence LPre is empty.

[0082] In the embodiment of the present application, the elements in the mini-heap H1 are S* type suffix tuples and L type suffix tuples, and the element popped out from the top of the mini-heap H1 is the suffix tuple with the smallest lexicographical order. The process of calculating the L type BWT sequence Lbwt is: pop out the suffix tuples at the top of the mini-heap H1 in sequence, read the predecessor tuple e of the suffix tuple from the external memory sequence LPre, construct the predecessor tuple e=<e.char,+-+rank,e.index>, push the predecessor tuple e into the mini-heap H1, and write the predecessor character e.char into the L type BWT sequence Lbwt; wherein e.char is the predecessor character in the predecessor tuple, rank is the rank, the initial value is 0, and e.index is the position of the predecessor character in the predecessor tuple; the sorting key of the mini-heap H1 is<e.char,rank> .

[0083] S110, pop out the top elements of the small top heap H1 in sequence and push them into the large top heap H2 until the small top heap H1 is empty;

[0084] In an embodiment of the present application, the small heap H1 stores the L-type suffix tuple of the string T, and the process of popping out the top tuple of the small heap H1 in sequence is to take out the L-type suffix of the string T in ascending order of the suffix; after pushing the top element of the small heap H1 into the large heap H2, the large heap H2 stores the L-type suffix tuple of the string T, and the L-type suffix of the string T can be taken out in descending order of the suffix.

[0085] S111. Pop the top tuples of the max heap H2 in sequence, read their predecessor tuples e from the sequence SPre, push e into the max heap H2, and write e.char to the end of the S-type BWT sequence Sbwt until the sequence SPre is empty.

[0086] In the embodiment of the present application, the elements in the max heap H2 are L-type suffix tuples and S-type suffix tuples. The element popped from the top of the max heap H2 is the suffix tuple with the largest lexicographic order. The process of calculating the S-type BWT sequence Lbwt is as follows: pop the top suffix tuple from the max heap H2 in sequence, read the predecessor tuple e of the suffix tuple from the external memory sequence SPre, and construct the predecessor tuple e=<e.char,--rank,e.index> , push the predecessor tuple e into the max heap H2, and write the predecessor character e.char into the S type BWT sequence Sbwt; where e.char is the predecessor character in the predecessor tuple, rank is the ranking, the initial value is n, and e.index is the position of the predecessor character in the predecessor tuple; the sort key of the max heap H2 is<e.char,rank> .

[0087] S112. Merge the L-type BWT sequence Lbwt and the S-type BWT sequence Sbwt of the string T to generate the final BWT sequence of the string T.

[0088] In an embodiment of the present application, the merging process of the L-type BWT sequence Lbwt and the S-type BWT sequence Sbwt of the string T is to scan the string T once, count the number of L and S-type suffixes in each suffix bucket, and then traverse Lbwt and Sbwt in ascending order. According to the number of L and S-type suffixes in each suffix bucket, the corresponding L-type and S-type BWT characters belonging to each bucket in Lbwt and Sbwt are written into the final BWT sequence of the string T in sequence.

[0089] Reference Figure 4 , shows a schematic diagram of the structure of a large-scale data BWT construction device provided in an embodiment of the present application, which may specifically include the following modules:

[0090] The substring predecessor sequence calculation module scans the string T and first calculates the L-type substring predecessor tuple sequence LPre according to the inductive sorting rule. Then, based on the L-type substring ranking, it calculates the S-type substring predecessor tuple sequence SPre and saves the LPre and SPre sequences to external memory.

[0091] The substring sorting and naming module sorts and names the S* type substrings to generate a new contracted string T1;

[0092] The suffix array calculation module calculates SA(T1) directly based on the string T1 or recursively;

[0093] S* type suffix calculation module, calculates the ranking of the S* type suffix of string T according to SA(T1);

[0094] The suffix-precursor sequence calculation module calculates the L-type suffix-precursor tuple sequence LPre and the S-type suffix-precursor tuple sequence SPre according to the ranking of the S*-type suffixes of the string T;

[0095] The L-type BWT calculation module scans the L-type suffix predecessor tuple sequence LPre and uses the small top heap H1 to calculate the L-type BWT sequence Lbwt until the sequence LPre is empty;

[0096] The S-type BWT calculation module scans the S-type suffix predecessor tuple sequence SPre and uses the max-heap H2 to calculate the S-type BWT sequence Sbwt until the sequence SPre is empty.

[0097] The merging module merges the L-type BWT sequence Lbwt and the S-type BWT sequence Sbwt to generate the final BWT sequence of the string T.

[0098] Reference Figure 5, an embodiment of the present application also discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for constructing a large data BWT as described in the aforementioned embodiments.

[0099] The terminal device 300 may be a terminal device in the aforementioned embodiments, and may be a computing device such as a desktop computer, a cloud server, etc. The terminal device 300 may include, but is not limited to, a processor 310 and a memory 320. It will be understood by those skilled in the art that Figure 5 It is only an example of the terminal device 300 and does not constitute a limitation of the terminal device 300. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 300 may also include input and output devices, network access devices, buses, etc.

[0100] The processor 310 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0101] The memory 320 may be an internal storage unit of the terminal device 300, such as a memory of the terminal device 300. The memory 320 may also be a pluggable external storage device of the terminal device 300, such as a pluggable hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 300. Furthermore, the memory 320 may include both an internal storage unit of the terminal device 300 and an external storage device. The memory 320 is used to store the computer program 321 and other programs and data required by the terminal device 300. The memory 320 may also be used to temporarily or permanently store data that has been output or is to be output.

[0102] The terms used in the drawings to describe positional relationships are for illustrative purposes only and should not be construed as limiting this patent;

[0103] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. A person skilled in the art would be able to make other variations or modifications based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A method for constructing a large data BWT, characterized in that: include: Scan the input string T, calculate the L-type substring predecessor tuple sequence LPre and the s-type substring predecessor tuple sequence SPre, and save them to external memory respectively; The process of calculating the L-type substring predecessor sequence LPre of the string T is as follows: (1) scanning the string T from right to left, retrieving the s*-type characters therein, and traversing the s*-type characters and L-type substrings of T in ascending order according to the rule of inductive sorting; (2) if the predecessor character of the currently traversed S*-type character or L-type substring is of L type, then constructing a tuple of the L-type predecessor character and position, and writing the tuple into the sequence LPre; (3) repeating the process until all the s*-type characters and L-type substrings of T are traversed; The process of calculating the s-type substring predecessor sequence SPre of the string T is as follows: (1) according to the rule of inductive sorting, traverse the L-type substring and the s-type substring of T in descending order; (2) if the predecessor character of the currently traversed L-type substring or s-type substring is of type s, then construct a tuple of the s-type predecessor character and position, and write the tuple into the sequence SPre; (3) repeat the process until all L-type substrings and s-type substrings are traversed; Scan the string T, retrieve the s* type character, and push the s* type substring tuple into the small top heap H1; Pop out the top tuple of the small top heap H1 in sequence, read its predecessor tuple e from the external memory sequence LPre, and push e into the small top heap H1 until the LPre sequence is empty; Pop out the top elements of the small top heap H1 in sequence and push them into the large top heap H2 until the small top heap H1 is empty; Pop out the top tuple of the big heap H2 in sequence, read its predecessor tuple e from the external memory sequence SPre, and push e into the big heap H2 until the SPre sequence is empty; Pop out the s* type substring tuples at the top of the max-heap H2 in sequence, name the tuples, and generate the contracted string T1 according to their positions in T; Calculate the suffix array SA(T1) of string T1, calculate the s* type suffix ranking of T based on SA(T1), and push the s* type suffix tuple into the small top heap H1; Scan the string T, and calculate the L-type suffix predecessor tuple sequence LPre and the s-type suffix predecessor tuple sequence SPre according to the ranking of the s* type suffixes, and save them to the external memory respectively; The process of calculating the L-type suffix predecessor sequence LPre of the string T is as follows: (1) according to the rule of inductive sorting, the S*-type suffixes and L-type suffixes of T are traversed in ascending order; (2) if the predecessor character of the currently traversed S*-type suffix or L-type suffix is ​​of L type, then a tuple is constructed with the L-type predecessor character and position, and the tuple is written into the sequence LPre; (3) the process is repeated until all the S*-type suffixes and L-type suffixes of T are traversed; The process of calculating the s-type suffix predecessor sequence SPre of the string T is as follows: (1) according to the rule of inductive sorting, traverse the L-type suffixes and S-type suffixes of T in descending order; (2) if the predecessor character of the currently traversed L-type suffix or S-type suffix is ​​of type S, then construct a tuple with the S-type predecessor character and position, and write the tuple into the sequence SPre; (3) repeat the process until all L-type suffixes and S-type suffixes are traversed; Pop the top tuples of the H1 heap in sequence, read the drive tuple E from the sequence LPre, and push E into H1. At the same time, write the preceding character e.char to the end of the L-type BWT sequence Lbwt until the sequence LPre is empty. Pop out the top elements of the small top heap H1 in sequence and push them into the large top heap H2 until the small top heap H1 is empty; Pop out the top tuples of the big heap H2 in sequence, read its predecessor tuple e from the sequence SPre, and push e into H2. At the same time, write the predecessor character E.char to the end of the S type BWT sequence Sbwt until the sequence SPre is empty: The L-type BWT sequence Lbwt and the S-type BWT sequence Sbwt of the string T are merged to generate the final BWT sequence of the string T.

2. The method according to claim 1, characterized in that The process of calculating the L-type BWT sequence Lbwt is as follows: popping out the top suffix tuples from the top heap H1 in sequence, reading the predecessor tuple e of the suffix tuple from the external memory sequence LPre, and constructing the predecessor tuple E=<e.char,++rank,e.index> , push the predecessor tuple e into the mini-heap H1, and write the predecessor character e.char into the L-type BWT sequence Lbwt: where e.char is the predecessor character in the predecessor tuple, rank is the rank, the initial value is 0, and c.index is the position of the predecessor character in the predecessor tuple; the sorting key of the mini-heap H1 is <e.char, rank>.

3. The method according to claim 1, characterized in that: The process of calculating the S type BWT sequence Sbwt is: popping out the suffix tuples at the top of the max heap H2 in sequence, reading the predecessor tuple e of the suffix tuple from the external memory sequence SPre, constructing a predecessor tuple e = <e.char,--rank, e, index>, pushing the predecessor tuple e into the max heap H2, and writing the predecessor character e.char into the S type BWT sequence Sbwt: wherein e.char is the predecessor character in the predecessor tuple, rank is the rank, the initial value is n, and e.index is the position of the predecessor character in the predecessor tuple: the sorting key of the max heap H2 is <e.char, rank>.

4. The method according to claim 1, characterized in that: The merging process of the L-type BWT sequence Lbwt and the S-type BWT sequence Sbwt of the string T is to scan the string T times, count the number of L-type suffixes and S-type suffixes in each suffix bucket, and then traverse Lbwt and Sbwt in ascending order. According to the number of L-type suffixes and S-type suffixes in each suffix bucket, the corresponding L-type and S-type BWT characters belonging to each bucket in the sequence Lbwt and sequence Sbwt are written into the final BWT sequence of the string T in sequence.

5. A device for constructing a large-scale data BWT, characterized in that: include: The substring precursor sequence calculation module is used to calculate the L-type substring precursor tuple sequence LPre and the S-type substring precursor tuple sequence SPre, and save them to the external memory respectively; the substring sorting and naming module sorts and names the S*-type substrings of the string T to generate a new contracted string T1; the suffix array calculation module directly calculates SA(T1) based on the string T1 or recursively; the S*-type suffix calculation module calculates the ranking of the S*-type suffix of the string T according to SA(T1); the suffix precursor sequence calculation module calculates the L-type according to the ranking of the S*-type suffix of the string T. Suffix-precursor tuple sequence LPre and S-type suffix-precursor tuple sequence SPreL-type BWT calculation module, scans the L-type suffix-precursor tuple sequence LPre, and uses the small top heap H1 to calculate the L-type BWT sequence Lbwt until the sequence LPre is empty; S-type BWT calculation module, scans the S-type suffix-precursor tuple sequence SPre, and uses the large top heap H2 to calculate the S-type BWT sequence Sbwt until the sequence SPre is empty; merging module, merges the L-type BWT sequence Lbwt and the s-type BWT sequence Sbwt to generate the final BWT sequence of the string T.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for constructing a large data BWT as described in any one of claims 1 to 4 is implemented.

Images