Method and device for verifying correctness of large text sequence BWT, equipment and medium

Abstract

The application relates to the technical field of computers, and provides a correctness verification method, device and equipment of a large text sequence BWT and a medium. The method comprises the following steps: blocking a character string X and a suffix array thereof, recursively calling block induction sorting to contract the character string X until the characters in the contracted character string X1 are unique; in the backtracking stage of the recursion 0 layer, the order of S* type suffixes of the character string X is calculated according to the suffix array of X1, and the hash value of the descending S* type suffix sequence is calculated; according to the order of the S* type suffixes of the character string X, the L type and S type suffix predecessor character sequence and the hash value thereof of each block are calculated; the L* type suffix order and the hash value of each block of the L type BWT sequence are calculated by using a small top heap; the S* type suffix sequence and the hash value of each block of the S type BWT sequence are calculated by using a large top heap; and the two types of BWT sequences are combined, and the final verification result is output, so that the time and space overheads of BWT correctness verification are reduced.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for verifying the correctness of a large text sequence BWT. Background Technology

[0002] Burrows-Wheeler Transform (BWT) is a reversible text transformation primarily used in data compression and information processing. The BWT of string X is a string of the same length as X, containing more consecutive identical characters, which is more conducive to data compression. Secondly, BWT is reversible; the original string X can be calculated from the BWT of X. Therefore, the original string X can be compressed by compressing its BWT. The correctness of the BWT for large text sequences is crucial for the correctness of large text sequence compression. Therefore, verifying the correctness of the BWT for large text sequences is an important step in achieving large text data compression.

[0003] Among them, Chinese patent CN119420366A discloses a method, device and terminal equipment for constructing large data BWT, but does not describe the verification of the correctness of large data BWT.

[0004] Chinese patent CN117971826A discloses a method and apparatus for constructing a large data linked list with built-in verification function. During the construction of the large data linked list, the correctness of the linked list is verified at the same time. The verification result is given at the same time as the linked list is generated. Only the linked list is verified, and the correctness verification of the large text sequence BWT is not involved.

[0005] Chinese patent CN115982310B provides a linked list generation method and electronic device with built-in verification function, which integrates the verification process of suffix linked lists into the construction process of linked lists, so that the constructed suffix linked lists do not need to be verified separately, and also do not involve the correctness verification of large text sequence BWT.

[0006] Chinese patent CN107015951B discloses a method and system for verifying the correctness of suffix arrays. In the process of constructing suffix arrays, the correctness verification of suffix arrays is realized simultaneously. Only the suffix arrays are verified, and the correctness verification of large text sequence BWTs is not involved.

[0007] Chinese patent CN107015952A discloses a method and system for verifying the correctness of suffix arrays and longest common prefixes. After the SA and LCPA are constructed, the correctness verification of SA and LCPA can be performed simultaneously, reducing the time and space overhead of SA and LCPA correctness verification, but it does not involve the correctness verification of large text sequence BWT.

[0008] Existing string BWT correctness verification methods involve recovering the original string from a known BWT and comparing it with the original string to verify the BWT's correctness. For small datasets, this verification method can be completed quickly in memory; however, for large text sequences, both the string and its BWT are stored in external memory. The entire verification calculation process requires a large number of random I / O operations on external memory, consuming massive amounts of computational resources and resulting in low verification efficiency. Summary of the Invention

[0009] The purpose of this invention is to provide a method, apparatus, device, and medium for verifying the correctness of large text sequence BWTs. This method iteratively calculates the hash value of each BWT sequence block during the block-based BWT construction process. Simultaneously with the generation of the final BWT, the correctness of the final BWT sequence is determined by comparing the hash value of each BWT sequence block with its sequential hash value in the final BWT sequence, as well as the sequential hash value of the S* type suffix. This method eliminates the need for independent verification in terms of additional time and space, and can complete correctness verification synchronously during BWT generation, avoiding the additional time and space overhead of independently verifying large text sequence BWTs. It aims to solve the problem of low verification efficiency in existing technologies.

[0010] Firstly, this application provides a method for verifying the correctness of large text sequence BWTs, comprising the following steps:

[0011] S1: Divide the string X and its suffix array (SA) into multiple string blocks and multiple suffix bucket blocks respectively; the suffix array of the string X is represented by SA(X); the suffix bucket block is composed of multiple consecutive suffix buckets; the suffix bucket is composed of suffixes with the same first character in SA;

[0012] S2: Using external storage block inductive sorting, the S* type substrings of the string X are sorted and named to generate a shrunken string; the external storage block inductive sorting method is to divide the string and suffix array into blocks and use external storage heap sort to simulate the memory inductive sorting method.

[0013] S3: Determine whether the characters in the shrunk string are unique;

[0014] S4: When the characters in the shrunk string are unique, determine whether the current recursion level is level 0;

[0015] S5: If the current recursion level is not level 0, calculate the L-type predecessor character sequence block and the S-type predecessor character sequence block of the current level string according to the order of the S* type suffix of the current level string, and save them to external storage.

[0016] S6: Using the external storage block sorting method, sort the L-type suffixes and S-type suffixes in turn to generate the suffix array of the current layer string;

[0017] S7: If the current recursive layer is layer 0, calculate the order of the S* type suffixes of string X according to the suffix array of the shrunken string. During the calculation process, call the hash function to iteratively calculate the hash value of the descending S* type suffix sequence.

[0018] S8: Calculate the predecessor character sequence of type L and type S for each block of string X and save it to external storage. During the calculation process, call the hash function to iteratively calculate the hash value of the two types of predecessor sequences for each block.

[0019] S9: Use a min-heap to calculate the L-type BWT sequence, save the L* type suffix sequence to external storage, and call the hash function during the calculation to iteratively calculate the hash value of the predecessor block to which each character in the L-type BWT sequence belongs;

[0020] S10: Use a max-heap to calculate the S-type BWT sequence. During the calculation process, call the hash function to iteratively calculate the hash value of the descending S* type suffix sequence and the corresponding predecessor block of each character in the S-type BWT sequence.

[0021] S11: Merge L-type and S-type BWT sequences to generate the final BWT sequence;

[0022] S12: Compare the hash values ​​of the descending S* type suffix sequence, the hash values ​​of each block of the L type BWT sequence, and the hash values ​​of each block of the S type BWT sequence, and output the final verification result.

[0023] In one embodiment, the i-th block is denoted as Bi, and step S8 includes the following steps:

[0024] Calculate the order of S* type suffixes in the current block Bi based on the order of the S* type suffix sequence of string X;

[0025] The block Bi is loaded into memory, and the S-type buckets of the suffix array SA(Bi) are initialized with ordered S* type suffix first characters; the suffix array of the block Bi is represented by SA(Bi);

[0026] Scan the suffix array SA(Bi) to the right, and write the L-type predecessor characters into the L-type suffix buckets of the suffix array SA(Bi) and the L-type predecessor character block corresponding to the current block Bi in external storage in turn; during this process, whenever an L-type predecessor character is obtained, call the Karp-Rabin hash function to iteratively calculate the hash value of the L-type predecessor character sequence of the current block.

[0027] Scan the suffix array SA(Bi) to the left, and write the S-type predecessor characters into the S-type suffix buckets of the suffix array SA(Bi) and the corresponding S-type predecessor character block in external storage for the current block Bi in turn; during this process, whenever an S-type predecessor character is obtained, call the Karp-Rabin hash function to iteratively calculate the hash value of the S-type predecessor character sequence of the current block.

[0028] In one embodiment, step S9 includes the following steps:

[0029] S901: Scan each suffix bucket in the suffix array of string X in ascending order. For each suffix bucket, scan each suffix bucket in ascending order. Suppose that the current step is to scan the i-th suffix bucket, the min-heap H1 is empty, and the sequence number idx = 0. Sequentially push the suffixes of the i-th suffix bucket into the min-heap H1 from external storage. Each time a suffix is ​​pushed in, increment the sequence number idx of the suffix tuple by 1.

[0030] S902: If the min-heap H1 is not empty, remove the top suffix e; if the first character e.chr of the suffix e belongs to the j-th bucket of the i-th suffix bucket block, then execute S904 on the suffix e until the first character of the suffix e does not belong to the j-th bucket or the min-heap H1 is empty.

[0031] S903: Traverse the S* type suffix sequence of string X in ascending order, and take out the first suffix e in turn. If the first character e.chr of suffix e belongs to bucket j, then execute S904 on suffix e, until the first element of the S* type suffix sequence does not belong to bucket j or the S* type suffix sequence is traversed.

[0032] S904: For each suffix 'e', ​​calculate the string X block k to which it belongs based on the suffix position 'e.pos', and iteratively calculate the hash value of the L-type BWT sequence block k using the Karp-Rabin hash function; extract the predecessor character 'pchar' of suffix 'e' from the head of the predecessor character sequence block k, and append the predecessor character 'pchar' to the tail of the L-type BWT sequence; calculate the suffix bucket 'q' to which the predecessor suffix 'e1' belongs based on the predecessor character 'pchar'; if the predecessor suffix 'e1' belongs to the current bucket i, then set e1 = ...<pchar, e.pos-1, idx++> Push it into the min-heap H1, otherwise set e1=<pchar, e.pos-1> Write the suffix to the secondary storage bucket q; if the lexicographical order of the predecessor character pchar is less than e.chr, then the suffix e is an L* type suffix, and set the suffix e = ...<e.chr,e.pos> Append to the end of the L* type suffix sequence.

[0033] In one embodiment, step S10 includes the following steps:

[0034] S1001: Scan each suffix bucket in descending order. For each suffix bucket, scan each bucket in descending order. Suppose that the current scan is of the i-th suffix bucket. Initialize the max heap H2 to be empty and the sequence number idx = n. Sequentially push the suffix of the i-th suffix bucket into the max heap H2 from external storage. Each time a suffix is ​​pushed, decrement the sequence number idx of the suffix tuple by 1.

[0035] S1002: If the max-heap H2 is not empty, remove the top suffix e; if the first character e.chr of the suffix e belongs to the j-th bucket of the i-th suffix bucket block, then execute S1004 on the suffix e until the top element of the heap does not belong to the j-th bucket or the max-heap H2 is empty.

[0036] S1003: Traverse the L* type suffix sequence of string X in descending order, and take out the first suffix e in turn; if the first character e.chr of suffix e belongs to bucket j, then execute S1004 on suffix e, until the first element of the L* type suffix sequence does not belong to bucket j or the L* type suffix sequence is traversed.

[0037] S1004: For each suffix 'e', ​​based on the suffix position 'e.pos', calculate the block k of the string X to which the suffix 'e' belongs, and iteratively calculate the hash value of block k in the S-type BWT sequence using the Karp-Rabin hash function; extract the predecessor character 'pchar' of the suffix 'e' from the head of the predecessor character sequence block k, and append the predecessor character 'pchar' to the tail of the S-type BWT sequence; calculate the suffix bucket to which the predecessor suffix 'e1' of the suffix 'e' belongs based on the predecessor character 'pchar'; if 'e1' belongs to the current bucket i, then set 'e1' = ...<pchar, e.pos-1, idx--> Push it into the max-heap H2, otherwise set e1=<pchar, e.pos-1> Write the suffix to its corresponding bucket in external storage; if the lexicographical order of the predecessor character pchar is greater than e.chr, then the suffix e is an S* type suffix, and use the Karp-Rabin hash function to iteratively calculate the hash value of the S* type suffix sequence.

[0038] In one embodiment, step S11 includes the following steps:

[0039] Scan string X and count the number of L-type suffixes and S-type suffixes in each character bucket;

[0040] Scan the L-type BWT sequence in ascending order, and extract the corresponding number of characters from each L-type character bucket according to the statistical data.

[0041] Scan the S-type BWT sequence in ascending order, and extract the corresponding number of characters from each S-type character bucket according to the statistical data.

[0042] The L-type and S-type BWT characters in each suffix bucket are merged sequentially to generate the final BWT sequence.

[0043] In one embodiment, comparing the hash values ​​of the descending S* type suffix sequences refers to comparing whether the hash values ​​of the descending S* type suffix sequences in steps S7 and S10 are the same.

[0044] In one embodiment, comparing the hash values ​​of each block of the L-type BWT sequence means comparing whether the hash values ​​of the L-type predecessor character sequence block in step S8 and the hash values ​​of the L-type BWT sequence block in step S9 are the same.

[0045] In one embodiment, comparing the hash values ​​of each block of the S-type BWT sequence means comparing whether the hash values ​​of the S-type predecessor character sequence block in step S8 and the hash values ​​of the S-type BWT sequence block in step S10 are the same.

[0046] Secondly, this application provides a correctness verification apparatus for a large text sequence BWT, comprising:

[0047] The preprocessing module is used to divide the string X and its suffix array SA(X) into multiple blocks and suffix buckets respectively;

[0048] The string shrinking module is used to calculate the order of substrings of type S* and name them sequentially to generate a shrunken string X1;

[0049] The Determiner A module is used to determine whether the characters in the shrunk string X1 are unique.

[0050] The Determiner B module is used to determine whether the current recursion level is level 0.

[0051] The suffix predecessor character calculation module is used to calculate the predecessor character sequence blocks of the L-type and S-type suffixes of string X and save them to external storage;

[0052] The suffix array calculation module is used to deduce the order of L-type and S-type suffixes using the external memory inductive sorting method, and generate the current layer suffix array;

[0053] The module for calculating the hash value of the preceding character sequence of each block of a string is used to calculate the preceding character sequence blocks and their hash values ​​for L-type and S-type suffixes of strings, as well as the hash value of the descending S* type suffix sequence.

[0054] The L-type BWT sequence block hash value calculation module is used to calculate the L-type BWT sequence. During the process, the hash function is called to iteratively calculate the hash value of the L-type BWT sequence block, and the L* type suffix sequence is saved to external storage.

[0055] The S-type BWT sequence hash value calculation module is used to calculate the S-type BWT sequence. During this process, the hash function is called to iteratively calculate the hash value of each block of the S-type BWT sequence, as well as the hash value of the descending S* type suffix sequence.

[0056] The BWT generation and correctness verification module is used to merge L-type and S-type BWT sequences, compare the hash values ​​of each block of the descending S* type suffix sequence and the string L-type and S-type BWT sequences, and output the final verification result.

[0057] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the correctness verification method for a large text sequence BWT as described in any of the preceding claims.

[0058] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the correctness verification method for a large text sequence BWT as described in any of the preceding claims.

[0059] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the correctness verification method for large text sequence BWT as described in any of the preceding claims.

[0060] The beneficial effects of the method, apparatus, device, medium, and computer program product for verifying the correctness of a large text sequence BWT provided by this invention are as follows: The S* type substring of string X is recursively sorted and shrunk until the characters in the shrunk string X1 are unique; otherwise, recursion is executed. At the higher recursive level, a suffix array is calculated. At recursion level 0, L-type and S-type BWT sequences of string X are calculated sequentially. During the calculation process, the hash values ​​of each block of the two types of BWT sequences and the descending S* type suffix are iteratively calculated, and then the two are merged to generate the final BWT sequence. During the BWT sequence calculation process, the hash values ​​of the blocks to which the L-type and S-type characters of the BWT sequence belong and the descending S* type suffix sequence are iteratively calculated. Finally, the correctness of the final BWT is verified by comparing the hash values ​​of the L-type and S-type BWT sequence blocks and the descending S* type suffix sequence before and after heap sort. In applications involving large-scale text data compression, a correctness verification method for large-scale text sequence BWTs proposed in this embodiment can be adopted. This method integrates the calculation of correctness verification into the BWT construction process, eliminating the need for separate verification of the BWT and reducing the time and space overhead of BWT correctness verification. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic diagram of the character type of string X, the suffix array SA(X), and the storage structure of the BWT sequence bwt(X) provided in the embodiments of this application;

[0063] Figure 2 This is a schematic diagram of the storage structure of the suffix bucket and its type of string X, L-type BWT sequence and S-type BWT sequence provided in the embodiments of this application;

[0064] Figure 3 This is a schematic diagram showing the positional relationship between L-type and S-type BWT blocks of string X in the final BWT sequence, provided in an embodiment of this application.

[0065] Figure 4 This is a flowchart illustrating a method for verifying the correctness of a large string sequence BWT provided in an embodiment of this application;

[0066] Figure 5 This is a schematic diagram of the structure of a correctness verification device for a large string sequence BWT provided in an embodiment of this application;

[0067] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0068] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0069] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0070] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0071] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once detected," or "in response to detection."

[0072] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0073] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0074] First, the technical terms that may be used in the embodiments of this application will be uniformly explained here.

[0075] String X: An array or sequence of n consecutive characters, where X[i] represents the character at position i, X[i, j] represents the string from position i to position j, and X[i, n-1] represents the suffix starting from position i. Dictionary order, also known as alphabetical order, originally refers to the order in which English words appear in a dictionary. In computer science, it has been extended to refer to the ranking relationship between two arbitrary strings determined by the order in which English words appear in a dictionary.

[0076] Character and suffix types: If X[i] is lexicographically greater than X[i+1], or X[i] is equal to X[i+1] and X[i+1] is of type L, then character X[i] is of type L (also known as reverse character); otherwise, character X[i] is of type S (also known as ascending character). The end character X[n-1] of the string is defined as type S. Where 1≤i≤n-1. If X[i] is of type S and X[i-1] is of type L, then X[i] is of type S*; if X[i] is of type L and X[i-1] is of type S, then X[i] is of type L*. The type of a suffix starting with X[i] is determined by the type of X[i].

[0077] Substrings and their types: A substring is a string consisting of a character in string X starting from the character to the nearest S* type character to the right of that character; the type of a substring is determined by its first character.

[0078] Preceding character: If a suffix / substring has a character to its left, that character is called the preceding character of the suffix / substring.

[0079] Suffix array, suffix bucket, and suffix bucket block: The suffix array SA(X) of string X is an integer array of the same length as X, which stores the order of suffixes; a continuous interval consisting of suffixes with the same first character is called a suffix bucket; a suffix bucket includes L-type suffix buckets and S-type suffix buckets, with the L-type suffix bucket located to the left of the S-type suffix bucket; multiple consecutive suffix buckets constitute a suffix bucket block.

[0080] BWT Sequence: The BWT sequence of string X is a string of the same length as X, denoted by bwt(X). Its calculation rule is: if SA[i]≠0, then bwt[i] = X[SA[i]-1]; otherwise, bwt[i]=X[n-1].

[0081] L / S type BWT sequence: The sequence consisting of the preceding and succeeding characters of the ordered L / S type suffix is ​​called the L / S type BWT sequence, denoted as Lbwt / Sbwt.

[0082] Karp-Rabin hash functions: a set of functions for calculating the hash value of a string or sequence, including two formulas: (1) FP(A[-1]) = 0; (2) FP(A[i]) = FP(A[i-1])· +A[i] mod L, where the parameter L is a prime number, and the parameter It is a random integer between [1, L). The larger L is, the lower the probability of error.

[0083] Taking the string X[0,12]="mmiissiissii$" as an example, let's explain the above terms:

[0084] See Figure 1 The text shows the character types of string X, the suffix array SA(X), and the storage structure of the BWT sequence bwt(X). Line 3 shows the type of each character in X. For example, the lexicographical order of X

[05] ='s' is greater than that of X

[06] ='i', so X

[05] is of type L. Since X

[04] equals X

[05] , X

[04] is of type L. Since X

[03] is of type S, X

[04] is of type L*. Line 4 shows the storage structure of the suffix array SA(X) of X, which is an integer array of length 13, storing the starting position of the ascending suffix of X from left to right. Line 5 shows the storage structure of the BWT sequence bwt(X) of X. bwt(X) can be calculated based on SA(X). For example, bwt[2]=X[SA[2]-1]=X

[09] , because SA

[08] =0, bwt

[08] =X

[12] .

[0085] See Figure 2 The diagram illustrates the suffix buckets and bucket types of string X, as well as the storage structures of L-type and S-type BWT sequences. Line 3 shows the composition of the suffix buckets in SA(X), consisting of four buckets: the '$' suffix bucket located in SA

[00] , the 'i' suffix bucket located in SA[01,06], the 'm' suffix bucket located in SA[07,08], and the 's' suffix bucket located in SA[09,12]. Line 4 shows the type of the suffix buckets; for example, in the 'i' suffix bucket, SA[01,02] and SA[03,06] are the L-type and S-type suffix buckets for 'i', respectively. Lines 6 and 7 show the L-type and S-type BWT sequences, respectively.

[0086] See Figure 3This shows the two blocks of string X, and the correspondence between each character in the L-type and S-type BWT sequence blocks of X in the final BWT sequence. The second line shows the two blocks of X, X[00, 06] and X[07, 12]; the sixth line shows the L-type and S-type BWT blocks corresponding to the two blocks of X, where {X

[05] , X

[01] , X

[02] } is the L-type BWT block of the first block, {X

[00] , X

[12] , X

[04] , X

[03] } is the S-type BWT block of the first block, {X

[11] , X

[06] } is the L-type BWT block of the second block, and {X

[10] , X

[09] , X

[08] , X

[07] } is the S-type BWT block of the second block; the last line shows the position order of each character in the BWT sequence in the two types of BWT blocks, for example, the last line points to {03, 04, 12], The solid arrow at position {06} indicates the position of the character in the L-type BWT block of the first block of X in the final BWT sequence, and the dashed arrow at position {07, 08, 10, 12} in the last line indicates the position of the character in the S-type BWT block of the first block of X in the final BWT sequence.

[0087] The specific computational framework of this application embodiment is as follows: First, X is recursively sorted and shrunk using the external memory block inductive sorting method to generate a shrunk string X1; second, the suffix array SA(X1) of X1 is calculated; then, at the algorithm recursion level 0, the hash value of each BWT sequence block and the S* type suffix sequence of string X is iteratively calculated using a hash function; next, during the calculation of the BWT sequence of string X, the hash value of the BWT block to which each character in the BWT sequence belongs and the descending S* type suffix sequence are simultaneously iteratively calculated; finally, the hash values ​​of the corresponding BWT blocks and S* type suffix sequences before and after generating the final BWT are compared to see if they are consistent, and the final verification result is output.

[0088] The technical solution of this application will be described below through specific embodiments.

[0089] Example 1

[0090] Reference Figure 4 This diagram illustrates a step-by-step flowchart of a method for verifying the correctness of a large text sequence BWT provided in an embodiment of this application. Specifically, it may include the following steps:

[0091] S1: Based on the memory capacity of the computing device, divide the string X and its suffix array SA(X) into multiple string blocks and multiple suffix buckets. That is, divide the string X into multiple string blocks and divide the suffix array SA(X) of the string X into multiple suffix buckets.

[0092] In this embodiment of the application, dividing the string X and its suffix array SA(X) into blocks is to calculate the access order of the predecessor characters of the suffix in each block during the inductive sorting process.

[0093] In this embodiment of the application, the step of dividing string X into blocks is as follows: S101, calculate the maximum length of the string X block based on the memory capacity of the computing device and the workspace required for inductive sorting; S102, read string X sequentially from external storage and calculate the length of the currently detected S* type substring; S103, based on the maximum length of the string X block and the length of the currently detected S* type substring, determine whether the currently detected S* type substring can be merged into the current block or treated as a separate block; S104, continue to execute the above S102 and S103 until the string X is read completely.

[0094] In this embodiment of the application, the process of dividing the suffix array SA(X) into suffix buckets is as follows: scan the string X, calculate the number of suffixes in each suffix bucket in the suffix array SA(X), accumulate the number of suffixes in adjacent suffix buckets, and if it exceeds the minimum memory working space required for inductive sorting, the currently accumulated suffix buckets constitute a new suffix bucket block. Repeat this process until the accumulation process ends.

[0095] S2: Use external storage block sorting to sort and name the S* type substrings of string X, and generate a shrunken string X1.

[0096] In this embodiment of the application, the process of sorting the S* type substrings of string X is as follows: S201: Using the external memory inductive sorting method, the order of the L type substrings is recursively deduced using the S* type characters; S202: The order of the S type substrings is recursively deduced using the order of the L type substrings; S203: The S* type substrings are retrieved from the sorted S type substrings.

[0097] In this embodiment, the process of naming the S* type substrings of string X is as follows: S204: Scan the ordered sequence of S* type substrings from left to right, and compare whether adjacent S* type substrings are the same; S205: If two S* type substrings are completely identical, then the names of the two substrings are the same; otherwise, the name of the S* type substring on the right is incremented by 1. The name of the S* type substring is a positive integer, and the substring name can start from 1 and increment sequentially by 1.

[0098] In this embodiment of the application, the shrunken string X1 is a new string obtained by arranging the named S* type substrings according to their positions in the string X.

[0099] S3: Determine if the characters in the shrunk string X1 are unique. If not, execute the recursion and return to step S1.

[0100] S4: When the characters in the shrunk string X1 are unique, determine whether the current recursion level is level 0. Specifically, if the current recursion level is not level 0, calculate the suffix array SA(X1) of the current recursion level; otherwise, calculate the BWT sequence of string X. The suffix array of the shrunk string X1 is represented by SA(X1).

[0101] S5: If the current recursive layer is not layer 0, calculate the predecessor character sequence of each block of the contracted string X1 with L-type suffix and the predecessor character sequence of each block with S-type suffix, and save them to the corresponding predecessor blocks in external storage.

[0102] In this embodiment, step S5 specifically involves: dividing the shrunken string X1 into multiple blocks according to the memory capacity of the computing device, and sequentially calculating and saving the L-type suffix predecessor character sequence and S-type suffix predecessor character sequence of each block to external storage. Assuming the current calculation is for the i-th block Bi, step S5 specifically includes the following steps: S501: Load the current block Bi into memory, and initialize the S-type bucket of the suffix array SA(Bi) with the S* type characters of block Bi. S502: Scan the suffix array SA(Bi) to the right, and sequentially write the L-type predecessor characters into the L-type suffix bucket of the suffix array SA(Bi) and the L-type predecessor character block corresponding to the current block Bi in external storage. S503: Scan the suffix array SA(Bi) to the left, and sequentially write the S-type predecessor characters into the S-type suffix bucket of the suffix array SA(Bi) and the S-type predecessor character block corresponding to the current block Bi in external storage.

[0103] S6: Using the external memory inductive sorting method, sort the L-type suffixes and S-type suffixes in turn to generate the suffix array SA(X1) of the contracted string X1.

[0104] In this embodiment, step S6 specifically includes: S601: dividing the shrunk string X1 into multiple blocks, and dividing the suffix array SA(X1) into multiple suffix buckets; S602: calculating the L-type predecessor blocks and S-type predecessor blocks corresponding to each block of the shrunk string X1; S603: calculating the order of L-type suffixes using a min-heap H1 based on the S*-type suffix order and L-type predecessor blocks of the shrunk string X1; S604: calculating the order of S-type suffixes using a max-heap H2 based on the L*-type suffix order and S-type predecessor blocks of the shrunk string X1; S605: merging the L-type and S-type suffixes in ascending order to obtain the suffix array SA(X1). The elements of the min-heap H1 and max-heap H2 are triples.<chr, pos, idx> These represent the first character of the suffix, the position of the suffix, and the sort number of the suffix, respectively. The sort key is a tuple.<chr, idx> .

[0105] S7: If the current recursion level is level 0, calculate the order of the S* type suffixes of string X based on the suffix array SA(X1) of the contracted string X1. During the calculation, call the hash function to iteratively calculate the hash value of the descending S* type suffix sequence.

[0106] In this embodiment of the application, the step of calculating the order of S* type suffixes according to SA(X1) is as follows: calculate the position of the S* type suffix in string X; according to the correspondence between the elements in SA(X1) and the positions of the S* type suffixes in string X; scan SA(X1) to calculate the order of the S* type suffixes in string X.

[0107] The process of calculating the hash value of the descending S* type suffix sequence is as follows: scan the S* type suffix sequence in descending order, and iteratively call the Karp-Rabin hash function to calculate the hash value of the descending S* type suffix sequence.

[0108] S8: Calculate the predecessor character sequences of type L and type S for each block of string X, and save them to the corresponding predecessor blocks in external storage. During the calculation process, call the Karp-Rabin hash function to iteratively calculate the hash values ​​of the two types of predecessor blocks for each block.

[0109] In this embodiment, the process of calculating the predecessor character sequences of each block of string X of type L and type S and saving them to the corresponding predecessor blocks in external storage is the same as step S5. The main difference between step S8 and step S5 is that, in the process of calculating the predecessor character sequences, it is necessary to calculate the hash value of each predecessor character sequence block.

[0110] In this embodiment of the application, the process of calculating the hash value of each L-type and S-type predecessor sequence block of string X is as follows:

[0111] S801: Assuming that the i-th block Bi is currently being calculated, calculate the order of the S* type suffixes in the current block Bi according to the order of the S* type suffix sequence of string X.

[0112] S802: Load the block Bi into memory and initialize the S-type buckets of the suffix array SA(Bi) with the first character of the ordered S* type suffix.

[0113] S803: Scan the suffix array SA(Bi) to the right, and sequentially write the L-type predecessor characters into the L-type suffix buckets of the suffix array SA(Bi) and the corresponding L-type predecessor character block in external storage for the current block Bi. During this process, whenever an L-type predecessor character is obtained, the Karp-Rabin hash function is called to iteratively calculate the hash value of the L-type predecessor character sequence for the current block.

[0114] S804: Scan the suffix array SA(Bi) to the left, and sequentially write the S-type predecessor characters into the S-type suffix buckets of the suffix array SA(Bi) and the corresponding S-type predecessor character block in external storage for the current block Bi. During this process, whenever an S-type predecessor character is obtained, the Karp-Rabin hash function is called to iteratively calculate the hash value of the current block's S-type predecessor character sequence.

[0115] S9: Use a min-heap to calculate the L-type BWT sequence, save the L* type suffix sequence to external storage, and call the hash function during the calculation to calculate the hash value of each block of the L-type BWT sequence.

[0116] In this embodiment of the application, the hash value of each block of the L-type BWT sequence refers to the hash value of the BWT character sequence belonging to a certain L-type predecessor character block of string X in the L-type BWT sequence calculated by iteratively calling the Karp-Rabin hash function during the calculation of the L-type BWT sequence.

[0117] In this embodiment of the application, the calculation process of using a min-heap H1 to calculate the L-type BWT sequence and the hash values ​​of each block is as follows:

[0118] S901: Scan the suffix array SA(X) in ascending order. For each suffix bucket, scan each bucket in ascending order. Assume that we are currently scanning the i-th suffix bucket, the min-heap H1 is empty, and the sequence number idx = 0. Sequentially push the suffixes of the i-th suffix bucket from external storage into the min-heap H1. Each time a suffix is ​​pushed, the sequence number idx of the suffix tuple is incremented by 1.

[0119] S902: If the min-heap H1 is not empty, extract the top suffix e. If the first character of the suffix e.chr belongs to bucket j of the current bucket block, then execute S904 on the suffix e until the current top element of the heap does not belong to bucket j or the min-heap H1 is empty.

[0120] S903: Traverse the S* type suffix sequence of string X in ascending order, and extract the first suffix e in each queue. If the first character e.chr of suffix e belongs to bucket j, then execute S904 for suffix e, until the first element of the S* type suffix sequence does not belong to bucket j or the S* type suffix sequence has been traversed.

[0121] S904: For each suffix 'e', ​​calculate the string block k to which 'e' belongs based on its position 'e.pos', and iteratively calculate the hash value of block k in the L-type BWT sequence using the Karp-Rabin hash function. Take the predecessor character 'pchar' from the head of block k of the predecessor character sequence and append it to the end of the L-type BWT sequence. Calculate the suffix bucket 'q' to which the predecessor suffix 'e1' belongs based on 'pchar'. If the predecessor suffix 'e1' belongs to the current bucket i, then set e1 = ...<pchar, e.pos-1, idx++> Push it into the min-heap H1, otherwise set e1=<pchar, e.pos-1> Write the suffix to the secondary storage bucket q. If the lexicographical order of the preceding character pchar is less than e.chr, then the suffix e is an L* type suffix, and set the suffix e = ...<e.chr,e.pos> Append to the end of the L* type suffix sequence.

[0122] S10. Use a max-heap to calculate the S-type BWT sequence. During the calculation, call the hash function to calculate the hash value of the descending S* type suffix sequence and the hash value of each block of the S-type BWT sequence.

[0123] In this embodiment of the application, the hash value of each block of the S-type BWT sequence refers to the hash value of the BWT character sequence belonging to a certain S-type predecessor character block of string X in the S-type BWT sequence calculated by iteratively calling the Karp-Rabin hash function during the calculation of the S-type BWT sequence.

[0124] In this embodiment of the application, the calculation process of using a max-heap H2 to calculate the S-type BWT sequence and the hash values ​​of each block is as follows:

[0125] S1001: Scan the suffix array SA(X) in descending order. For each suffix bucket, scan each bucket in descending order. Let's assume we start scanning the i-th suffix bucket, the max-heap H2 is empty, and the sequence number idx = n. Sequentially push the suffixes of the i-th suffix bucket from external storage into the max-heap H2. Each time a suffix is ​​pushed, decrement the sequence number idx of the suffix tuple by 1.

[0126] S1002: If the max-heap H2 is not empty, extract the top suffix 'e'. If the first character of the suffix 'e.chr' belongs to bucket j of the current bucket block, then execute S1004 on the suffix 'e' until the current top element of the heap does not belong to bucket j or the max-heap H2 is empty.

[0127] S1003: Traverse the L* type suffix sequence of string X in descending order, and take out the first suffix e in turn. If the first character e.chr of suffix e belongs to bucket j, then execute S1004 for suffix e, until the first element of the L* type suffix sequence does not belong to bucket j or the L* type suffix sequence has been traversed.

[0128] S1004: For each suffix 'e', ​​based on the suffix position 'e.pos', calculate the string block 'k' to which suffix 'e' belongs, and iteratively calculate the hash value of block 'k' in the S-type BWT sequence using the Karp-Rabin hash function. Take the predecessor character 'pchar' of suffix 'e' from the head of block 'k' and append it to the end of the S-type BWT sequence. Calculate the suffix bucket to which the predecessor suffix 'e1' belongs based on 'pchar'. If 'e1' belongs to the current bucket 'i', then set 'e1' = ...<pchar, e.pos-1, idx--> Push it into the max-heap H2, otherwise set e1=<pchar, e.pos-1> Write the suffix to its corresponding bucket in external storage. If the lexicographical order of the preceding character pchar is greater than e.chr, then the suffix e is an S* type suffix, and the hash value of the S* type suffix sequence is calculated iteratively using the Karp-Rabin hash function.

[0129] S11. Merge the L-type BWT sequence and the S-type BWT sequence to generate the final BWT sequence of string X.

[0130] In this embodiment of the application, the process of merging the L-type BWT sequence and the S-type BWT sequence is as follows:

[0131] S1101: Scan string X and count the number of L-type suffixes and S-type suffixes in each character bucket.

[0132] S1102: Scan the L-type BWT sequence in ascending order. Based on the statistical data in S1101, extract the corresponding number of characters from each L-type character bucket in the L-type BWT sequence.

[0133] S1103: Scan the S-type BWT sequence in ascending order. Based on the statistical data of S1101, extract the corresponding number of characters from each S-type character bucket in the S-type BWT sequence.

[0134] S1104: Merge the L-type and S-type BWT characters in each suffix bucket in sequence to generate the final BWT sequence.

[0135] S12. Compare the hash values ​​of the descending S* type suffix sequence, the L type BWT sequence block, and the S type BWT sequence block before and after heap sort, and output the final verification result.

[0136] In this embodiment, "before heap sorting" refers to steps S7 and S8. "After heap sorting" refers to steps S9 and S10. Comparing the hash values ​​of the descending S* type suffix sequences means comparing whether the hash values ​​of the descending S* type suffix sequences in steps S7 and S10 are the same. Comparing the hash values ​​of each block of the L type BWT sequence means comparing whether the hash values ​​of the L type predecessor character sequence block in step S8 are the same as the hash values ​​of the L type BWT sequence block in step S9. Comparing the hash values ​​of each block of the S type BWT sequence means comparing whether the hash values ​​of the S type predecessor character sequence block in step S8 are the same as the hash values ​​of the S type BWT sequence block in step S10.

[0137] In this embodiment, the final verification result of a correct final BWT sequence means that the hash values ​​of the S* type suffix sequence, the L type BWT sequence block, and the S type BWT sequence block are all consistent. If any set of comparison results is incorrect, the final verification result is that the final BWT sequence is incorrect.

[0138] Example 2

[0139] Reference Figure 5 The diagram illustrates a structural schematic of a large text sequence BWT correctness verification device provided in an embodiment of this application, which may specifically include the following modules:

[0140] The preprocessing module 501 is used to sequentially read the string X from the external storage of the computing device and divide the string X into multiple blocks according to the memory capacity of the computing device, using S* type characters as delimiters; secondly, while reading the string X, it counts the number of suffixes in each suffix bucket of the suffix array SA(X), and divides the suffix array into multiple suffix bucket blocks according to the external storage sorting rules and memory capacity.

[0141] The string shrinking module 502 recursively calls the external storage inductive sorting method to sort the S* type substrings of string X. It names each S* type substring by comparing the similarities and differences of adjacent S* type substrings, and replaces the S* type substring with its name according to the position of the S* type substring in X, generating a shorter shrunken string X1.

[0142] Determiner A module 503 determines whether the characters in the shrunken string X1 are unique.

[0143] Determiner B module 504 determines whether the current recursion level is level 0.

[0144] The suffix predecessor character calculation module 505 calculates the order of S* type suffixes of the current recursive layer string based on the suffix array calculated by the previous layer; it loads the current recursive layer string into the memory of the computing device in blocks in turn, calculates the L type predecessor character sequence and the S type predecessor character sequence according to the inductive sorting rules, and saves them to external storage respectively.

[0145] The suffix array calculation module 506 calculates the order of L-type suffixes by using a min-heap to simulate external memory inductive sorting based on the block division of L-type and S-type predecessor characters, the suffix order of S* type, and the size of the suffix buckets. Next, it calculates the order of S-type suffixes by using a max-heap to simulate external memory inductive sorting based on the ordered L-type suffixes. Finally, it merges the L-type and S-type suffixes to generate the suffix array of the current recursive layer string.

[0146] The string block predecessor character sequence hash value calculation module 507 is used to calculate the order of S* type suffixes of string X based on the suffix array SA(X1); it loads each block of string X into memory sequentially and performs inductive sorting; based on the order of the S* type suffixes of the current block, it calculates the L type predecessor character sequence block and the L* type suffix sequence corresponding to the current string block and saves them to external storage; during the calculation, whenever an L type predecessor character is calculated, the Karp-Rabin hash function is called to iteratively calculate the hash value of the current L type predecessor character block. Based on the order of the L* type suffixes of the current block, it calculates the S type predecessor character sequence block corresponding to the current string block and saves it to external storage; during the calculation, whenever an S type predecessor character is calculated, the Karp-Rabin hash function is called to iteratively calculate the hash value of the current S type predecessor character block.

[0147] The L-type BWT sequence block hash value calculation module 508 is used to calculate the L-type BWT sequence based on the order of the S* type suffixes of string X, using a min-heap to simulate external memory inductive sorting, and write the L* type suffix sequence to external memory for later use; during the calculation, for each scanned L-type suffix, the BWT sequence block to which it belongs is calculated according to its position, and the Karp-Rabin hash function is called to iteratively calculate the hash value of the BWT sequence block to which it belongs.

[0148] The S-type BWT sequence block hash value calculation module 509 is used to calculate the S-type BWT sequence based on the order of the L* type suffixes of string X, using a max-heap to simulate external memory inductive sorting, and simultaneously calling the Karp-Rabin hash function to iteratively calculate the hash value of the descending S* type suffix sequence. During the calculation of the BWT sequence, for each scanned S-type suffix, its corresponding BWT sequence block is calculated based on its position, and the Karp-Rabin hash function is called to iteratively calculate the hash value of its corresponding BWT sequence block.

[0149] The BWT generation and correctness verification module 510 is used to scan the L-type BWT sequence and the S-type BWT sequence, and merge the two types of BWT sequences according to the number of L-type and S-type suffixes in the string X suffix bucket; the correctness of the final BWT is verified by comparing the hash values ​​of the descending S* type suffix sequences in modules 507 and 509, the block hash values ​​of the L-type BWT sequences in modules 507 and 508, and the block hash values ​​of the S-type BWT sequences in modules 507 and 509.

[0150] In this embodiment of the application, the final BWT sequence being correct means that the hash value comparison results of the S* type sequence and the L type and S type BWT sequence blocks are all consistent; if there is a set of incorrect comparison results, it means that the final BWT sequence is incorrect.

[0151] Example 3

[0152] Combination Figure 6 An electronic device 1 includes a memory 11, a processor 10, and a computer program 12 stored in the memory 11 and executable on the processor 10. When the processor 10 executes the computer program 12, it implements any of the large text sequence BWT correctness verification methods in Embodiment 1.

[0153] For example, the computer program 12 may be divided into one or more units, which are stored in the memory 11 and executed by the processor 10 to complete the present application. The one or more units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 12 in the electronic device 1.

[0154] The electronic device 1 can be any electronic device with control functions. The electronic device 1 may include, but is not limited to, a processor 10 and a memory 11. Those skilled in the art will understand that... Figure 6 This is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 1 may also include input / output devices, network access devices, buses, etc.

[0155] The processor 10 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0156] The memory 11 can be an internal storage unit of the electronic device 1, such as a hard disk or RAM. The memory 11 can also be an external storage device of the electronic device 1, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 is used to store the computer program and other programs and data required by the terminal device. The memory 11 can also be used to temporarily store data that has been output or will be output.

[0157] Example 4

[0158] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements any of the large text sequence BWT correctness verification methods in Embodiment 1.

[0159] For example, the instruction can perform the following steps: Divide the string X and its suffix array SA(X) into multiple string blocks and multiple suffix buckets; use external storage block inductive sorting to sort and name the S* type substrings of the string X, generating a shrunken string; if the characters in the shrunken string are not unique, perform recursion; determine if the current recursion level is level 0; if the current recursion level is not level 0, calculate the L type predecessor character sequence blocks and S type predecessor character sequence blocks of the current level string according to the order of the S* type suffixes of the current level string, and save them to external storage; use external storage block inductive sorting to sort the L type suffixes and S type suffixes sequentially, generating the suffix array of the current level string; if the current recursion level is level 0, calculate the order of the S* type suffixes of the string X according to the suffix array of the shrunken string, calling a hash function iteratively to calculate descending order during the calculation process. The algorithm calculates the hash value of the S* type suffix sequence; it calculates the L type and S type predecessor character sequences for each block of string X and saves them to external storage. During the calculation, iteratively calculates the hash value of the two types of predecessor sequences for each block using a hash function; it calculates the L type BWT sequence using a min-heap, saves the L* type suffix sequence to external storage, and iteratively calculates the hash value of the predecessor block corresponding to each character in the L type BWT sequence using a hash function; it calculates the S type BWT sequence using a max-heap, iteratively calculating the hash value of the descending S* type suffix sequence and the predecessor block corresponding to each character in the S type BWT sequence using a hash function; it merges the L type and S type BWT sequences to generate the final BWT sequence; it compares the hash values ​​of the descending S* type suffix sequence, the hash values ​​of each block of the L type BWT sequence, and the hash values ​​of each block of the S type BWT sequence, and outputs the final verification result.

[0160] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0161] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0162] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0163] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the large text sequence BWT correctness verification methods provided in the embodiments of this application, the beneficial effects that any of the large text sequence BWT correctness verification methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0164] Example 5

[0165] This application also provides a computer program product, including a computer program. When executed by a processor, the computer program implements the steps in the large text sequence BWT correctness verification method of any one of Embodiment 1. The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form.

[0166] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0167] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0168] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0171] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium can be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0172] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0173] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for verifying the correctness of a large text sequence BWT, characterized in that, include: S1: Divide the string X and its suffix array SA(X) into multiple string blocks and suffix buckets; S2: Sort and name the S* type substrings of the string X to generate a condensed string; S3: Determine whether the characters in the shrunk string are unique; S4: When the characters in the shrunk string are unique, determine whether the current recursion level is level 0; S5: If the current recursion level is not level 0, then calculate the L-type predecessor character sequence block and the S-type predecessor character sequence block of the current level string according to the order of the S* type suffix of the current level string. S6: Sort the L-type suffixes and S-type suffixes to generate a suffix array for the current layer of strings; S7: If the current recursion level is level 0, calculate the order of the S* type suffixes of string X. During the calculation, call the hash function to iteratively calculate the hash value of the descending S* type suffix sequence. S8: Calculate the predecessor character sequence of type L and the predecessor character sequence of type S for each block of string X, save them to external storage, and call the hash function to iteratively calculate the hash value of the two types of predecessor sequences for each block; S9: Use a min-heap to calculate the L-type BWT sequence, save the L* type suffix sequence to external storage, and call the hash function to iteratively calculate the hash value of the predecessor block to which each character in the L-type BWT sequence belongs; S10: Use a max-heap to calculate the S-type BWT sequence, and call the hash function to iteratively calculate the hash value of the descending S* type suffix sequence and the corresponding predecessor block of each character in the S-type BWT sequence; S11: Merge L-type and S-type BWT sequences to generate the final BWT sequence; S12: Compare the hash values ​​of the descending S* type suffix sequence, the hash values ​​of each block of the L type BWT sequence, and the hash values ​​of each block of the S type BWT sequence, and output the final verification result.

2. The method for verifying the correctness of large text sequence BWTs according to claim 1, characterized in that, The i-th block is denoted as Bi. Step S8 includes the following steps: Calculate the order of S* type suffixes in the current block Bi based on the order of the S* type suffix sequence of string X; The block Bi is loaded into memory, and the S-type buckets of the suffix array SA(Bi) are initialized with the first character of the ordered S* type suffix. Scan the suffix array SA(Bi) to the right, and write the L-type predecessor characters into the L-type suffix buckets of the suffix array SA(Bi) and the L-type predecessor character block corresponding to the current block Bi in external storage in turn; during this process, whenever an L-type predecessor character is obtained, call the Karp-Rabin hash function to iteratively calculate the hash value of the L-type predecessor character sequence of the current block. Scan the suffix array SA(Bi) to the left, and write the S-type predecessor characters into the S-type suffix buckets of the suffix array SA(Bi) and the corresponding S-type predecessor character block in external storage for the current block Bi in turn; during this process, whenever an S-type predecessor character is obtained, call the Karp-Rabin hash function to iteratively calculate the hash value of the S-type predecessor character sequence of the current block.

3. The method for verifying the correctness of large text sequence BWTs according to claim 1, characterized in that, Step S9 includes the following steps: S901: Scan each suffix bucket in ascending order. For each suffix bucket, scan each suffix bucket in ascending order. Suppose that the current step is to scan the i-th suffix bucket, the min-heap H1 is empty, and the sequence number idx = 0. Sequentially push the suffixes of the i-th suffix bucket into the min-heap H1 from external storage. For each suffix pushed in, increment the sequence number idx of the suffix tuple by 1. S902: If the min-heap H1 is not empty, remove the top suffix e; if the first character e.chr of the suffix e belongs to the j-th bucket of the i-th suffix bucket block, then execute S904 on the suffix e until the first character of the suffix e does not belong to the j-th bucket or the min-heap H1 is empty. S903: Traverse the S* type suffix sequence of string X in ascending order, and take out the first suffix e in turn. If the first character e.chr of suffix e belongs to bucket j, then execute S904 on suffix e, until the first element of the S* type suffix sequence does not belong to bucket j or the S* type suffix sequence is traversed. S904: For each suffix 'e', ​​calculate the string X block k to which it belongs based on the suffix position 'e.pos', and iteratively calculate the hash value of the L-type BWT sequence block k using the Karp-Rabin hash function; extract the predecessor character 'pchar' of suffix 'e' from the head of the predecessor character sequence block k, and append the predecessor character 'pchar' to the tail of the L-type BWT sequence; calculate the suffix bucket 'q' to which the predecessor suffix 'e1' belongs based on the predecessor character 'pchar'; if the predecessor suffix 'e1' belongs to the current bucket i, then set e1 = ...<pchar, e.pos-1, idx++> Push it into the min-heap H1, otherwise set e1=<pchar, e.pos-1> Write the suffix to the secondary storage bucket q; if the lexicographical order of the predecessor character pchar is less than e.chr, then the suffix e is an L* type suffix, and set the suffix e = ...<e.chr,e.pos> Append to the end of the L* type suffix sequence.

4. The method for verifying the correctness of large text sequence BWTs according to claim 1, characterized in that, Step S10 includes the following steps: S1001: Scan each suffix bucket in descending order. For each suffix bucket, scan each bucket in descending order. Suppose that the current scan is of the i-th suffix bucket. Initialize the max heap H2 to be empty and the sequence number idx = n. Sequentially push the suffix of the i-th suffix bucket into the max heap H2 from external storage. Each time a suffix is ​​pushed, decrement the sequence number idx of the suffix tuple by 1. S1002: If the max-heap H2 is not empty, remove the top suffix e; if the first character e.chr of the suffix e belongs to the j-th bucket of the i-th suffix bucket block, then execute S1004 on the suffix e until the top element of the heap does not belong to the j-th bucket or the max-heap H2 is empty. S1003: Traverse the L* type suffix sequence of string X in descending order, and take out the first suffix e in turn; if the first character e.chr of suffix e belongs to bucket j, then execute S1004 on suffix e, until the first element of the L* type suffix sequence does not belong to bucket j or the L* type suffix sequence is traversed. S1004: For each suffix 'e', ​​based on the suffix position 'e.pos', calculate the block k of the string X to which the suffix 'e' belongs, and iteratively calculate the hash value of block k in the S-type BWT sequence using the Karp-Rabin hash function; extract the predecessor character 'pchar' of the suffix 'e' from the head of the predecessor character sequence block k, and append the predecessor character 'pchar' to the tail of the S-type BWT sequence; calculate the suffix bucket to which the predecessor suffix 'e1' of the suffix 'e' belongs based on the predecessor character 'pchar'; if 'e1' belongs to the current bucket i, then set 'e1' = ...<pchar,e.pos-1, idx--> Push it into the max-heap H2, otherwise set e1=<pchar, e.pos-1> Write the suffix to its corresponding bucket in external storage; if the lexicographical order of the predecessor character pchar is greater than e.chr, then the suffix e is an S* type suffix, and use the Karp-Rabin hash function to iteratively calculate the hash value of the S* type suffix sequence.

5. The method for verifying the correctness of large text sequence BWTs according to claim 1, characterized in that, Step S11 includes the following steps: Scan string X and count the number of L-type suffixes and S-type suffixes in each character bucket; Scan the L-type BWT sequence in ascending order, and extract the corresponding number of characters from each L-type character bucket according to the statistical data. Scan the S-type BWT sequence in ascending order, and extract the corresponding number of characters from each S-type character bucket according to the statistical data. The L-type and S-type BWT characters in each suffix bucket are merged sequentially to generate the final BWT sequence.

6. The method for verifying the correctness of a large text sequence BWT according to any one of claims 1 to 5, characterized in that: The comparison of the hash values ​​of the descending S* type suffix sequences refers to comparing whether the hash values ​​of the descending S* type suffix sequences in steps S7 and S10 are the same. Comparing the hash values ​​of each block of the L-type BWT sequence means comparing whether the hash values ​​of the L-type predecessor character sequence block in step S8 and the hash values ​​of the L-type BWT sequence block in step S9 are the same. Comparing the hash values ​​of each block of the S-type BWT sequence means comparing whether the hash values ​​of the S-type predecessor character sequence block in step S8 are the same as the hash values ​​of the S-type BWT sequence block in step S10.

7. A correctness verification device for a large text sequence BWT, characterized in that, include: The preprocessing module is used to divide the string X and its suffix array SA(X) into multiple blocks and suffix buckets respectively; The string shrinking module is used to calculate the order of substrings of type S* and name them sequentially to generate a shrunken string X1; The Determiner A module is used to determine whether the characters in the shrunk string X1 are unique. The Determiner B module is used to determine whether the current recursion level is level 0. The suffix predecessor character calculation module is used to calculate the predecessor character sequence blocks of the L-type and S-type suffixes of string X and save them to external storage; The suffix array calculation module is used to deduce the order of L-type and S-type suffixes using the external memory inductive sorting method, and generate the current layer suffix array; The module for calculating the hash value of the preceding character sequence of each block of a string is used to calculate the preceding character sequence blocks and their hash values ​​for L-type and S-type suffixes of strings, as well as the hash value of the descending S* type suffix sequence. The L-type BWT sequence block hash value calculation module is used to calculate the L-type BWT sequence. During the process, the hash function is called to iteratively calculate the hash value of the L-type BWT sequence block, and the L* type suffix sequence is saved to external storage. The S-type BWT sequence hash value calculation module is used to calculate the S-type BWT sequence. During this process, the hash function is called to iteratively calculate the hash value of each block of the S-type BWT sequence, as well as the hash value of the descending S* type suffix sequence. The BWT generation and correctness verification module is used to merge L-type and S-type BWT sequences, compare the hash values ​​of each block of the descending S* type suffix sequence and the string L-type and S-type BWT sequences, and output the final verification result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the correctness verification method for large text sequence BWT as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the correctness verification method for large text sequence BWT as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps in the correctness verification method for a large text sequence BWT as described in any one of claims 1 to 6.

Images