Index creation computer program, mapping computer program, computer for creating index, computer for mapping, and hierarchical index data structure
The hierarchical index solution addresses the memory resource limitations of BWA-MEM by optimizing the suffix tree structure for cache storage, enhancing processing speed and efficiency in genome analysis.
Patent Information
- Application Number
- JP2023086595
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Conventional BWA-MEM algorithms require large memory resources due to the size of the reference array index, leading to slow processing speeds because of frequent access to external memory, which limits the processing time even with high-performance computers.
A hierarchical index is created by setting a number of layers in a suffix tree, dividing it into upper and lower trees, and storing data in cache memory to reduce memory access time.
This approach allows for fast and efficient mapping processing by reducing memory access latency and enabling parallel processing, significantly decreasing the time required for human genome analysis.
Smart Images

Figure 2024169825000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a computer program for creating an index suitable for genome analysis, a computer program for mapping, an apparatus for executing these programs, and a data structure of a hierarchical index used in these programs. [Background technology]
[0002] The human genome is a set of genetic information that a person possesses, and the material that carries this information is DNA (deoxyribonucleic acid), which is a string of approximately 3 billion pairs of bases. The bases are adenine (A), guanine (G), cytosine (C), and thymine (T). In other words, a person's genetic information is determined by the arrangement (sequence) of these bases.
[0003] A device called a sequencer is used to read the human genome. The sequencer reads the sample human genome, cuts it into a specified upper limit (several hundred base pairs), amplifies it, and outputs a huge data sequence consisting of hundreds of millions of pieces of genetic information. Current sequencers can read one human genome in about an hour.
[0004] The individual pieces of data read by the sequencer are compared with the human genome reference sequence, which is defined as the standard genome sequence for humans, and are then reconstructed into the original length of the human genome sequence and analyzed. For example, the position of each piece of data is examined (mapped) in comparison with the human genome reference sequence. After mapping is completed, analysis is performed to determine what mutations are present. Since the pieces of data read from the sequencer usually contain errors, the errors are reduced by performing statistical processing using, for example, 30 times the amount of redundant data per sample, which is equivalent to the human genome sequence for one person. Therefore, analysis of the human genome sequence requires a huge amount of calculation, and therefore typically high-performance computers such as supercomputers, cluster computers, and FPGA (Field-Programmable Gate Array)-based computers are used.
[0005] For mapping data fragments, a program tool such as BWA (Burrow-Wheeler Aligner) is used. BWA consists of three algorithms, BWA-backtrack, BWA-SW, and BWA-MEM. Of these, BWA-MEM is widely used as a high-speed algorithm that supports Indel (insertion deletion). BWA-MEM is an algorithm that creates an index using a suffix sequence for a part of a query string based on a read data fragment that appears repeatedly in the human genome reference sequence, and performs mapping (Non-Patent Document 1).
[0006] The mapped data fragment is compared with a partial sequence at the same position in the human genome reference sequence to analyze what kind of mutation exists. For example, an alignment algorithm is used to compare the partial sequence of the human genome reference sequence with the data fragment. In the alignment algorithm, a degree of mutation (similarity) is calculated for each element of the sequence called an alignment table according to dynamic programming to identify an approximate character string. [Prior art documents] [Non-patent literature]
[0007] [Non-Patent Document 1] Heng Li, "Aligning sequence reads, clone sequences and assembly contigs with BWA-MEM", May 26, 2013, arXiv:1303.3997 (q-bio.GN) Summary of the Invention [Problem to be solved by the invention]
[0008] The conventional BWA-MEM described above is widely used as an algorithm that allows gaps and enables high-speed mapping, but has the problem that the size of the index created from the reference sequence becomes very large, requiring a large amount of memory resources. For example, in the case of analyzing the human genome, the amount of data for the index of the human genome reference sequence becomes enormous. This index needs to be stored in an external memory (main memory) because it cannot be accommodated at once in a cache memory that can be accessed at higher speeds. As a result, the external memory needs to be accessed a number of times according to the number of characters (i.e., the number of bases) in the query string to be searched, and the processing speed of the processor is limited by the time it takes to access the memory.
[0009] More specifically, in BWA-MEM, in order to identify the positions where the query strings read out in a fragmented state by the sequencer appear in the human genome reference sequence, random access to the main memory is performed the same number of times as the total number of characters in the query strings. Here, the latency of one random access to the main memory (the time from when the access occurs until the data is actually obtained) is approximately 1 μs (0.000001 sec) per bank on current computers. Therefore, in the case of analysis of the 3 billion base pair human genome, if the redundancy is 30, the total access time T relative to the total number of characters is: T=3,000,000,000×0.000001×30=90,000(seconds) In other words, in the analysis of the human genome, random access to main memory alone would take approximately 90,000 seconds (approximately 25 hours). For this reason, even if a computer equipped with a high-performance processor was used, there was a limit to how much the mapping time could be shortened due to the constraints imposed by memory access time.
[0010] An object of the present invention is to provide a technique for creating a hierarchical index based on a reference string, which is suitable for a mapping process. Another object of the present invention is to provide a technique for performing a mapping process quickly and / or efficiently, for example, to identify the position in a reference sequence where a piece of data (query string) read by a sequencer or the like appears. [Means for solving the problem]
[0011] The present invention, which aims to solve the above problems, includes the following invention-specific matters and technical features.
[0012] In other words, the present invention related to the above problem is a computer program for creating an index that enables a computing device to create an index to be used when identifying a substring in a reference string based on a query string, the computer program for creating an index being characterized in having a step of setting the number of hierarchical levels from the top level to a predetermined value in a suffix tree in which the suffix array of the reference string is hierarchically organized, thereby omitting hierarchical data below the predetermined value.
[0013] In relation to the above-mentioned index creating computer program, the predetermined value for the number of hierarchical levels may be set to 1000 or less.
[0014] In relation to the above-mentioned index creating computer program, the predetermined value for the number of levels may be set to be equal to or greater than the number of characters in the query string.
[0015] In relation to the above-mentioned computer program for creating an index, the program may further include a step of dividing the suffix tree at an intermediate level to generate a supertree including a level higher than the intermediate level, and a plurality of subtrees (hereinafter referred to as a group of subtrees) including a level lower than the intermediate level and connected to the lower side of the supertree.
[0016] In relation to the above-mentioned computer program for creating an index, the amount of data in the subtree may be set to a value that can be stored in a cache memory in the computing device.
[0017] In relation to the above-mentioned index creating computer program, the data amount of the subtree may be set to 10 MB or less.
[0018] The present invention, which relates to the above-mentioned problem, is a computer program for creating an index that enables a computing device to create an index used when identifying a substring in a reference string based on a query string, the computer program for creating an index comprising a step of dividing a suffix tree, which is a hierarchical arrangement of a suffix array of the reference string, at an intermediate level to create a supertree that includes a higher level than the intermediate level, and a plurality of subtrees (hereinafter referred to as a group of subtrees) that include a lower level than the intermediate level and are connected to the lower side of the supertree.
[0019] The present invention in relation to the above problem is a mapping computer program for causing a computing device to implement a method for identifying a substring in a reference string based on a query string, the mapping computer program including the above-mentioned index creation computer program as a part, characterized in that the method further includes mapping the query string to the reference string by referring to the hierarchical index in order to identify a substring in the reference string that matches at least a portion of the query string.
[0020] The present invention, which relates to the above-mentioned problem, is a mapping computer program for causing a computing device to realize a method for identifying a substring in a reference string based on a query string, wherein a hierarchical index generated based on the reference string is divided at an intermediate level to have a supertree including a level higher than the intermediate level and a plurality of subtrees (hereinafter referred to as a subtree group) including a level lower than the intermediate level and connected to a lower side of the supertree, the method including: referring to the hierarchical index to map the query string to the reference string in order to identify a substring in the reference string that matches at least a part of the query string, the mapping of the query string including a determination step of determining to which of the subtrees in the subtree group each of the query strings belongs by performing a matching process of the plurality of query strings using the supertree, and an appearance position determination step of determining an appearance position of the query string relative to the reference string by performing a matching process of the plurality of query strings that belong to a specific subtree determined by the determination step, using the specific subtree.
[0021] In relation to the above mapping computer program, the step of determining the occurrence positions in the subtree may be performed in parallel.
[0022] In relation to the above mapping computer program, the determining step may be performed in parallel.
[0023] In relation to the above mapping computer program, in the step of determining the occurrence position, the subtree may be expanded in a cache memory in the computing device.
[0024] In relation to the above mapping computer program, the computer program may be characterized in that it includes as a part thereof any one of the above index creation computer programs.
[0025] The present invention related to the above problem is a computer apparatus for creating an index, the computer apparatus having a computing device including a storage device, for creating an index used when identifying a substring in a reference string based on a query string, the computing device having a partial suffix tree generation means for reading out the reference string stored in the storage device, setting to a predetermined value the number of levels from the top level to lower levels in a suffix tree in which the suffix array of the reference string is hierarchically organized, and generating a partial suffix tree in which hierarchical data lower than the predetermined value is omitted.
[0026] The present invention related to the above problem is a computer apparatus for creating an index, which has a computing device including a storage device and creates an index used when identifying a substring in a reference string based on a query string, characterized in that the computing device has a suffix tree splitting means for splitting a suffix tree, which is a hierarchical arrangement of a suffix array of the reference string stored in the storage device, at an intermediate level to create a supertree including a higher level and a plurality of subtrees (hereinafter referred to as a group of subtrees) connected to the lower side of the supertree.
[0027] The present invention relating to the above problem is a mapping computer apparatus having a computing device including a storage device, and identifying a substring in a reference string based on a query string, wherein the storage device stores a supertree including a hierarchy higher than an intermediate hierarchy, which is generated by dividing a hierarchical index generated based on the reference string at an intermediate hierarchy, and a plurality of subtrees (hereinafter referred to as a subtree group) including a hierarchy lower than the intermediate hierarchy and connected to a lower side of the supertree, and the computing device has mapping means for mapping the query string to the reference string by referring to the supertree and the subtree in order to identify a substring in the reference string that matches at least a part of the query string, and the mapping means has: a determination means for determining which of the subtrees in the subtree group each of the query strings belongs to by performing a matching process on the plurality of query strings with reference to the supertree stored in the storage device; and an appearance position determination means for determining an appearance position of the query string relative to the reference string by performing a matching process on the plurality of query strings that belong to a specific subtree determined by the determination means with reference to the specific subtree stored in the storage device.
[0028] The present invention, which relates to the above problem, is a hierarchical index data structure of a reference string, used in a mapping computer program for enabling a computing device to realize a method for identifying a substring in a reference string based on a query string, the hierarchical index data structure being a suffix tree in which a suffix array of the reference string is hierarchically arranged, the number of levels from the top level to the lower levels being set to a predetermined value, and hierarchical data lower than the predetermined value being omitted.
[0029] In relation to the data structure of the above-mentioned hierarchical index, the hierarchical index may be characterized in that the suffix tree is split at an intermediate level, so that the hierarchical index has a data fragment of a supertree that includes a level higher than the intermediate level, and data fragments of multiple subtrees that include levels lower than the intermediate level and are connected to the lower side of the supertree.
[0030] The present invention, which relates to the above problem, is a data structure of a hierarchical index of a reference string, used in a mapping computer program for enabling a computing device to implement a method for identifying a substring in a reference string based on a query string, wherein the hierarchical index is divided at an intermediate level, and has a data piece of a supertree that includes a level higher than the intermediate level, and data pieces of multiple subtrees that include a level lower than the intermediate level and are connected to the lower side of the supertree.
[0031] The present invention can also be realized as a recording medium storing the computer program. The present invention can also be realized as an apparatus consisting of hardware and / or firmware configured to execute the method. Furthermore, when the index creation computer device, mapping computer device, etc. are realized by a plurality of hardware components (or functional modules that realize specific functions) logically working together, these devices become index creation computer systems and mapping computer systems. Furthermore, when the index creation computer device, mapping computer device, etc. are realized by a processor module, these devices become index creation processor modules and mapping processor modules. In other words, it does not matter whether each device or functional module is located in a single housing or whether it is realized by a single functional component.
[0032] In this specification, a means or unit does not simply mean a physical means, but also includes cases where the functions of the means or unit are realized by software. Also, the functions of one means or unit may be realized by two or more physical means, and the functions of two or more means or units may be realized by one physical means. Effect of the Invention
[0033] According to one aspect of the present invention, a hierarchical index based on reference strings can be created that is suitable for mapping operations, and this hierarchical index can provide a fast and / or efficient mapping operation.
[0034] Furthermore, according to the present invention, analysis can be performed quickly and / or efficiently using a human genome reference sequence based on data fragments read out by a sequencer.
[0035] Other technical features, objects, and operational effects or advantages of the present invention will become apparent from the following embodiments described with reference to the accompanying drawings. The effects described in this specification are merely examples and are not limited thereto, and other effects may also be present. [Brief description of the drawings]
[0036] [Figure 1] FIG. 1 is a block diagram showing an example of a schematic configuration of a computer system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a flowchart outlining an example of approximate string matching processing by the computer system. [Diagram 3] FIG. 3 is a flowchart illustrating an example of an index creation process performed by the computer system. [Figure 4] FIG. 4 is a diagram showing an example of a cyclic sequence generated from a reference character string used in the computer system. [Diagram 5] FIG. 5 is a diagram showing an example of the cyclic array after it has been sorted. [Figure 6] FIG. 6 is a diagram showing an example of the structure of a suffix tree generated based on a sorted cyclic array. [Figure 7] FIG. 7 is a diagram for explaining the process of generating a partial suffix tree using the same suffix tree. [Figure 8] FIG. 8 is a diagram showing an example of the structure of a partial suffix tree generated from the same suffix tree. [Figure 9] FIG. 9A is a diagram for explaining the process of dividing the partial suffix tree, and FIG. 9B is a diagram showing an example of a divided supertree. [Figure 10] FIG. 10 is a flowchart illustrating an example of a mapping process performed by the computer system. [Figure 11] FIG. 11 is a diagram showing an example of a query character string used in the computer system. [Figure 12] 12(A) to (D) are diagrams for explaining the mapping process by the computer system using a subtree. [Figure 13] 13(A) to (D) are diagrams for explaining the mapping process by the computer system using a subtree. [Figure 14] 14(A) to (D) are diagrams for explaining the mapping process by the computer system using a subtree. [Figure 15] FIG. 15 is a diagram showing a modification of the partial suffix tree. [Figure 16] FIG. 16 is a diagram for explaining a modified example when dividing the partial suffix tree. [Figure 17] FIG. 17 is a diagram illustrating an example of a hardware configuration of the computing device. [Figure 18] FIG. 18 is a diagram illustrating an example of a processor module in the hardware configuration of the computing device. [Figure 19] FIG. 19 is a block diagram showing an example of the functional configuration of the computing device. [Figure 20]FIG. 20 is a flowchart illustrating a modified example of the mapping process by the computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0037] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the embodiment described below is merely an example, and there is no intention to exclude various modifications and application of techniques not specified below. The present invention can be implemented by various modifications (for example, combining the respective embodiments) within the scope of the gist of the present invention. In addition, in the following description of the drawings, the same or similar parts are represented by the same or similar reference numerals. The drawings are schematic and do not necessarily correspond to actual dimensions, ratios, etc. The drawings may also include parts with different dimensional relationships and ratios.
[0038] FIG. 1 is a block diagram showing an example of a schematic configuration of a computer system according to an embodiment of the present invention. As shown in the figure, the computer system 1 includes, for example, a host computer 10, a plurality of lower computers 20(1)-20(n), and a database 30. The host computer 10 and the lower computers 20(1)-20(n) are communicatively connected via a predetermined interface. In the present disclosure, the computer system 1 is configured as a centralized computer system in which the host computer 10 comprehensively controls the plurality of lower computers 20(1)-20(n), but is not limited thereto and may be configured as, for example, a distributed computer system. In a distributed computer system, each computer can operate cooperatively by parallel distributed processing, but for a specific process, only one representative computer may execute the process.
[0039] The computing hardware common to the upper computer 10 and the lower computer 20 is a computing device 100 as exemplified in Fig. 17. The computing device 100 has a memory module 102 serving as a main storage device such as a DRAM, a processor module 104 serving as a central computer, a chipset 106 serving as a bridge between the processor module 104 and an external device, an internal storage device 108 such as an HDD or SDD, an I / O controller 110, an output interface 112, an input / output interface 114, a communication interface 116, and an external storage device 118 such as an external HDD.
[0040] 18, the processor module 104 includes a plurality of cores A, B, C, and D that execute independent calculation processes, first cache memories (L1 cache memories) L1a, L1b, L1c, and L1d and second cache memories (L2 cache memories) L2a, L2b, L2c, and L2d that are provided for each of the cores A, B, C, and D, and a third cache memory (L3 cache memory) L3n that is shared by all the cores A, B, C, and D. The access latency of the cores A, B, C, and D is shortest for the L1 cache memories L1a, L1b, L1c, and L1d, followed by the L2 cache memories L2a, L2b, L2c, and L2d, and longest for the L3 cache memory L3n. On the other hand, the access latency of the L1, L2, and L3 cache memories is significantly shorter than that of the memory module (main memory) 102. With these structures, the multiple cores A, B, C, and D in the processor module 104 can process tasks in parallel. For example, each of the multiple cores A, B, C, and D can perform analysis processing in comparison with a reference string based on a given individual query string. Here, a case where the cache memory is divided into three stages is illustrated, but the present invention is not limited to this, and may be one or two stages, or may be four or more stages. Hereinafter, the first cache memory (L1 cache memory) L1a, L1b, L1c, and L1d, the second cache memory (L2 cache memory) L2a, L2b, L2c, and L2d, and the third cache memory (L3 cache memory) L3n can be collectively referred to as "cache memory". In the present invention, the number of cores in the processor module 104 is not particularly limited. The number of cores can be selected from a variety of numbers, from several to tens of thousands.
[0041] 17 and 18 show examples of general-purpose hardware structures, but the present invention is not limited thereto. All or part of these hardware structures may employ hardware that is individually customized (programmed) by an FPGA or the like. Furthermore, hereinafter, the memory module 102, the internal storage device 108, the external storage device 118, the first cache memory (L1 cache memory) L1a, L1b, L1c, and L1d, the second cache memory (L2 cache memory) L2a, L2b, L2c, and L2d, and the third cache memory (L3 cache memory) L3n, which are hardware capable of storing data, may be collectively referred to as a "storage device."
[0042] Returning to FIG. 1, in the present disclosure, the lower computers 20 process tasks in parallel under the control of the upper computer 10. For example, each of the multiple lower computers 20 performs analysis processing in comparison with a reference string based on an individual given query string. Hereinafter, the lower computers 20(1) to 20(n) may be simply referred to as "lower computers 20" unless there is a particular need to distinguish between them.
[0043] The database 30 stores various data, such as reference character strings, under the control of the host computer 10. As an example, the database 30 stores a human genome reference sequence. The human genome reference sequence is data showing a sequence of base pairs defined as a standard genome sequence for humans. The database 30 also stores an index created based on the reference character string. The index is a type of data array structure used to search for the reference character string. In the figure, the database 30 is configured to be connected so as to be accessible only to the host computer 10, but this is not limited thereto, and the lower computer 20 may also be connected so as to be accessible. The lower computer 20 may also have a similar database. Here, a case where a human genome reference sequence is used is illustrated, but a metagenomic reference sequence that compiles multiple organisms (animals), plants, etc. may also be used.
[0044] 2 is a flowchart for explaining an example of an outline of approximate string matching processing by the computer system 1 according to an embodiment of the present invention. This processing is realized, for example, by the host computer 10 and the multiple lower level computers 20 executing an approximate string matching program in the processor module 102 in cooperation with other hardware components. As a result, the computer system 1, or the host computer 10, or the multiple lower level computers 20 function as an approximate string matching computer system 200 shown in FIG. 19. In addition, the approximate string matching program has multiple processing blocks (subroutines), and as a result, the approximate string matching computer system 200, in more detail, includes an index creation computer system 201, a mapping computer system 202, a string pair creation computer system 203, and an approximate string derivation computer system 204.
[0045] As shown in FIG. 2, first, the host computer 10 reads out the reference string stored in the database 30, and creates an index based on the read reference string (index creation process: S201). Specifically, the host computer 10 creates an index having a predetermined data array structure by expanding and / or sorting and arranging substrings in the reference string according to a predetermined condition. The data array structure is a hierarchical tree structure. In the present disclosure, such an index is referred to as a hierarchical index. The host computer 10 stores the created index based on the reference string in the database 30. The process of creating a hierarchical index based on the reference string will be described in detail later. Note that if a hierarchical index based on the reference string has already been created and stored in the database 30, the index creation process S201 may be omitted. In this way, the computer system 1 functions as an index creation computer system 201.
[0046] Next, the host computer 10 receives a query string from an external device (not shown), refers to the hierarchical index based on the received query string, and performs a mapping process of the query string to the reference string (mapping process: S202). In one example, the external device is a sequencer that reads out a human genome, and the query string is a data piece of about 150 to 200 base pairs output from the sequencer. The query string may be temporarily stored in the database 30. The number of characters in the query string is preferably 50 or more, more preferably 100 or more. On the other hand, the number of characters in the query string is preferably 1000 or less, desirably 300 or less, and more desirably 200 or less. In the present disclosure, the lower computer 20 also performs mapping by referring to the hierarchical index based on the assigned query string under the control of the upper computer 10. Thereby, the computer system 1 functions as a mapping computer system 202.
[0047] Mapping is a process of identifying a substring (matching string) in a reference string that matches at least a portion of a query string. That is, mapping identifies the start position and length (number of characters) of a substring in the reference string that matches at least a portion of the query string. In the mapping according to the present disclosure, a hierarchical index is searched or explored for the query string to identify the start position of the query string in the reference string. In order to speed up processing, the created hierarchical index is preferably expanded in a high-speed memory such as a cache memory in memory module 102 or processor module 104.
[0048] In the above example, the upper computer 10 and the lower computer 20 each perform mapping according to the assigned query string, but this is not limited to this. For example, only the upper computer 10 may perform a search by referring to the hierarchical index based on the assigned query string.
[0049] Next, the host computer 10 and the lower computer 20 each create a string pair for approximate string matching (string pair creation process: S203), which will be described later. The string pair is composed of a matched string in the reference string, including a matching string identified by mapping, and a matching string in the query string. In this way, the computer system 1 functions as a string pair creation computer system 203.
[0050] Specifically, the host computer 10 and the lower computer 20 create a matched string by adding a corresponding string of a predetermined length in the reference string to the beginning and end of the matching string identified by mapping. In other words, the matched string is a string in the vicinity of the matching string including the matching string identified in the reference string. For example, if the reference string is "CCGATCTGTATACCCTACGA" and the matching string is "TACC", the string "TATACCCT" obtained by adding two characters to the beginning and end becomes the matched string. In the case of human genome analysis, the length of bases added to the beginning and end of the matching string for the reference string may be, for example, about 50 bases.
[0051] In addition, the host computer 10 creates a comparison string by adding a corresponding string of a predetermined length in the query string to the end of the match string. In the case of analyzing the human genome, the length of the string to be added to the end of the match string for the query string is, for example, about 50 bases.
[0052] In the present disclosure, a reference string is added with a character string of a predetermined length to the beginning and end of a matching string, but this is not limited thereto, and for example, a character string of a predetermined length may be added only to either the beginning or the end. Also, a query string may be added with a character string of a predetermined length to the beginning and end of a matching string, or the matching string itself may be treated as a matching string without adding a character string.
[0053] Next, the upper computer 10 and the lower computer 20 derive at least one approximate string based on a string pair consisting of a matched string based on the reference string and a matched string based on the query string (approximate string derivation process: S204). A predetermined alignment using dynamic programming is applied to derive the approximate string. Alignment is a method of comparing two sequences (strings) while allowing substitution, insertion, and deletion between their elements, and calculating the degree of variation (similarity) according to a defined score / penalty. As an algorithm for realizing alignment, the Smith-Waterman algorithm and the Needleman-Wunsch algorithm are known. Dynamic programming is a method of calculating an optimal solution for the next stage based on an optimal solution obtained at a certain stage. That is, in the calculation of the degree of variation, a score is calculated for each element of an array-like alignment table according to dynamic programming, and an element having the maximum score is determined, thereby deriving at least one approximate string. In this way, the computer system 1 functions as a computer system 204 for deriving approximate strings.
[0054] As described above, in the approximate string matching process of this embodiment, a hierarchical index based on a reference string is created, and then a matching string (and its length) is identified by searching the hierarchical index according to a given query string, and an approximate string is derived by performing approximate string matching on a string pair consisting of a matched string and a match string based on the identified matching string. In other words, the approximate string matching program of this embodiment has an index creation program based on a reference string, a mapping program that searches the hierarchical index according to the query string, a program that creates a string pair consisting of a matched string and a match string, and an alignment program that derives an approximate string by performing approximate string matching using the string pair. Note that the program that creates the string pair and the alignment program can be publicly known, so detailed explanations will be omitted.
[0055] <Index creation program> Fig. 3 is a flowchart for explaining an example of index creation processing by the computer system according to this embodiment. That is, Fig. 3 shows details of the hierarchical index creation processing (S201) shown in Fig. 2. Fig. 19 also shows a functional block configuration of the index creation computer system 201, which has a reference string receiving means 302, a suffix tree generating means 304, a partial suffix tree generating means 306, and a partial suffix tree dividing means 308. Each functional block will be explained below with reference to the flowchart etc.
[0056] (Reference string receiving means 302) First, as shown in Fig. 3, the host computer 10 receives a reference character string for creating a hierarchical index (S302). For example, in the case of a human genome reference sequence, the host computer 10 accesses the database 30 and reads out the stored human genome reference sequence. For ease of understanding, the following description will be given on the assumption that the human genome reference sequence is a 20-character reference character string "CCGATCTGTATACCCTACGA" consisting of four types of characters "A", "C", "G", and "T". Note that the actual human genome reference sequence is 3 billion characters.
[0057] (Suffix tree generation means 304) Next, the host computer 10 generates a suffix tree for the received reference string (S304). Specifically, for example, "$" is added as a terminal character of the reference string. This terminal character is the smallest character that is not included in the reference string. The reference string, which has 21 characters including this terminal character, is rotated one character at a time to generate 21 cyclic strings (see FIG. 4). Note that in FIG. 4, the sequence number of the cyclic string is added to the beginning (left end) of each cyclic string. Each sequence number indicates the start position of appearance of each cyclic string based on the reference cyclic string number 0 (reference string). Note that when the actual human genome reference sequence is 3 billion characters, the number of cyclic strings (number of sequences) is 3 billion.
[0058] Furthermore, the host computer 10 cuts the strings after the terminal character "$" in the cyclic string (see the shaded area in FIG. 4), then sorts these cyclic strings (see FIG. 5) to generate a suffix tree expressed in a hierarchical tree structure. This suffix tree is shown in FIG. 6. The number of levels of the suffix tree is shown at the top of FIG. 6. In this embodiment, when the number of nodes on the lower level side of the tree structure becomes 1, the remaining strings are grouped together at their terminal nodes (leaves). As a result, the number of levels of the suffix tree in this embodiment is 3, including the terminal nodes (leaves). In the case of an actual human genome reference sequence, the number of levels is much larger (for example, about 20 levels). The above procedure for generating a suffix tree is a reference example, and the present invention is not limited to this. The program for generating a suffix tree can adopt various known methods.
[0059] (Partial suffix tree generating means 306) Next, the host computer 10 generates a partial suffix tree from the generated suffix tree (S306). Specifically, as shown in the shaded area in FIG. 7, hierarchical data on the lower hierarchical side than a predetermined level set in advance is forcibly cut. The suffix tree whose lower hierarchical side has been cut is referred to as a partial suffix tree, and the predetermined level is referred to as the cut level. In this embodiment, for convenience of explanation, the cut level is 2, and hierarchical data on the lower hierarchical side is cut. The partial suffix tree is shown in FIG. 8. In the partial suffix tree in FIG. 8, when multiple characters are stored in a terminal junction (leaf), the first character (or a predetermined character from the beginning) is left and the subsequent character string is cut, thereby simplifying the tree structure. In the suffix tree, information on the position and length of the cyclic character string that matches each node of the tree is required, but by cutting the level, it is possible to replace this with the character itself without significantly increasing the amount of data, thereby speeding up the search.
[0060] In the partial suffix tree of Figure 8, each array number of the cyclic string is added to the right end of the terminal node (leaf) of the partial suffix tree. For example, in the case of the root(#)-CGA system, since there are two nodes (branches) lower down in the original suffix tree (see Figure 7), "root(#)-CGA-$" and "root(#)-CGAT...", the array number 17, which is the upper (older brother) node in the sorting order, and the array number 1, which is the lower (younger brother) node, are added to the right of the leaf of the root(#)-CGA of the partial suffix tree, in that order. For example, in the case of the root(#)-TAC lineage, since there are two nodes (branches) lower down in the original suffix tree (see Figure 7), "root(#)-TACC..." and "root(#)-TACG...", the upper (older brother) node in the sort order, sequence number 10, and the lower (younger brother) node, sequence number 15, are added to the right of the leaf of root(#)-TAC in the partial suffix tree, in that order. In these cases, this means that the occurrence positions of the identified query string are identified in two places.
[0061] In the case of an actual human genome reference sequence, the number of layers of the (whole) suffix tree is enormous. In this case, the value of the cut layer number set to create a partial suffix tree is preferably 1000 or less, desirably 500 or less, more preferably 300 or less, and even more preferably 200 or less. On the other hand, the value of the cut layer number is preferably 30 or more, desirably 50 or more, and even more preferably 80 or more. In addition, the value of the cut layer number set is preferably set to the number of characters of the query string or more. For example, when the query string is 150 base pairs, the cut layer number is set to 150 or more. Specifically, when the cut layer number of the human genome reference sequence is set to 150, the (whole) suffix tree (data volume of about 250 GB) is abbreviated to a partial suffix tree of 150 layers (data volume of about 100 GB), so the amount of data can be significantly reduced.
[0062] In the simplified example of Figure 8, of the 19 terminal nodes (leaves) in the partial suffix tree, two, the "root(#)-CGA" system and the "root(#)-TAC" system, are assigned multiple sequence numbers. However, in the case of a partial suffix tree created based on the human genome reference sequence, the enormous number of levels is reduced to 150 levels, and as a result, each terminal node (leaf) is assigned multiple sequence numbers, but this number rarely exceeds 3. In genome searches, since there is a high probability of mutation in reality, narrowing down the sequence number to one will often result in matching to the wrong location. For this reason, in order to improve the accuracy of mapping, it is desirable to cut the tree at the stage where multiple sequence numbers remain.
[0063] Furthermore, in the above embodiment, the partial suffix tree is generated through two means, the suffix tree generating means 304 and the partial suffix tree generating means 306, but the procedure for generating the partial suffix tree is not limited to this. Any process for generating a partial suffix tree using a received reference string can be used.
[0064] (Partial suffix tree division means 308) Next, the upper computer 10 divides the partial suffix tree at the intermediate hierarchies (intermediate junctions) to create a supertree including the upper hierarchies and a plurality of subtrees (hereinafter referred to as a group of subtrees) connected to the lower side of the supertree (S308). The division condition of the intermediate junctions that makes the division valid is, for example, "when tracing the intermediate junctions from the terminal junction to the upper hierarchies, the number of terminal junctions (leaves) belonging to the lower hierarchies than the intermediate junction is equal to or less than a predetermined value." For example, FIG. 9(A) shows a case where the division is made when "when tracing the intermediate junctions from the terminal junction to the upper hierarchies, the number of terminal junctions (leaves) belonging to the lower hierarchies than the intermediate junction is equal to or less than two values." Here, the partial suffix tree is divided into one supertree including the root (#) and 14 subtrees connected to the lower side of this supertree. In addition, data that becomes a management number is assigned to each subtree. The rule for assigning this management number is arbitrary, but for example, the eldest sibling number (the highest number in the sorting order) in the array number belonging to the subtree can be used.Specifically, the subtrees are sorted in the following order: the first subtree of control number 20 split at the junction of "root(#)-$", the second subtree of control number 19 split at the junction of "root(#)-A-$", the third subtree of control number 11 split at the junction of "root(#)-AC", the fourth subtree of control number 9 split at the junction of "root(#)-AT", the fifth subtree of control number 12 split at the junction of "root(#)-CCC", the sixth subtree of control number 0 split at the junction of "root(#)-CCG", the seventh subtree of control number 13 split at the junction of "root(#)-CCT", the eighth subtree of control number 17 split at the junction of "root(#)-CG". The following subtrees are generated: the 9th subtree of management number 14 split at the junction of "root(#)-CT", the 10th subtree of management number 18 split at the junction of "root(#)-GA-$", the 11th subtree of management number 2 split at the junction of "root(#)-GAT", the 10th subtree of management number 18 split at the junction of "root(#)-GA-$", the 11th subtree of management number 7 split at the junction of "root(#)-GT", the 12th subtree of management number 10 split at the junction of "root(#)-TA", the 13th subtree of management number 4 split at the junction of "root(#)-TC", and the 14th subtree of management number 6 split at the junction of "root(#)-TG". The amount of data of the data pieces of each subtree of the subtree group is preferably set to a size that fits in the cache memory (especially the L1 cache memory) of the processor module 104. In other words, it is preferable that the "number (predetermined value) of terminal nodes (leaves) belonging to a lower hierarchical level" in the splitting condition of an intermediate node for which splitting is established is set taking into consideration the capacity of the cache memory.
[0065] Figure 9(B) shows the partial suffix tree in a state where the supertree, the split node (the terminal node in the supertree), and the subtree management number remain. In this way, the data piece of the supertree is separated from the subtree, and the amount of data is greatly reduced. It is preferable that the data amount of this supertree be such that it can fit in a memory module (main memory), and more preferably, it is of a size that can fit in cache memory. As a result of the above, hierarchical index data is completed in which the partial suffix tree is divided into a supertree data piece and a group of subtree data pieces.
[0066] (Data structure of hierarchical index data) As shown in FIG. 9A, the data structure of the hierarchical index data generated by this embodiment is a suffix tree structure in which the suffix array of the reference string is hierarchically arranged, and the number of layers from the top layer to the lower layers is set to a predetermined value, so that the hierarchical data below the predetermined value is omitted. As a result, it is possible to reduce the amount of data of the hierarchical index data. In particular, in this embodiment, although the data structure has a number of layers proportional to the length of the reference string in the worst case, by setting the predetermined value to 1000 or less, a partial suffix tree with 1000 layers or less is used, so that the amount of data can be dramatically reduced. In the currently popular shotgun sequence method, the number of characters of the query string output from the sequencer is about 150 to 200, so that even a partial suffix tree with 1000 layers or less can be sufficiently mapped, so that it is possible to rationally achieve both reduction in data amount and maintenance of mapping quality.
[0067] Furthermore, the data structure of the hierarchical index data generated by this embodiment is structured such that the partial suffix tree is divided at an intermediate level, and includes a data piece of an upper tree including a level higher than the intermediate level, and data pieces of multiple subtrees including a level lower than the intermediate level and connected to the lower side of the upper tree. The expression "divided into data pieces" means that the computer can store the data pieces in the memory module 102 (storage device) as separate data, or the processor module 104 can process the data pieces as separate data. This reduces the data amount of the upper tree data piece and the data amount of each subtree data piece, so that they can be stored in the memory module 102, and further, each data piece can be expanded in its entirety in the cache memory. As a result, when a so-called general-purpose computer is used, when the processor module 104 processes the mapping process described later in parallel, the probability of a search process hitting in the cache memory becomes extremely high, and high-speed mapping can be performed.
[0068] In this embodiment, the case where the number of division stages existing in the middle of the hierarchy from the terminal node (leaf) in the partial suffix tree toward the root (root (#)) is 1 is illustrated, but the present invention is not limited to this. For example, as shown in FIG. 16, a multi-stage division may be performed by providing a plurality of division nodes (gray nodes) in the middle of the hierarchy from the terminal node (leaf) toward the root (root (#)). In FIG. 16, the division is performed when "the number of terminal nodes (leaves) belonging to a lower hierarchy than itself at the intermediate node passed when going back from the terminal node (leaf) to the root (root (#)) is 3 or less". Furthermore, for the subtree divided at this division node, the total number of nodes is virtually converted to 1, and then, further, the nodes are passed back from the division node toward the upper hierarchy, and similarly, an additional division is performed when "the number of terminal nodes (leaves) belonging to a lower hierarchy than itself at the intermediate node passed is 3 or less".
[0069] For a partial suffix tree reduced to 150 hierarchical levels based on the human genome reference sequence, the division condition "when tracing back intermediate junctions from an end junction to a higher hierarchical level, the number of end junctions (leaves) belonging to a lower hierarchical level than the intermediate junction is equal to or less than a predetermined value" is preferably set within a range of 30,000 to 300,000. Desirably, the predetermined value is set to 50,000 to 200,000. For example, when the predetermined value is set to 30,000, the tree can be divided into one top tree including the root (root (#)) and about 1 million subtrees belonging to this top tree. For example, when the predetermined value is set to 100,000, the tree can be divided into one top tree including the root (root (#)) and about 300,000 subtrees belonging to this top tree. For example, when the predetermined value is set to 30,000, the tree can be divided into one top tree including the root (root (#)) and about 100,000 subtrees belonging to this top tree. Under these conditions, the data size of the top-level tree will be 10 MB or less (for example, about 1 MB), a size that can be fully expanded in the memory module 102 (main memory). Furthermore, under these conditions, the data amount of each subtree will be 10 MB or less, preferably 1 MB or less, a size that can be fully expanded in the memory module 102 (main memory), and furthermore, a size that can be stored in the cache memory.
[0070] <Mapping program> FIG. 10 is a flowchart for explaining an example of mapping processing by the computer system according to this embodiment. That is, FIG. 10 shows details of the mapping processing (S202) of the query string to the reference string shown in FIG. 2. By this mapping processing, the query string is mapped to the reference string, and the appearance start position in the reference string is identified. Also, FIG. 19 shows a functional block configuration of the mapping computer system 202, which has a higher-level tree expanding means 402, a query string group receiving means 404, a query string group expanding means 405, a query string group determining means 406, a first repeating means 407, a lower-level tree and query string group set expanding means 408, a memory remaining amount determining means 409, a query string appearance position determining means 410, and a second repeating means 412. Each functional block will be explained below with reference to a flowchart or the like. Note that although the mapping processing by the higher-level computer 10 will be explained below, the mapping processing by the lower-level computer 20 operating in parallel is similar.
[0071] (Upper tree expansion means 402) 10, the host computer 10 reads out the upper tree in the hierarchical index from the database 30, and spreads and stores it on the memory module 102 (S402). From the viewpoint of high speed, the host computer 10 spreads and stores the data constituting the upper tree contiguously on the memory module 102. Here, spreading contiguously includes arranging the data at contiguous memory addresses in the same bank.
[0072] (Query string group receiving means 404) Next, the host computer 10 receives a plurality of query strings (query string group) for mapping to the reference string (S404). The received query string is temporarily stored in a storage means such as the database 30. For example, if the reference string is a human genome reference sequence, the query string is a data fragment of a base pair read from a sequencer. In the following, for ease of understanding, as shown in FIG. 11, the query string is described as having three characters and the number of the query strings is 40. However, it should be noted that the query string of the data fragment of a base pair read from an actual sequencer is about 150 characters, and the number of the query strings is about 700 million. Note that, although an example is shown here of a case where the query string group is temporarily stored in the database 30, if the sequencer itself, which is an external device, has a database function, the database on the sequencer side may be used as it is.
[0073] (Query string group expansion means 405) Next, the host computer 10 reads out the query string group stored in the database 30, and expands and stores it on the memory module 102 (S405). From the viewpoint of high speed, the host computer 10 continuously expands and stores the data constituting the query string group on the memory module 102. However, since the total data amount of the query string group is large and expansion of the upper tree to the memory module 102 by the upper tree expansion means 402 needs to be prioritized, only a part of the query string group is expanded to the memory module 102 of the host computer 10.
[0074] (Query string group determination means 406) Next, the host computer 10 extracts each query string from the group of query strings expanded in the memory module 102, and determines which subtree (subtree management number) the query string belongs to by referring to the supertree (FIG. 9(B)) expanded in the same memory module 102 (S406). As a result, the subtree management number associated with each query string is determined, and the data is stored in the memory module 102. As a result, as shown in FIG. 11, each of the 40 query strings is stored in the memory module 102 with the subtree management number to which it belongs. The subtree management number data associated with each of the group of query strings is finally stored in the database 30. Note that, as shown in FIG. 18, when the processor module has four cores A, B, C, and D, it is preferable to speed up the above determination process by processing them in parallel.
[0075] (First repeating means 407) As described in the query string group expansion means 405, only a part of the query string group can be expanded in the memory module 102 of the host computer 10 due to its upper capacity limit. Therefore, the host computer 10 refers to the database 30 and determines whether or not there are any undetermined query string groups remaining in step S406 (S407). If there are any undetermined query string groups in the database 30, the process returns to step S405, where the remaining query string groups are expanded and stored in the memory module 102, and the subtree management number associated with each query string is determined (S406).
[0076] On the other hand, if there are no undetermined query strings in step S407, all query strings have been grouped by subtree, and the process proceeds to step S408. In the query strings shown in Fig. 11, the shaded strings indicate strings that were not used when determining with reference to the supertree.
[0077] (Subtree and query string group set expansion means 408) Next, the host computer 10 reads out a specific subtree in the hierarchical index from the database 30, and expands it on the main memory module 102. Furthermore, the host computer 10 also expands and stores on the same memory module 102 a group of query strings that belong to the same subtree management number as the expanded specific subtree. As a result, the specific subtree and the group of query strings belonging to it are expanded as a set in the memory module 102 (S408). There are various methods for expanding the group of query strings in the memory module 102. For example, the group of query strings may be arranged in an array by connecting a previous query string and a subsequent query string with a pointer so that the query strings in the group of query strings are successively read out by the processor module 104.
[0078] (Memory remaining capacity determination means 409) Furthermore, the host computer 10 refers to the remaining capacity of the memory module 102, and if there is still free capacity available to expand another specific subtree and a query string group belonging to the same subtree management number as the subtree, the process returns to step S408 and expands the other specific subtree and query string group as a set. As a result, multiple sets of subtrees and query string groups are expanded simultaneously in the memory module 102. In this embodiment, four sets of subtrees and query string groups with subtree management numbers 11, 9, 12, and 0 are expanded in the memory module 102.
[0079] (Query string appearance position determination means 410) Next, the host computer 10 sequentially extracts each query string from the set-expanded query string group, searches the subtree to which it belongs, and identifies which array element (appearance position) each query string ultimately belongs to (S410).
[0080] More specifically, for example, if a processor module has four cores A, B, C, and D, the identification process of four sets of subtrees and query strings with subtree management numbers 11, 9, 12, and 0 is assigned to each of the cores A, B, C, and D, and parallel processing is performed. Since the data amount of each subtree is preset to be equal to or less than the capacity of the cache memory, as shown in FIG. 12(A) to (D), in the identification process, each subtree data is expanded on four L1 cache memories L1a, L1b, L1c, and L1d. As a result, the search process in each subtree in each of the cores A, B, C, and D is completed within the L1 cache memories L1a, L1b, L1c, and L1d, realizing high-speed processing. In addition, for convenience of explanation, in FIG. 12(A) to (D), the number of layers is indicated at the top of the top of each subtree, and the management number of each subtree is attached to the left of each subtree. Also, on the left side of each subtree, the array number (appearance position) of each terminal node (leaf) of the subtree is shown. On the right side of each query string identified by cores A, B, C, and D, the final array number (appearance position) is shown. Note that shaded characters in the query string indicate characters that were not used when determining the appearance position by referring to the subtree (characters used in determining the superior tree). In this step S410, the appearance positions are identified for all subtrees and query strings set and expanded in memory module 102. Data related to the identification results is stored in database 30.
[0081] (Second repeating means 412) Next, the host computer 10 judges whether or not any unprocessed sets of subtrees and query strings remain in the database 30 (S412). In FIG. 12, four sets of subtrees and query strings out of the 16 have been processed, so the process returns to step S408, and the four sets of subtrees and query strings with subtree management numbers 13, 17, 14, and 18 are expanded and stored in the memory module 102 as sets, as shown in FIG. 13(A) to (D). After that, the four cores A, B, C, and D refer to the subtrees to which each query string belongs, in sets, to identify which array number (appearance position) each query string ultimately belongs to (S410). Furthermore, it is judged whether or not any unprocessed sets of subtrees and query strings remain (S412). Since there are still four sets of unprocessed subtrees and query strings, the process returns to step S408, and as shown in Figures 14(A) to (D), the four sets of subtrees and query strings with subtree management numbers 7, 10, 4, and 6 are expanded and stored in memory module 102 as sets, and the four cores A, B, C, and D refer to the subtrees to which each query string belongs, set by set, to identify which array element (appearance position) each query string ultimately belongs to (S410). Next, it is determined whether any sets of unprocessed subtrees and query strings remain (S412). Since there are no more unprocessed sets, the mapping process is terminated.
[0082] The processing of S408 to S412 is performed in parallel by the four cores A, B, C, and D in the processor module 104 of the host computer 10, so that the processing time can be significantly reduced. Although not shown in the figure, the mapping process can also be simultaneously performed (distributed processing) by a plurality of lower computers 20(1) to 20(n) to further reduce the processing time.
[0083] 10, when unprocessed query strings remain in step S407, information on the processed query strings is written to the database 30 and the memory module 102 is released, and then the unprocessed query strings are read from the database 30 to the memory module 102 to determine the subtree management number associated with each query string (S406). Then, after step S407, the determination process for all query strings in the database 30 is completed, and the process proceeds to steps S408 to S412 to perform the identification process. However, the present invention is not limited to this. For example, as shown in FIG. 20, when the determination process is completed in step S406 for the query strings expanded in the memory module 102 in step S405, the process proceeds directly to step S408, and all subtrees corresponding to the subtree management numbers determined in the determination process are imported and set in a set expansion state, and the identification process can be performed in steps 410 and 412. Thereafter, in step S407, it may be determined whether or not there is an unprocessed query string, and if there is an unprocessed query string, the process may return to step S405.
[0084] As described above, in the mapping computer program of this embodiment, the host computer 10 uses data pieces of the host tree with a reduced data amount to determine in advance which subtrees multiple query strings belong to, thereby achieving high-speed processing. Furthermore, after the determination is completed, the subtrees and the query strings belonging thereto are expanded as a set, and the cores A, B, C, and D of the processor module 104 continuously search for multiple query strings belonging to the specific subtree, thereby continuously identifying the positions where query strings appear. In the identification process, each subtree is taken into the cache memory, so that the number of random accesses to the memory module 102 can be significantly reduced.
[0085] In the above mapping process, a reference string that matches the entire query string is identified, but the present invention is not limited to this. For example, a partial string may be extracted from the query string, and the mapping process may be performed on this partial string.
[0086] <Modification of hierarchical index data structure> The suffix tree used in the hierarchical index in the above embodiment is a suffix tree that is generated by hierarchizing (forming a tree) each character for a plurality of cyclic strings (see FIG. 5) obtained by cycling a reference string having ACGT one character at a time, as shown in FIG. 6, but the present invention is not limited to this. A suffix tree may be generated by forming a hierarchical suffix tree by forming a set of multiple characters for a cyclic string (see FIG. 5). FIG. 15 shows a case in which a string pair is formed in units of two characters for a cyclic string (see FIG. 5) and a suffix tree is generated from this string pair. In this way, the number of levels is reduced, so that the amount of data such as partial suffixes, supertrees, and subtrees can be further reduced. On the other hand, if five or more characters are hierarchized as one set, the number of nodes increases, and the amount of data may increase. Therefore, when a suffix tree is generated by forming a set of multiple strings, it is preferable that the number of characters in the set is about 2 to 4.
[0087] The above-mentioned embodiments are merely examples for explaining the present invention, and are not intended to limit the present invention to these embodiments. The present invention can be implemented in various forms without departing from the spirit of the present invention. For example, the present embodiment illustrates a case where a hierarchical index is created or mapping is performed for a reference character string that is genetic information, but the present invention is not limited to this, and can be applied to a case where a hierarchical index is created or mapping is performed for a reference character string other than genetic information.
[0088] Also, for example, in the methods disclosed herein, steps, operations, or functions may be performed in parallel or in a different order, unless the results are inconsistent. The steps, operations, and functions described are provided merely as examples, and some of the steps, operations, and functions may be omitted or combined into one, or other steps, operations, or functions may be added, without departing from the spirit of the invention.
[0089] Furthermore, in the above embodiment, the hardware configuration of the computing device is a general-purpose structure, but the present invention is not limited to this. For example, hardware that is individually customized using an FPGA (Field Programmable Gate Array) or the like may be adopted. Specifically, all or part of the functional blocks shown in FIG. 19 can be realized by an FPGA.
[0090] Furthermore, the detailed expression method (notation method) for implementing the hierarchical index data structure shown in the above embodiment is not particularly limited. For example, if the tree structure is expressed by the LOUDS (Level-Ordered Unary Degree Sequence) method, it is possible to further reduce the memory amount.
[0091] In addition, although various embodiments are disclosed in this specification, specific features (technical matters) in one embodiment can be added to or replaced with specific features in other embodiments, with appropriate modifications, and such forms are also included in the spirit of the present invention. [Explanation of symbols]
[0092] 1. Computer Systems 10 Top Computers 20 Lower level computers 30 Databases
Claims
1. 1. A computer program product for indexing that causes a computing device to create an index for use in identifying substrings in a reference string based on a query string, the computer program product comprising: a step of setting a number of layers from the top layer to a lower layer in a suffix tree in which the suffix array of the reference string is hierarchically arranged to a predetermined value, thereby omitting hierarchical data lower than the predetermined value. A computer program for creating indexes.
2. The predetermined value of the number of hierarchical levels is set to 1000 or less.
2. A computer program for creating an index according to claim 1.
3. The predetermined value of the number of levels is set to be equal to or greater than the number of characters in the query string.
2. A computer program for creating an index according to claim 1.
4. The method further comprises a step of dividing the suffix tree at an intermediate level to generate a supertree including a level higher than the intermediate level and a plurality of subtrees including a level lower than the intermediate level and connected to a lower side of the supertree (hereinafter referred to as a subtree group).
2. A computer program for creating an index according to claim 1.
5. The amount of data in the subtree is set to a value that can be stored in a cache memory in the computing device.
5. A computer program for creating an index according to claim 4.
6. The data amount of the subtree is set to 10 MB or less.
5. A computer program for creating an index according to claim 4.
7. 1. A computer program product for indexing that causes a computing device to create an index for use in identifying substrings in a reference string based on a query string, the computer program product comprising: The method further comprises a step of dividing a suffix tree, which is a hierarchical tree of a suffix array of the reference character string, at an intermediate level to create a supertree including a level higher than the intermediate level, and a plurality of subtrees (hereinafter referred to as a subtree group) including a level lower than the intermediate level and connected to the lower side of the supertree. A computer program for creating indexes.
8. 1. A computer program product for mapping, for causing a computing device to implement a method for identifying a substring in a reference string based on a query string, the computer program product comprising: The computer program for creating an index according to any one of claims 1 to 7 is included as a part of the computer program, The method further comprises: and referencing the hierarchical index to map the query string to the reference string to identify a substring in the reference string that matches at least a portion of the query string. A computer program for mapping.
9. 1. A computer program product for mapping, for causing a computing device to implement a method for identifying a substring in a reference string based on a query string, the computer program product comprising: a hierarchical index generated based on the reference character string is divided at an intermediate level to have a supertree including a level higher than the intermediate level, and a plurality of subtrees (hereinafter referred to as a subtree group) including a level lower than the intermediate level and connected to a lower side of the supertree; The method comprises: referencing the hierarchical index to map the query string to the reference string to identify substrings in the reference string that match at least a portion of the query string; Mapping the query string comprises: a determining step of determining to which subtree in the group of subtrees each of the query strings belongs by performing a matching process on the plurality of query strings using the supertree; and performing a matching process using the specific subtree for the plurality of query strings belonging to the specific subtree determined by the determining step, thereby determining an appearance position of the query string relative to the reference string. A computer program for mapping.
10. The step of determining the occurrence positions in the subtree is performed in parallel.
10. A computer program for mapping according to claim 9.
11. The discrimination step is performed in parallel.
10. A computer program for mapping according to claim 9.
12. In the step of determining the location of occurrence, the subtree is expanded in a cache memory in the computing device.
10. A computer program for mapping according to claim 9.
13. A computer program for creating an index according to any one of claims 1 to 7, 10. A computer program for mapping according to claim 9.
14. 1. An indexing computer apparatus having a computing device including a storage device, the indexing computer apparatus configured to generate an index for use in identifying substrings in a reference string based on a query string, the indexing computer apparatus comprising: The computing device comprises: a partial suffix tree generating means for reading out the reference string stored in the storage device, setting a predetermined value to the number of layers from the highest layer to a lower layer in a suffix tree in which a suffix array of the reference string is hierarchically arranged, and generating a partial suffix tree in which data on the lower layer than the predetermined value is omitted. Indexing computer equipment.
15. 1. An indexing computer apparatus having a computing device including a storage device, the indexing computer apparatus configured to generate an index for use in identifying substrings in a reference string based on a query string, the indexing computer apparatus comprising: The computing device comprises: The system further comprises a suffix tree division means for dividing a suffix tree, which is a hierarchical tree of the suffix array of the reference character string stored in the storage device, at an intermediate level to create a supertree including a higher level and a plurality of subtrees (hereinafter referred to as a subtree group) connected to the lower side of the supertree. Indexing computer equipment.
16. 1. A mapping computer apparatus having a computing device including a storage device, the mapping computer apparatus identifying a substring in a reference string based on a query string, the mapping computer apparatus comprising: the storage device stores a supertree including a hierarchy higher than the intermediate hierarchy, the supertree being generated by dividing a hierarchical index generated based on the reference character string at an intermediate hierarchy, and a plurality of subtrees (hereinafter referred to as a subtree group) including a hierarchy lower than the intermediate hierarchy and connected to a lower side of the supertree; The computing device comprises: a mapping means for mapping the query string to the reference string by referencing the supertree and the subtree to identify a substring in the reference string which matches at least a portion of the query string; The mapping means comprises: a determination means for determining to which subtree in the group of subtrees each of the query strings belongs by performing a matching process on the plurality of query strings with reference to the supertrees stored in the storage device; and an appearance position determination means for performing a matching process with respect to the plurality of query strings belonging to the specific subtree determined by the determination means, by referring to the specific subtree stored in the storage device, thereby determining an appearance position of the query string with respect to the reference string. Computer equipment for mapping.
17. 1. A data structure of a hierarchical index of a reference string for use in a mapping computer program for implementing on a computing device a method for identifying substrings in a reference string based on a query string, the data structure comprising: a suffix tree in which the suffix array of the reference string is hierarchically organized, the number of layers from the top layer to the lower layers is set to a predetermined value, and data of layers lower than the predetermined value is omitted. A hierarchical index data structure.
18. the hierarchical index is characterized in that the suffix tree is divided at an intermediate level, and thus the hierarchical index has a data piece of a supertree including a level higher than the intermediate level, and data pieces of a plurality of subtrees including levels lower than the intermediate level and connected to the lower side of the supertree.
17. The hierarchical index data structure of claim 16.
19. 1. A data structure of a hierarchical index of a reference string for use in a mapping computer program for implementing on a computing device a method for identifying substrings in a reference string based on a query string, the data structure comprising: the hierarchical index is divided at an intermediate level to have a data piece of a supertree including a level higher than the intermediate level, and data pieces of a plurality of subtrees including a level lower than the intermediate level and connected to the lower side of the supertree, A hierarchical index data structure.
