Character string collation device, character string collation method, and program

Abstract

A character string collation device which generates an output collation result that is a result of collating one or more designated character strings for input text data that is a sequence of code words encoded using a plurality of code books, said character string collation device being configured in a manner such that a code book-specific collation state is defined by a combination of a collation degree indicating up to which character a match can be verified for which character string among the one or more designated character strings, and code book information indicating which of the plurality of code books a given book is, and so as to comprise: a state transition rule generation unit that uses one or more designated character strings and a plurality of code books to generate, for each code book-specific collation state, a state transition rule indicating a code book-specific collation state to which a transition is to occur, and a collation result to be outputted when inputting a code word included in the code book indicated by the code book information used to define the code book-specific collation state at the collation degree used to define the code book-specific collation state; and a collation unit that uses the state transition rule to generate an output collation result from input text data.

Description

String matching device, string matching method, program

[0001] This invention relates to a technique for matching strings of characters.

[0002] It is known that automata can be used to efficiently match specified strings against text data such as documents, tables, and program source code. For example, consider matching the location of the six-character string "ababcb" within a text file. The simplest method is to first check if the first six characters of the text file match the specified string, then check if the next six characters of the text file match the specified string, and so on, shifting the matching location within the text file one character at a time. If we consider the processing time per character match as the unit of processing, in the worst case, this method requires processing time roughly equal to the product of the number of characters in the specified string and the total number of characters in the text file. Therefore, the processing time becomes large when the specified string is long.

[0003] Furthermore, the above method may result in unnecessary processing. For example, if the text data contains "ababa," it does not match the specified string "ababcb" because the fifth character is not "c." However, since the third and fourth characters "ab" match the string "ababcb," it is clear that we only need to check if it matches the six characters starting with "aba" shifted two characters, rather than checking if it matches the six characters starting with "baba" shifted one character in the text data. The processes that can be omitted depend on the structure of the specified string. Automata can represent processing flows that take such omissions into account, thus enabling efficient matching. Specifically, once the string to be matched is specified, an automaton corresponding to that string is first created, and the matching process is performed using that automaton. This makes it possible to reduce the amount of processing.

[0004] Figure 1 shows an example of an automaton used to match the string "ababcb". The circular nodes represent the states of the automaton, and each node contains the characters that must match in the text data to transition to that state. The automaton in Figure 1 has six states: S0, S1, S2, S3, S4, and S5. ε is a string of length 0, and S0 represents the initial state. The operation of the automaton is described below. The automaton performs string matching by inputting one character at a time from the beginning of the text data, and the automaton repeatedly transitions through states according to the arrows. Processing starts from the initial state S0. In each state, if the character written on the node at the end of the solid arrow extending from the node representing that state matches the input character, the state transitions according to the solid arrow; otherwise, the state transitions according to the dashed arrow. If there is no match, the state transition process described above is repeated in the next state. Therefore, the above state transition process is recursively repeated until the input character matches the character written on the node at the end of the solid arrow, or until the system self-transitions from initial state S0 to initial state S0 without matching. If the system transitions to a node with a double circle, the automaton outputs the position in the text data at that time as the matching result and transitions back to initial state S0. Then, the next character in the text data is input to the automaton.

[0005] As an example, let's consider the case of matching the string "ababcb" with the text data "abababcbac". In the simplest method described above, we would need to perform 30 comparison operations because we are matching the string "ababcb" with the characters "ababab", "bababc", "ababcb", "babcba", and "abbcbac" in the text data. On the other hand, when using the automaton in Figure 1, in the initial state S0, we input the text data starting from the first character "a". The first four characters, a, b, a, and b, match the characters written on the nodes representing states S1, S2, S3, and S4, respectively, so the automaton transitions from state S0 to state S4 following the solid arrow. The fifth character, "a", does not match the character "c" written on the node representing state S5, so the automaton transitions to state S2 following the dashed arrow extending from the node representing state S4. Comparing the fifth character, "a", with the character written on the node representing state S3, it matches the character written on the node representing state S3, so the automaton transitions to state S3. The characters b, c, and b from the 6th to the 8th character match the characters written on the nodes representing states S4 and S5, and the double-circled node, respectively. Therefore, the automaton transitions to state S0, represented by the double-circled node, outputs information indicating a match at the 8th character of the text data as the matching result, and then transitions back to the initial state S0. The 9th character, a, matches the character written on the node representing state S1, so the automaton transitions to state S1. The last character, c, does not match the character written on the node representing state S2, so following the dashed arrow, the automaton transitions back to the initial state S0. Using an automaton, matching can be done in 11 comparison operations. In other words, using an automaton, matching can be performed with an amount of processing roughly equal to the total number of characters in the text data, regardless of the length of the string to be matched.

[0006] As this example shows, the state of the automaton can be said to correspond to the degree of matching, specifically how many characters of the string to be matched have been confirmed to match.

[0007] The technology described in Non-Patent Document 1 applies this automaton to encoded text data. The technology in Non-Patent Document 1 compresses the text data by encoding it using a fixed-length code, and by obtaining the matching result directly from the encoded text data, it is possible to save storage space for the text data and perform matching while omitting the text data decoding process. Figure 2 shows an example of a codebook used in the technology described in Non-Patent Document 1. The codebook in Figure 2 represents an encoding rule in which one string corresponds to a codeword with a fixed number of bits (3 bits). For example, the string "aaa" is encoded as the codeword "000".

[0008] The technology described in Non-Patent Document 1 uses an automaton for string matching and a codebook for encoding text data. By pre-determining how the string corresponding to each codeword transitions through each state of the automaton, it becomes possible to directly obtain matching results from text data, which is a sequence of codewords. Figure 3 shows a state transition table used in the technology described in Non-Patent Document 1. The state transition table in Figure 3 was created based on the automaton in Figure 1 and the codebook in Figure 2. A state transition table is created for each state of the automaton. The state transition table consists of a triplicate of codeword, relative position, and destination. Here, the relative position is a set of values ​​indicating which character of the string corresponding to the codeword transitioned to the double-circled node (i.e., whether the matching was successful), and represents the matching result. Here, φ represents an empty set, indicating that the string corresponding to that codeword does not transition to the double-circled node. The destination is the state to which the system transitions as a result of matching the string corresponding to the codeword. The state transition table in Figure 3 shows, for example, that when the codeword 000 is input in state S0, the system transitions to state S2 without outputting anything.

[0009] As an example, we will explain the case where the string "ababcb" is matched using the state transition table in Figure 3 from the text data 001011110101100, which is obtained by encoding the text data "aababcbac" using the codebook in Figure 2. The state is set to the initial state S0, and processing starts from the first codeword 001. When the first codeword 001 is input in state S0, the relative position corresponding to codeword 001 is φ, and the transition destination is S2, so nothing is output and the system transitions to state S2. When the second codeword 011 is input in state S2, the relative position corresponding to codeword 011 is φ, and the transition destination is S4, so nothing is output and the system transitions to state S4. When the third codeword 110 is input in state S4, the relative position corresponding to codeword 110 is φ, and the transition destination is S5, so nothing is output and the system transitions to state S5. In state S5, when the fourth codeword 101 is input, the relative position corresponding to codeword 101 is {1}, and the destination is S0, so {1} is output and the system transitions to state S0. In state S0, when the fifth codeword 100 is input, the relative position corresponding to codeword 100 is φ, and the destination is S0, so nothing is output and the system transitions to state S0. From the above, it can be seen that the matching was successful at the first character of string b corresponding to the fourth codeword 101.

[0010] In the state transition table of Figure 3, the relative position was either an empty set or a set with 1 element, but the relative position can also be a set with 2 or more elements. In other words, it is possible to successfully match multiple locations with a single codeword. Figure 4 shows an example of an automaton used for matching string a. Figure 5 shows a state transition table used in the technology of Non-Patent Document 1. The state transition table in Figure 5 was created based on the automaton in Figure 4 and the codebook in Figure 2. In the state transition table in Figure 5, for example, the relative position corresponding to codeword 000 is {1, 2, 3}. This is because codeword 000 corresponds to the string aaa, and matches the specified string a at the 1st, 2nd, and 3rd characters of string aaa.

[0011] T. Kida, T. Matsumoto, Y. Shibata, M. Takeda, A. Shinohara, S. Arikawa, “Collage system: a unifying framework for compressed pattern matching,” Theoretical Computer Science, Vol.298, Issue 1, pp.253-272, 2003.

[0012] The fixed-length codes used in the technique of Non-Patent Document 1 often have low compression efficiency, and there is a problem that the effect of saving the storage capacity of text data is small.

[0013] Therefore, an object of the present invention is to provide a technique for efficiently collating a specified character string from text data encoded using a fixed-length code with higher compression efficiency.

[0014] One aspect of the present invention is a character string collation device that generates an output collation result that is the result of collating one or more specified character strings with input text data. The input text data is a series of code words encoded using a plurality of codebooks, and the collation state for each codebook is determined by a combination of the degree of collation indicating up to which character position it has been confirmed that the characters match for any of the one or more specified character strings and the codebook information indicating which of the plurality of codebooks it is. Using the one or more specified character strings and the plurality of codebooks, for each collation state for each codebook, a state transition rule generation unit that generates a state transition rule indicating a collation result that is the output when a code word included in the codebook indicated by the codebook information used to determine the collation state for each codebook is input in the degree of collation used to determine the collation state for each codebook and the collation state for each codebook that is the destination of the transition, and a collation unit that obtains a collation result corresponding to the code word constituting the input text data using the state transition rule and generates the output collation result from the obtained collation result.

[0015] According to the present invention, it becomes possible to efficiently collate a specified character string from text data encoded using a fixed-length code with higher compression efficiency.

[0016] It is a diagram showing an example of an automaton used for string matching. It is a diagram showing an example of a code book used in the technique of Non-Patent Document 1. It is a diagram showing an example of a state transition table used in the technique of Non-Patent Document 1. It is a diagram showing an example of an automaton used for string matching. It is a diagram showing an example of a state transition table used in the technique of Non-Patent Document 1. It is a diagram showing an example of a code book of Reference Non-Patent Document 1. It is a diagram showing an analysis tree equivalent to the code book of FIG. 6. It is a diagram showing a code book equivalent to the code book of FIG. 6. It is a diagram showing an example of a state transition table in the first embodiment. It is a diagram showing an example of a state transition table in the first embodiment. It is a diagram showing an example of a state transition table in the first embodiment. It is a diagram showing an example of a state transition table in the first embodiment. It is a diagram showing an example of a code book in Modification Example 1 of the first embodiment. It is a block diagram showing the configuration of the string matching device 100. It is a flowchart showing the operation of the string matching device 100. It is a diagram showing an example of an automaton of Reference Non-Patent Document 2. It is a diagram showing an example of a state transition table in the second embodiment. It is a diagram showing an example of an automaton of Reference Non-Patent Document 3. It is a diagram showing an example of a sub-word output table of Reference Non-Patent Document 3. It is a diagram showing an example of a state transition table in the third embodiment. It is a diagram showing an example of a state transition table in Modification Example 1 of the third embodiment. It is a block diagram showing the configuration of the tokenization device 200. It is a flowchart showing the operation of the tokenization device 200. It is a diagram showing an example of the functional configuration of a computer that realizes each device in the embodiments of the present invention.

[0017] Hereinafter, embodiments of the present invention will be described in detail. Note that components having the same function are denoted by the same numbers, and redundant explanations are omitted.

[0018] Prior to the description of each embodiment, the notation method in this specification will be described.

[0019] ^ (caret) represents a superscript. For example, x y^z is y z is a superscript for x, and x y^z is y z represents that it is a subscript for x. Also, _ (underscore) represents a subscript. For example, x y_z is y zis a superscript to x, and x y_z is y z This indicates that it is a subscript for x.

[0020] The superscripts "^" and "~" such as ^x and ~x for a given character x should ideally be placed directly above "x", but due to the constraints of notation in the specification, they are written as ^x and ~x.

[0021] <Technical Background> In the string matching of the first embodiment, a specified string is matched from text data encoded into a sequence of fixed-length codewords using multiple codebooks.

[0022] <<1: Code Books>> In the first embodiment, the technology of Reference Non-Patent Document 1 is used as a fixed-length code using multiple code books.

[0023] (Reference Non-Patent Literature 1: H. Yamamoto, H. Yokoo, “Average-Sense Optimality and Competitive Optimality for Almost Instantaneous VF Codes,” in IEEE Transactions on Information Theory, Vol.47, No.6, pp.2174-2184, 2001.) In general, for coding rules that keep the codeword length constant, ensuring that any input string can be uniquely coded allows for the construction of a code with better compression efficiency. For this purpose, the technology in Reference Non-Patent Literature 1 uses multiple codebooks. Figure 6 shows an example of a codebook from Reference Non-Patent Literature 1. Figure 7 shows a parse tree equivalent to the codebook in Figure 6. The fixed-length code of Reference Non-Patent Literature 1 will be explained with reference to Figure 7. The parse tree in Figure 7 represents one coding rule with two parse trees T0 and T1. In parse trees T0 and T1, codewords are also assigned to nodes other than the leaves. By defining a rule for switching between parse trees, such as using parse tree T1 to encode the next string when a codeword assigned to a node other than a leaf is output, and using parse tree T0 to encode the next string when a codeword assigned to a leaf is output, compression efficiency is increased while ensuring that any input string can be uniquely encoded. Figure 8 shows a codebook equivalent to the codebook in Figure 6. The codebook in Figure 8 is the codebook in Figure 6 with information indicating the next codebook to be used added as a link destination.

[0024] <<2: State Transition Table>> The state transition table in the first embodiment will be described. The state transition table in the first embodiment is configured using an automaton corresponding to the character string to be collated and a code book of fixed-length codes, similar to Non-Patent Document 1. More specifically, the state transition table in the first embodiment is configured using the automaton of Non-Patent Document 1 and the code book of Reference Non-Patent Document 1. In the technique of Non-Patent Document 1, since there is one code book, a state transition table was created for each state of the automaton. In contrast, in the first embodiment, since there are multiple code books, a state transition table is created for each state determined by the combination of the state of the automaton and the code book. This state is called the collation state by code book. The collation state by code book S i,j is the state determined by the combination of the i-th state S i of the automaton and the j-th code book T j . The configuration procedure of the state transition table of the collation state by code book S i,j consists of the following three processes.

[0025] (1) Obtain the relative position and the destination state that are the outputs when the character string corresponding to the code word included in the code book T i in the state S j is input to the automaton.

[0026] (2) Next, obtain the next code book to be used corresponding to the code word using the j-th code book T j .

[0027] (3) Set the relative position obtained in (1) as the relative position in the state transition table of the collation state by code book S i,j . Also, set the collation state by code book determined as the combination of the destination state obtained in (1) and the next code book to be used obtained in (2) as the destination state.

[0028] The above procedures (1) to (3) are repeated for all code words included in the code book T j . Thereby, the state transition table of the collation state by code book S i,j is constructed.

[0029] FIGS. 9 to 12 are diagrams showing examples of the state transition table. FIGS. 9 to 12 are state transition tables constructed using the automaton of FIG. 1 and the code book of FIG. 8. The collation state by code book S5,1 The procedure for constructing a state transition table will be explained using the following as an example. The strings corresponding to the codewords contained in the codebook T1 are input into the automaton, and the state of the automaton that it transitions to when processing starts from the automaton's fifth state, S5, is determined. If the strings corresponding to the codewords 000, 001, 010, 011, 100, 101, 110, and 111 are input, the automaton's states transition to S1, S1, S2, S0, S0, S0, S0, and S1, respectively. Since the strings corresponding to codewords 000 through 101 start with 'b', they transition to the state represented by the double-circled node, and the relative position for codewords 000, 001, 010, 011, 100, and 101 is found to be {1}. For the other strings corresponding to codewords 110 and 111, the transition does not occur to the state represented by the double-circled node, so the relative position is found to be φ. Furthermore, in codebook T1, the link destination is T1 only for codewords 001 and 110, and for all other codewords, the link destination is T0. From the above, the relative positions corresponding to codewords 000, 001, 010, 011, 100, 101, 110, and 111 are {1}, {1}, {1}, {1}, {1}, {1}, φ, and φ. Also, by combining the transition destination automaton states S1, S1, S2, S0, S0, S0, S0, S1 and the link destinations T0, T1, T0, T0, T0, T0, T1, T0, the transition destinations corresponding to codewords 000, 001, 010, 011, 100, 101, 110, and 111 are S 1,0 , S 1,1 , S 2,0 , S 0,0 , S 0,0 , S 0,0 , S 0,1 , S 1,0 This can be determined.

[0030] <<3: Matching Procedure>> The matching procedure of the first embodiment will now be described. This matching procedure takes encoded text data (i.e., a sequence of codewords) and a specified string as input, and uses multiple codebooks used to encode the text data to obtain and output the matching result. The matching procedure consists of the following three processes.

[0031] (1) Generate an automaton corresponding to a given string. For example, the same method as in Non-Patent Document 1 can be used to generate the automaton.

[0032] (2) A state transition table is generated using the automaton generated in (1) and multiple codebooks. The method described above is used to generate the state transition table.

[0033] (3) Starting from the initial codebook matching state, the codewords constituting the text data are read one by one in order, and the process of determining the relative position and transition destination according to the state transition table in (2) is repeated. All relative positions obtained through the repeated process are output as the output matching result. Here, the initial codebook matching state is the codebook matching state S 0,k (where k is the codebook number used first when encoding the text data.)

[0034] This section describes an example of matching the string "ababcb" from the text data 001010110011, which is encoded using the codebook in Figure 8 (where the first codebook used is encoded as T0). For this purpose, the state transition tables in Figures 9 to 12 are used. When the codewords 001, 010, 110, and 011 are input, the codebook matching state is the initial codebook matching state S. 0,0 From S 1,1 , S 4,0 , S 5,1 , S 0,0 The transition occurs as follows. 5,1 From S 0,0 When the state transitions, the relative position {1} is obtained, so the relative position {1} is output as the matching result. In other words, it can be seen that the matching was successful at the first character b of the string bac, which corresponds to the fourth codeword 011 in the text data.

[0035] As can be seen from the state transition table, the matching result obtained when a single codeword is input is a relative position, but the output matching result can be any information obtained by checking whether the text data matches the specified string. In addition to the relative position mentioned above, that is, information about the position of the string in the text data, the output matching result can also be, for example, a flag indicating whether or not the specified string exists in the text data, or the number of times the specified string appears in the text data.

[0036] (Modification 1) As multiple codebooks that represent encoding rules with high compression efficiency, the codebooks described below can be used instead of the codebook in Reference Non-Patent Document 1.

[0037] The codebook in modified example 1 has a predetermined structure in order to represent coding rules that satisfy the condition that they can be uniquely coded (hereinafter referred to as the unique coding condition).

[0038] <<1: Codebook>> First, we will explain the concept of modes, which is necessary for explaining the codebook in Modification 1. A mode is defined as a set of strings for a codebook. A mode is assigned to each of the multiple codebooks used for encoding / decoding. Since the modes assigned to a codebook are defined as a set of strings, it is assumed that there are no duplicate strings included in the mode. It is preferable that the modes satisfy the word-first condition. Here, the word-first condition is the condition that for any string included in a set of strings as elements, that string cannot be the beginning of any other string included in that set. For example, if there are two codebooks, codebook T0 and codebook T1, the modes assigned to codebook T0 and codebook T1 can be defined as {aab, abc, aca} and {ac, ba}, respectively.

[0039] The codebook in Modification 1 consists of a set of three: an input string, an output codeword of a predetermined length, and the next codebook to be used (hereinafter referred to as the linked codebook). Figure 13 shows an example of a codebook. The codebook in Figure 13 is used to encode a string, which is a sequence of characters {a, b, c}, into a codeword, which is a sequence of codes {0, 1}, and consists of five codebooks: T0, T1, T2, T3, and T4. Note that the length of the output codeword in codebooks T0, T1, T2, T3, and T4 is 2. In Figure 13, ε represents a string of length 0. Note that the string ε can be the beginning of any string. Codebook T0 is assigned the mode {ε}, and codebook T1 is assigned the mode {a, b}. Furthermore, codebook T0 outputs the codeword 00 for string a and the next codebook to be used is T1, and outputs the codeword 01 for string ε and the next codebook to be used is T2.

[0040] Next, we will explain the conditions for the codebook in Modification 1 to satisfy the unique codeability condition. To do this, we define the concept of an expanded string. A string is an expanded string of a codebook if it is obtained by concatenating a string contained in the codebook with a string that is an element of the mode assigned to the next codebook to be used, which corresponds to the string in question. Here, a string contained in a codebook refers to a string that is an element of a triplet that constitutes the codebook. For example, the expanded strings of codebook T2 in Figure 13 are the string aca obtained by concatenating the string aca with the string ε, which is an element of mode {ε} assigned to the next codebook T0 that corresponds to the string aca; the string baa obtained by concatenating the string baa with the string ε, which is an element of mode {ε} assigned to the next codebook T0 that corresponds to the string baa; the strings acb and acc obtained by concatenating the string ac with the strings b and c, which are elements of mode {b, c} assigned to the next codebook T3 that corresponds to the string ac; and the strings bab and bac obtained by concatenating the string ba with the strings b and c, which are elements of mode {b, c} assigned to the next codebook T3 that corresponds to the string ba.

[0041] The following two conditions must be met for a codebook to satisfy the unique codeability requirement for encoding / decoding:

[0042] (1) For any codebook included in multiple codebooks, the set of all expanded strings from that codebook satisfies the word-first condition.

[0043] (2) For any codebook included in multiple codebooks, the expanded string of that codebook shall begin with one of the strings included in the mode assigned to that codebook.

[0044] Even if the multiple codebooks used for encoding / decoding do not satisfy the unique codeability condition, the codeword encoded using those codebooks can be uniquely decoded into a string. However, from the viewpoint of compression efficiency, it is preferable that the multiple codebooks used for encoding / decoding satisfy the unique codeability condition.

[0045] <<2: Encoding Procedure / Decoding Procedure>> First, the encoding procedure using the codebook in Modification 1 will be explained. This encoding procedure takes the string to be encoded (hereinafter referred to as the input string) as input and uses multiple codebooks {T k} k=0 K-1 Using (where K is an integer greater than or equal to 2), codewords corresponding to the strings that make up the input string are obtained, and a codeword obtained by concatenating all the obtained codewords (hereinafter referred to as the output codeword) is output. Multiple codebooks {T k} k=0 K-1In the process of obtaining a codeword corresponding to the strings that make up the input string using the method, if there exists a string included in the current codebook (hereinafter referred to as string σ) that matches the beginning of the current string, and there exists a string included in the mode assigned to the next codebook to be used that corresponds to string σ that matches the beginning of the string obtained by removing the beginning of the string that matches string σ from the current string, then the codeword corresponding to string σ is obtained, the next codebook to be used that corresponds to string σ is set as the current codebook, and the string obtained by removing the beginning of the string that matches string σ from the current string is set as the current string, and this process is repeated until the length of the current string becomes 0. Here, the current string is the string that is the target of the encoding process at the present moment, and the current string at the time encoding started is the input string.

[0046] More specifically, the encoding procedure consists of the following three processes:

[0047] (1) A prescribed code book T k Encoding begins using the current codebook as k. Here, k is any integer between 0 and K-1 (inclusive); for example, k=0. The input string is then used as the current string.

[0048] (2) The current string is compared with the strings contained in the current codebook. If there is a string contained in the current codebook (hereinafter referred to as string σ) that matches the beginning of the current string, and there is a string contained in the mode assigned to the next codebook to be used that corresponds to string σ that matches the beginning of the string obtained by removing the beginning of the string that matches string σ from the current string, then the codeword corresponding to string σ is obtained, the next codebook to be used that corresponds to string σ is set as the current codebook, and the string obtained by removing the beginning of the string that matches string σ from the current string is set as the current string.

[0049] (3) If the length of the current string obtained by the process in (2) is 0, the codeword obtained by concatenating all the codewords obtained by the process in (2) is output as the output codeword and the encoding process is terminated. Otherwise, the process returns to (2).

[0050] Next, the decoding procedure using the codebook in Modification 1 will be described. This decoding procedure takes the codeword to be decoded (hereinafter referred to as the input codeword) as input and uses multiple codebooks {T k} k=0 K-1 (However, multiple codebooks {T k} k=0 K-1 The codebook used in the encoding procedure is used to obtain strings corresponding to the codewords that make up the input codeword, and all the obtained strings are concatenated to produce a string (hereinafter referred to as the output string). Multiple codebooks {T k} k=0 K-1 In the process of obtaining a string corresponding to a codeword that makes up the input codeword using the method, a string corresponding to a codeword included in the current codebook that matches the beginning of the current codeword (hereinafter referred to as codeword w) is obtained, the next codebook to be used corresponding to that codeword w is set as the current codebook, and the codeword obtained by removing the beginning that matches codeword w from the current codeword is set as the current codeword. This process is repeated until the length of the current codeword becomes 0. Here, the current codeword is the codeword that is the target of the decoding process at the present time, and the current codeword at the time decoding starts is the input codeword.

[0051] More specifically, the decryption procedure consists of the following three steps:

[0052] (1) A prescribed code book T k Decoding begins using k as the current codebook, where k is the codebook number used when the encoding process for the input codeword was started. The input codeword is then used as the current codeword.

[0053] (2) Using the current codebook, compare the current codeword with the codewords included in the codebook. Obtain a string corresponding to a codeword included in the current codebook that matches the beginning of the current codeword (hereinafter referred to as codeword w), designate the next codebook to be used corresponding to codeword w as the current codebook, and remove the beginning that matches codeword w from the current codeword to obtain the current codeword.

[0054] (3) If the length of the current codeword obtained by the process in (2) is 0, the string obtained by concatenating all the strings obtained by the process in (2) is output as the output string and the decoding process is terminated. Otherwise, the process returns to (2).

[0055] As can be seen from the above, a mode can be said to be a set assigned in the encoding process to confirm whether it is correct to use the next codebook corresponding to the string contained in the codebook as the current codebook used to encode the next current codeword.

[0056] <<3: State Transition Table>> The state transition table in Modification 1 will now be explained. The codebook of the first embodiment and the codebook of Modification 1 differ in whether or not there is a confirmation mechanism to determine the codebook to be used next in the encoding process. Therefore, the state transition table in Modification 1 can be constructed in the same way as the state transition table in the first embodiment. The matching procedure may also be the same as the matching procedure in the first embodiment.

[0057] <First Embodiment> The string matching device 100 takes input text data, which is a sequence of codewords encoded using multiple codebooks, and one specified string as input, and outputs an output matching result, which is the result of matching the input text data with the one specified string.

[0058] The string matching device 100 will be described below with reference to Figures 14 and 15. Figure 14 is a block diagram showing the configuration of the string matching device 100. Figure 15 is a flowchart showing the operation of the string matching device 100. As shown in Figure 14, the string matching device 100 includes a state transition rule generation unit 110, a matching unit 120, and a recording unit 190. The recording unit 190 is a component that appropriately records information necessary for the processing of the string matching device 100. For example, the recording unit 190 records multiple codebooks used for encoding. Note that instead of recording these multiple codebooks in the recording unit 190, they may be input.

[0059] The operation of the string matching device 100 will be explained in accordance with Figure 15.

[0060] In S110, the state transition rule generation unit 110 defines a codebook-specific matching state as a state determined by a combination of a matching degree indicating how many characters of a single specified string input have been confirmed to match, and codebook information indicating which of the multiple codebooks used for encoding it is. Using the single specified string and the multiple codebooks, the unit generates and outputs a state transition rule for each codebook-specific matching state, which is the output when a codeword included in the codebook indicated by the codebook information used to determine the codebook-specific matching state is input at the matching degree used to determine the codebook-specific matching state, and the codebook-specific matching state to transition to. The state transition rule generation unit 110 performs, for example, the following two processes.

[0061] (1) A process to generate an automaton for matching a single specified string that is input. (2) A process to generate state transition rules for each codebook matching state, using the state of the automaton generated in (1) as the degree of matching. The state transition rules are, for example, a state transition table as described in <Technical Background>. In this case, the matching result that is output when a codeword is input is the relative position in the state transition table, and the codebook matching state that is the destination when a codeword is input is the destination in the state transition table.

[0062] In S120, the matching unit 120 uses the state transition rules generated in S110 to obtain matching results corresponding to the codewords that make up the input text data, generates output matching results from the obtained matching results, and outputs them.

[0063] If the state transition rule is a state transition table as described in <Technical Background>, the matching result that is output when a codeword is input is the relative position in the state transition table. However, the output matching result can be any information that can be obtained from all the matching results obtained, as described in <Technical Background>.

[0064] According to embodiments of the present invention, it is possible to efficiently match a specified string from text data encoded using a fixed-length code with higher compression efficiency. In this case, there is no need to decode the encoded text data. Furthermore, by encoding the text data using a fixed-length code with high compression efficiency, the storage capacity of the text data can be saved.

[0065] <Technical Background> In the string matching of the second embodiment, multiple specified strings are simultaneously matched from text data encoded into a sequence of fixed-length codewords using multiple codebooks.

[0066] <<1: Automata>> In the second embodiment, the technology of Reference Non-Patent Document 2 is used as an automaton for efficiently matching multiple specified strings.

[0067] (Reference Non-Patent Literature 2: AV Aho and MJ Corasick, “Efficient string matching: an aid to bibliographic search,” Commun. ACM Vol.18, No.6, pp.333-340, 1975.) The automaton in Reference Non-Patent Literature 2 will be explained using an example. Figure 16 shows an example of the automaton in Reference Non-Patent Literature 2. The automaton in Figure 16 is an automaton that matches multiple strings {a, ab, bc, bca, c, caa}. The nodes shown by circles represent the states of the automaton, and each node contains the string that was most recently matched when it transitioned to that state. The automaton in Figure 16 has six states: S0, S1, S2, S3, S4, and S5. Similar to the automaton in Figure 1, S0 represents the initial state. The operation of the automaton will be explained below. Text data is input into the automaton one character at a time from the beginning, and string matching is performed by the automaton repeatedly transitioning states according to the arrows. Processing begins from the initial state S0. In each state, if the last character of the string written at the node at the end of the solid arrow extending from the node representing that state matches the input character, the state transition proceeds according to the solid arrow; otherwise, the state transition proceeds according to the dashed arrow. The dashed arrow connects to the node containing the string that is the suffix of the string written at the original node. Nodes without a solid arrow will transition according to the dashed arrow regardless of the input character, and therefore represent the same state as the node at the end of the dashed arrow. The next character is entered when the state transition proceeds according to the solid arrow, or when it is no longer possible to transition according to the solid arrow in the initial state S0. Nodes with double circles correspond to the string to be matched. When a transition to a node with a double circle occurs, the matching result outputs a pair of information: which string was successfully matched and its position in the text data at that time.Furthermore, when transitioning between any nodes, if the node connected by a dashed arrow extending from that node is a double-circled node, the program outputs a pair of the string written on the node connected by the dashed arrow and its position in the text data at that time as the matching result.

[0068] As an example, let's consider the case where the string {a, ab, bc, bca, c, caa} is matched against the text data bcbcaa. The first character, b, causes the automaton to transition from the initial state S0 to state S2, and the second character, c, causes the automaton to transition to state S4. Since the node representing state S4 is a double-circle node, the automaton outputs the pair (bc, 2) indicating that the second character matched the string bc. Also, since the node representing state S3, which is connected to the node representing state S4 by a dashed arrow, is also a double-circle node, the automaton also outputs the pair (c, 2) indicating that the second character matched the string c. The third character, b, does not match the last character of the string written on the node representing state S5, so following the dashed arrow, the automaton transitions to state S3. Furthermore, since it does not match the last character of the string written on the node representing state S5, which is connected to the node representing state S3 by a solid arrow, the automaton transitions to state S0 following the dashed arrow. The third character, 'b', matches the last character of the string written on the node representing state S2, so the automaton transitions to state S2. The fourth character, 'c', causes the automaton to transition to state S4, and just like when the second character, 'c', was input, it outputs the pair (bc, 4) indicating that the fourth character matched the string 'bc', and the pair (c, 4) indicating that the fourth character matched the string 'c'. The fifth character, 'a', causes the automaton to transition to state S5, and it outputs the pair (bca, 5) indicating that the fifth character matched 'bca'. Since there is no solid arrow extending from the node representing state S5, the automaton follows the dashed arrow and transitions to state S5, which is the node on which the string 'ca' is written. The node representing state S5 and the node representing state S1, which is connected by a dashed arrow, are double-circled nodes, so the automaton outputs the pair (a, 5) indicating that the fifth character matched 'a'. The last input, the character 'a', causes the automaton to transition to state S1, which is a node with the string 'caa' written on it. It outputs the pair (cca, 6) indicating that 'caa' was matched at the sixth character, and also outputs the pair (a, 6) indicating that 'a' was matched at the sixth character, as the node connected by the dashed arrow from the node representing state S1 is also a double-circled node.Based on the above, the matching results output are (bc, 2), (c, 2), (bc, 4), (c, 4), (bca, 5), (a, 5), (cca, 6), (a, 6).

[0069] As can be seen from this example, similar to the automaton in the first embodiment, the state of the automaton corresponds to the degree of matching, specifically how many characters of the string to be matched have been confirmed to match. In contrast, the automaton in the first embodiment outputs information about which character match occurred because there is only one string to be matched, whereas the automaton in the second embodiment outputs information about which string matched and at which character.

[0070] <<2: State Transition Table>> The state transition table in the second embodiment will be described. The state transition table in the second embodiment is constructed using the automaton and the code book of Reference Non-Patent Document 1. The state transition table in the second embodiment is constructed using the same procedure as the state transition table in the first embodiment. Code Book Matching State S i,j The procedure for constructing the state transition table consists of the following three processes.

[0071] (1) State S i In the code book T j The automaton receives a string corresponding to a codeword from the given set as input, and outputs a pair of matched words and their relative positions, along with the destination state. Here, the matched word indicates which string matched, and the relative position indicates which character of the string corresponding to the codeword matched.

[0072] (2) Next, the j-th code book T j Use this to find the next codebook to use that corresponds to the codeword.

[0073] (3) The matching word and relative position pair obtained in (1) is matched in the codebook-specific matching state S i,j This represents a pair of matching words and their relative positions in the state transition table. The destination state is determined by the combination of the destination state found in (1) and the next codebook to be used found in (2), which is the codebook-specific matching state.

[0074] The above steps (1) to (3) are performed in the code book T j This is repeated for all codewords included. This results in a codebook-specific matching state S i,j A state transition table is constructed.

[0075] Figure 17 shows an example of a state transition table. Figure 17 is a part of a state transition table constructed using the automaton in Figure 16 and the codebook in Figure 8. The automaton of the second embodiment differs from the automaton of the first embodiment in terms of the matching results that are output. Therefore, the state transition table used in the second embodiment differs from the state transition table used in the first embodiment in terms of the data structure of the second column corresponding to the matching results. Specifically, the data structure of the second column of the state transition table used in the second embodiment is a set whose elements are pairs of matching words and relative positions (matching word, relative position). For example, the codebook-specific matching state S in Figure 17 2,1 This is based on the automaton state S2 in Figure 16 and the codebook T1 in Figure 8. In codebook T1, codeword 111 represents ca, so the set {(bc, 1), (c, 1), (bca, 2), (a, 2)}, which is the matching result output when the string ca is input in state S2, becomes the value of (matching word, relative position). Also, when the string ca is input, the system transitions to state S5, while the next codebook to be used becomes T0, so the transition destination is the codebook-specific matching state S 5,0 It will become.

[0076] <<3: Matching Procedure>> The matching procedure of the second embodiment will now be described. This matching procedure takes encoded text data (i.e., a sequence of codewords) and a number of specified strings as input, and uses the number of codebooks used to encode the text data to obtain and output the matching result. The matching procedure consists of the following three processes.

[0077] (1) Generate an automaton corresponding to a set of specified strings. For example, the same method as in Reference Non-Patent Document 2 can be used to generate the automaton.

[0078] (2) A state transition table is generated using the automaton generated in (1) and multiple codebooks. The method described above is used to generate the state transition table.

[0079] (3) Starting from the initial codebook matching state, the codewords constituting the text data are read one by one in order, and the process of finding matching word and relative position pairs and transition destinations according to the state transition table in (2) is repeated. All matching word and relative position pairs obtained through the repeated process are output as output matching results. Here, the initial codebook matching state is the codebook matching state S 0,k (where k is the codebook number used first when encoding the text data.)

[0080] As can be seen from the state transition table, the matching result obtained when one codeword is input is a pair of the matched word and its relative position. However, the output matching result can be any information obtained by checking whether the text data matches the specified string. In addition to the pair of the matched word and its relative position mentioned above, that is, information about the position of the string in the text data, the output matching result may also include, for example, a flag indicating whether the specified string exists in the text data, the number of times the specified string appears in the text data, the most frequently appearing string among the specified strings in the text data, or the order in which the specified strings appear in the text data.

[0081] (Modification 1) The code book described in Modification 1 of the First Embodiment can also be used for matching multiple strings. In this case, the state transition table is constructed using the automaton of Reference Non-Patent Document 2 and the code book of Modification 1 of the First Embodiment. The state transition table may be constructed in the same way as the state transition table in the second embodiment. The matching procedure may also be the same as the matching procedure in the second embodiment.

[0082] <Second Embodiment> The string matching device 100 takes input text data, which is a sequence of codewords encoded using multiple codebooks, and multiple specified strings as input, and outputs an output matching result, which is the result of matching the input text data with the multiple specified strings.

[0083] The string matching device 100 will be described below with reference to Figures 14 and 15. Figure 14 is a block diagram showing the configuration of the string matching device 100. Figure 15 is a flowchart showing the operation of the string matching device 100. As shown in Figure 14, the string matching device 100 includes a state transition rule generation unit 110, a matching unit 120, and a recording unit 190. The recording unit 190 is a component that appropriately records information necessary for the processing of the string matching device 100. For example, the recording unit 190 records multiple codebooks used for encoding. Note that instead of recording these multiple codebooks in the recording unit 190, they may be input.

[0084] The operation of the string matching device 100 will be explained in accordance with Figure 15.

[0085] In S110, the state transition rule generation unit 110 defines a codebook-specific matching state as a state determined by a combination of a matching degree indicating how many characters of a plurality of specified strings have been confirmed to match and codebook information indicating which of the plurality of codebooks used for encoding it is. Using the plurality of specified strings and the plurality of codebooks, the unit generates and outputs a state transition rule for each codebook-specific matching state, which is the output when a codeword included in the codebook indicated by the codebook information used to determine the codebook-specific matching state is input at the matching degree used to determine the codebook-specific matching state, and the codebook-specific matching state to which the transition will occur. The state transition rule generation unit 110 performs, for example, the following two processes.

[0086] (1) A process to generate an automaton for matching multiple specified strings that are input. (2) A process to generate state transition rules for each codebook matching state, using the state of the automaton generated in (1) as the degree of matching. The state transition rules are, for example, a state transition table as described in <Technical Background>. In this case, the matching result that is output when a codeword is input is a pair of matching word and relative position in the state transition table, and the codebook matching state to which the codeword is input is the destination in the state transition table.

[0087] In S120, the matching unit 120 uses the state transition rules generated in S110 to obtain matching results corresponding to the codewords that make up the input text data, generates output matching results from the obtained matching results, and outputs them.

[0088] If the state transition rule is a state transition table as described in <Technical Background>, the matching result that is output when a codeword is input is a pair of matching word and relative position in the state transition table. However, the output matching result can be any information that can be obtained from all the matching results obtained, as described in <Technical Background>.

[0089] According to embodiments of the present invention, it is possible to efficiently match a specified string from text data encoded using a fixed-length code with higher compression efficiency. In this case, there is no need to decode the encoded text data. Furthermore, by encoding the text data using a fixed-length code with high compression efficiency, the storage capacity of the text data can be saved.

[0090] <Technical Background> When inputting text data into a language model such as a neural network, tokenization is generally performed, which involves dividing the text data into subwords and converting them into tokens. Subword division is done by preparing a dictionary that lists subwords and searching for the longest matching subword against this dictionary. Each subword listed in the dictionary is pre-associated with a token ID, and tokenization is performed by outputting a sequence of token IDs corresponding to the subwords obtained through subword division.

[0091] In the third embodiment, tokenization, which is an application of string matching, will be described. In tokenization in this embodiment, string matching is performed using a dictionary on text data encoded in a sequence of fixed-length codewords, and then tokenization is performed.

[0092] <<1: Automata>> It is known that tokenization can also be made more efficient by using automata. In the third embodiment, the technology of Reference Non-Patent Literature 3 is used as an automaton for efficiently matching subwords included in a dictionary.

[0093] (Reference Non-Patent Literature 3: X. Song, A. Salcianu, Y. Song, D. Dopson, D. Zhou, “Fast WordPiece Tokenization,” arXiv:2012.15524v3 [cs.CL], 2021.) The automaton of Reference Non-Patent Literature 3 will be explained using an example. Figure 18 shows an example of the automaton of Reference Non-Patent Literature 3. The automaton in Figure 18 is an automaton that performs matching using the dictionary {a, abcb, ##b, ##bc, ##c}. The dictionary {a, abcb, ##b, ##bc, ##c} is given in advance, and a, abcb, ##b, ##bc, ##c are each strings called subwords. Here, ## is a suffix indicator, indicating that the string after ## appears at the end of some string. Figure 19 is an example of a table showing the output of the automaton in Figure 18 (hereinafter referred to as the subword output table). Figure 19 is a table in which the states of the automaton in Figure 18 and the candidate for mismatch are paired. A candidate for mismatch represents a set of subwords that the automaton outputs when it cannot transition to a state according to the solid arrow. Here, ^φ represents a set with 0 elements (i.e., an empty set with no elements). Note that the candidate for mismatch is a set, not a collection. The reason for treating the candidate for mismatch as a set is that the order of the subwords output by the automaton is significant. The nodes indicated by circles represent the states of the automaton, and each node contains the string that was most recently matched when the automaton transitioned to that state. The automaton in Figure 18 has 10 states: S0, S1, S2, S3, S4, S5, S6, S7, S8, and S9. Similar to the automaton in Figure 1, S0 represents the initial state. The operation of the automaton will be explained below. The automaton takes one character at a time from the beginning of the text data, and performs string matching against subwords in the dictionary by repeatedly transitioning states according to the arrows. Processing starts from the initial state S0.In each state, if the last character of the string written at the node at the end of the solid arrow extending from the node representing that state matches the input character, the state transition proceeds according to the solid arrow. If they do not match (i.e., the state transition cannot be made to any node according to the solid arrow), the state transition proceeds according to the dashed arrow. From nodes where no solid arrows extend, the state transitions to the node representing state S2, which corresponds to the start of the word ending. If a state transition is not possible from the initial state S0, a special token is used to indicate that it is a subword not found in the dictionary. <unk>Outputs the following. Follow the solid arrows to transition between states, <unk>If the output is [output], enter the next character. If you cannot transition to any node following the solid arrow, output the mismatch candidate corresponding to the current state according to the subword table. Repeat the above process until the last character of the text data to obtain the output of the mismatch candidate subword or <unk>Arranging them in order results in text data divided into subword units. Note that subwords and <unk>Of course, you can also use the corresponding token ID instead.

[0094] As an example, let's consider the case where the text data abcc is split into subword units using the dictionary {a, abcb, ##b, ##bc, ##c}. Following the solid arrow, the automaton transitions to state S5 based on the first three characters a, b, and c. The fourth character, c, does not match the last character of the string written in the node representing state S9, which is connected to the node representing state S5 by a solid arrow. Therefore, it is not possible to transition from state S5 following the solid arrow. Consequently, the automaton outputs the mismatch candidate (a) corresponding to state S5, and transitions to state S8 following the dashed arrow. Since there is no solid arrow extending from the node representing state S8, the automaton outputs the mismatch candidate (##bc) corresponding to state S8, and transitions to state S2. The fourth character, c, matches the last character of the string written in the node representing state S7, so the automaton transitions to state S7. Since no solid arrows extend from the node representing state S7, the automaton outputs a candidate for mismatch (##c) corresponding to state S7. As a result, a, ##bc, and ##c are obtained as the split of the text data abcc.

[0095] As this example shows, the automaton matches the text data as a result of matching subwords or <unk>The series, matching subword or <unk>It outputs one of the sequences of token IDs corresponding to the given value. Furthermore, the automaton's state can be said to correspond to the degree of matching, specifically how many characters of which subwords have been confirmed to match.

[0096] Furthermore, by creating an automaton according to the dictionary, if, for example, "abc" matches but "abcb" does not, the fact that the match up to "abc" was confirmed can be reused for matching with other subwords, thus reducing the amount of processing required (see Figure 18).

[0097] <<2: State Transition Table>> The state transition table in the third embodiment will be described. The state transition table in the third embodiment is constructed using the automaton of Reference Non-Patent Document 3 and the code book used in the technology of Non-Patent Document 1. The state transition table in the third embodiment is constructed with the states of the automaton as matching states, similar to the technology of Non-Patent Document 1. Matching State S i The procedure for constructing the state transition table is as follows:

[0098] (1) State S i This process calculates the mismatch judgment, confirmed word, mismatch candidate, and the destination state when a string corresponding to a codeword in the codebook is input to the automaton. Here, the mismatch judgment is information indicating whether a state transition was not possible from the state at the time the codeword was input, following the solid arrow. If the mismatch judgment is 1, the mismatch candidate obtained from the previous codeword is output. The confirmed word is a set of subwords that can be confirmed as the output for the codeword, and is obtained by sequentially arranging the outputs obtained when the string corresponding to the codeword is input to the automaton and processing starts from the state at the time the string was input. The mismatch candidate is a set of subwords that are output candidates for codewords input after the codeword, and is the mismatch candidate corresponding to the destination state when the string corresponding to the codeword is input to the automaton and processing starts from the state at the time the string was input.

[0099] Repeat the procedure in (1) above for all codewords included in the codebook. This will result in matching state S i A state transition table is constructed.

[0100] Figure 20 shows an example of a state transition table. Figure 20 is a part of a state transition table constructed using the automaton from Figure 18 and the codebook from Figure 2.

[0101] <<3: Matching Procedure>> The matching procedure of the third embodiment will now be described. This matching procedure takes encoded text data (i.e., a sequence of codewords) and a dictionary as input, determines the division of the text data using the codebook used to encode the text data, and outputs the result. The matching procedure consists of the following three processes.

[0102] (1) Generate an automaton corresponding to the dictionary. For example, the same method as in Reference Non-Patent Document 3 can be used to generate the automaton.

[0103] (2) A state transition table is generated using the automaton and codebook generated in (1). The method described above is used to generate the state transition table.

[0104] (3) Starting from the initial matching state, the codewords constituting the text data are read one by one in order, and the process of determining mismatch, confirmed words, mismatch candidates, and transition destinations is repeated according to the state transition table in (2). All confirmed words obtained through the iterative process and the pairs of mismatch candidates output according to the mismatch determination values ​​are arranged in order and output as the text data division. Here, the initial matching state is the initial state S0.

[0105] This section describes an example of splitting the text data 000000101… encoded from the string aaaaaab… using the codebook in Figure 2 into subword units. For this purpose, the state transition table in Figure 20 is used. When the first codeword 000 is input, the confirmed words a, a are output, the mismatch candidate (a) is stored, and the system transitions to matching state S3. When the second codeword 000 is input, the mismatch judgment value is 1, so the stored mismatch candidate a is output, and the confirmed words a, a are output. Furthermore, the mismatch candidate (a) is stored, and the system transitions to matching state S3. When the third codeword 101 is input, the mismatch judgment is 0, so the stored mismatch candidate is not output. Also, the confirmed word is ^φ, so nothing is output. Furthermore, the mismatch candidate (a) is stored, and the system transitions to matching state S4. By repeating this process character by character, the split text data 000000101… can be obtained directly from the encoded text data.

[0106] (Modification 1) In the third embodiment, the code book of Reference Non-Patent Document 1 used in the first and second embodiments can also be used. In this case, the state transition table can be constructed in the same manner as the procedure for constructing the state transition table in the first and second embodiments. That is, the state transition table will be constructed using the automaton of Reference Non-Patent Document 3 and the code book of Reference Non-Patent Document 1. Figure 21 is a diagram showing an example of a state transition table. Figure 21 is a part of a state transition table constructed using the automaton of Figure 18 and the code book of Figure 8. The matching procedure can also be the same as the matching procedure in the first and second embodiments.

[0107] (Modification 2) In the third embodiment, the code book described in Modification 1 of the first embodiment can also be used. In this case, the state transition table can be constructed in the same way as the procedure for constructing the state transition table in the first and second embodiments. That is, the state transition table will be constructed using the automaton of Reference Non-Patent Document 3 and the code book of Modification 1 of the first embodiment. The matching procedure can also be the same as the matching procedure in the first and second embodiments.

[0108] <Third Embodiment> The tokenization device 200 takes input text data, which is a sequence of codewords encoded using one codebook, and a dictionary as input, and applies subwords or that are included in the dictionary to the input text data. <unk>Outputs an output sequence which is a sequence of token IDs corresponding to the specified value. Here, a dictionary is a set whose elements are subwords (where each subword is associated with a token ID). <unk>This token indicates that it is a subword not included in the dictionary. <unk>It is assumed that a token ID is associated with this as well.

[0109] The tokenization device 200 will be described below with reference to Figures 22 and 23. Figure 22 is a block diagram showing the configuration of the tokenization device 200. Figure 23 is a flowchart showing the operation of the tokenization device 200. As shown in Figure 22, the tokenization device 200 includes a state transition rule generation unit 210, a tokenization unit 220, and a recording unit 290. The recording unit 290 is a component that appropriately records information necessary for the processing of the tokenization device 200. For example, the recording unit 290 records one codebook used for encoding. Note that instead of recording the codebook in the recording unit 290, it may be input.

[0110] The operation of the tokenization device 200 will be explained in accordance with Figure 23.

[0111] In S210, the state transition rule generation unit 210 determines the matching state by the degree of matching, which indicates how many characters of each subword in the input dictionary have been confirmed to match. Using the dictionary and one codebook used for encoding, the unit generates and outputs a state transition rule for each matching state, which indicates the sequence of token IDs that will be output when a codeword in the one codebook is input at the degree of matching used to determine the matching state, and the matching state to which the transition will occur. The state transition rule generation unit 210 performs, for example, the following two processes.

[0112] (1) A process to generate an automaton for matching all subwords contained in the input dictionary. (2) A process to generate state transition rules for each matching state, using the state of the automaton generated in (1) as the degree of matching. The state transition rules are, for example, a state transition table as explained in <Technical Background>. In this case, the sequence of token IDs that is output when a codeword is input is the confirmed word and the mismatch candidate in the state transition table, and the matching state to which the codeword is input is the transition destination in the state transition table. Note that the confirmed word and the mismatch candidate in the above state transition table are all subwords contained in the dictionary and <unk>It is in the order of (see Figure 20), but the subwords and <unk>Using the associated token ID, confirmed words and, in case of mismatch, candidates are subwords or <unk>It can be easily converted into a corresponding sequence of token IDs.

[0113] In S220, the tokenization unit 220 uses the state transition rules generated in S110 to obtain a sequence of token IDs corresponding to the codewords that make up the input text data, generates an output sequence from the obtained sequence of token IDs, and outputs it.

[0114] (Modification 1) In the above description, the input text data was assumed to be a sequence of codewords encoded using one codebook, but it may also be a sequence of codewords encoded using multiple codebooks. In this case, the tokenization device 200 takes the input text data, which is a sequence of codewords encoded using multiple codebooks, and a dictionary as input, and applies a subword or included in the dictionary to the input text data. <unk>Outputs the output sequence, which is the sequence of token IDs corresponding to [the specified value].

[0115] The tokenization device 200 will be described below with reference to Figures 22 and 23. Figure 22 is a block diagram showing the configuration of the tokenization device 200. Figure 23 is a flowchart showing the operation of the tokenization device 200. As shown in Figure 22, the tokenization device 200 includes a state transition rule generation unit 210, a tokenization unit 220, and a recording unit 290. The recording unit 290 is a component that appropriately records information necessary for the processing of the tokenization device 200. For example, the recording unit 290 records multiple codebooks used for encoding. Note that instead of recording these multiple codebooks in the recording unit 290, they may be input.

[0116] The operation of the tokenization device 200 will be explained in accordance with Figure 23.

[0117] In S210, the state transition rule generation unit 210 defines the codebook-specific matching state as a state determined by a combination of a matching degree indicating how many characters of each subword in the input dictionary have been confirmed to match, and codebook information indicating which of the multiple codebooks used for encoding is used. Using the dictionary and the multiple codebooks, the unit generates and outputs a state transition rule for each codebook-specific matching state, which indicates the sequence of token IDs that will be output when a codeword included in the codebook indicated by the codebook information used to determine the codebook-specific matching state is input at the matching degree used to determine the codebook-specific matching state, and the destination codebook-specific matching state. The state transition rule generation unit 210 performs, for example, the following two processes.

[0118] (1) A process to generate an automaton for matching all subwords contained in the input dictionary. (2) A process to generate state transition rules for each codebook matching state, using the state of the automaton generated in (1) as the degree of matching. The state transition rules are, for example, a state transition table as described in <Technical Background>. In this case, the sequence of token IDs that is output when a codeword is input is the confirmed word and the mismatch candidate in the state transition table, and the codebook matching state to which the codeword is input is the destination in the state transition table. Note that the confirmed word and the mismatch candidate in the above state transition table are all subwords contained in the dictionary and <unk>It is in the order of (see Figure 21), but the subwords and <unk>Using the associated token ID, confirmed words and, in case of mismatch, candidates are subwords or <unk>It can be easily converted into a corresponding sequence of token IDs.

[0119] In S220, the tokenization unit 220 uses the state transition rules generated in S110 to obtain a sequence of token IDs corresponding to the codewords that make up the input text data, generates an output sequence from the obtained sequence of token IDs, and outputs it.

[0120] The tokenization process in this embodiment can be summarized as follows:

[0121] [1] A dictionary is a set whose elements are subwords (where each subword is associated with a token ID). <unk>A token is used to indicate that a subword is not included in the dictionary, and the dictionary is used to select a subword or from the input text data that is included in the dictionary. <unk>A tokenization device that generates an output sequence which is a sequence of token IDs corresponding to the input text data, wherein the input text data is a sequence of codewords encoded using one codebook, the matching state is determined by a matching degree that indicates how many characters of each subword included in the dictionary have been confirmed to match, a state transition rule generation unit that generates a sequence of token IDs which will be the output when a codeword included in the one codebook is input at the matching degree used to determine the matching state, and a state transition rule that indicates the matching state to transition to, using the dictionary and the one codebook, and a tokenization unit that uses the state transition rule to obtain a sequence of token IDs corresponding to the codewords constituting the input text data, and generates the output sequence from the obtained sequence of token IDs.

[0122] [2] A tokenization device according to [1], wherein the state transition rule generation unit generates an automaton for matching all subwords included in the dictionary, and generates the state transition rule using the state of the automaton as the degree of matching.

[0123] [3] A dictionary is a set whose elements are subwords (where each subword is associated with a token ID). <unk>A token is used to indicate that a subword is not included in the dictionary, and the dictionary is used to select a subword or from the input text data that is included in the dictionary. <unk>A tokenization device that generates an output sequence which is a sequence of token IDs corresponding to the input text data, wherein the input text data is a sequence of codewords encoded using a plurality of codebooks, the codebook-specific matching state is a state determined by a combination of a matching degree indicating how many characters of which subwords included in the dictionary have been confirmed to match and codebook information indicating which of the plurality of codebooks it is, a state transition rule generation unit that generates a sequence of token IDs which will be the output when a codeword included in the codebook indicated by the codebook information used to determine the codebook-specific matching state is input at the matching degree used to determine the codebook-specific matching state, and a state transition rule indicating the destination codebook-specific matching state, using the dictionary and the plurality of codebooks, and a tokenization unit that uses the state transition rule to obtain a sequence of token IDs corresponding to the codewords constituting the input text data, and generates the output sequence from the obtained sequence of token IDs.

[0124] [4] A tokenization device according to [3], wherein the state transition rule generation unit generates an automaton for matching all subwords included in the dictionary, and generates the state transition rule using the state of the automaton as the degree of matching.

[0125] [5] A dictionary is a set whose elements are subwords (where each subword is associated with a token ID). <unk>The token is used to indicate that a subword is not included in the dictionary, and the tokenization device uses the dictionary to identify subwords included in the dictionary or <unk>A tokenization method that generates an output sequence which is a sequence of token IDs corresponding to the input text data, wherein the input text data is a sequence of codewords encoded using one codebook, the matching state is a state determined by a matching degree that indicates how many characters of each subword included in the dictionary have been confirmed to match, the tokenization device generates a state transition rule generation step which generates a sequence of token IDs that will be the output when a codeword included in the one codebook is input at the matching degree used to determine the matching state, and a state transition rule that indicates the matching state to transition to, for each matching state, using the dictionary and the one codebook, the tokenization device obtains a sequence of token IDs corresponding to the codewords constituting the input text data using the state transition rule, and generates the output sequence from the obtained sequence of token IDs, [6] The dictionary is a set of subwords as elements (where a token ID is associated with each subword), <unk>The token is used to indicate that a subword is not included in the dictionary, and the tokenization device uses the dictionary to identify subwords included in the dictionary or <unk>A tokenization method for generating an output sequence which is a sequence of token IDs corresponding to the input text data, wherein the input text data is a sequence of codewords encoded using a plurality of codebooks, the codebook-specific matching state is a state determined by a combination of a matching degree indicating how many characters of which subwords included in the dictionary have been confirmed to match and codebook information indicating which of the plurality of codebooks it is, a state transition rule generation step in which the tokenization device generates a state transition rule indicating a sequence of token IDs which will be the output when a codeword included in the codebook indicated by the codebook information used to determine the codebook-specific matching state is input at the matching degree used to determine the codebook-specific matching state, and a destination codebook-specific matching state, using the dictionary and the plurality of codebooks, and a tokenization step in which the tokenization device obtains a sequence of token IDs corresponding to the codewords constituting the input text data using the state transition rule, and generates the output sequence from the obtained sequence of token IDs, the tokenization method comprising: [7] A program for causing a computer to function as a tokenization device of any of [1] to [4].

[0126] According to embodiments of the present invention, it is possible to efficiently tokenize encoded text data using a dictionary. In this case, there is no need to decode the encoded text data. Furthermore, by encoding the text data using a fixed-length code with high compression efficiency, the storage capacity of the text data can be saved.

[0127] <Note> The functions realized by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor is considered to be circuitry or processing circuitry, including transistors and other circuits. A processor may be a programmed processor that executes a program stored in memory.

[0128] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0129] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0130] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 24, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0131] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0132] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0133] A computer executing such a program may, for example, first store the program, either recorded on a portable storage medium or transferred from a server computer, in its own memory. Then, when processing is to be executed, the computer reads the program stored in its memory and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed using a so-called ASP (Application Service Provider) type service, where the processing function is realized only by issuing execution instructions and obtaining results, without transferring the program from the server computer to this computer.In addition, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. Furthermore, the term "program" in this form includes information used for processing by an electronic computer that is equivalent to a program (data, etc., that is not a direct instruction to the computer but has the property of defining the processing of the computer).

[0134] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware.

[0135] The present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. Furthermore, the processes described in the above embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually as needed, depending on the processing capacity of the device performing the process.< / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk> < / unk>

Claims

1. A string matching device that generates an output matching result, which is the result of matching one or more specified strings against input text data, wherein the input text data is a sequence of codewords encoded using a plurality of codebooks, and the codebook-specific matching state is a state determined by a combination of a matching degree indicating how many characters of the one or more specified strings have been confirmed to match and codebook information indicating which of the plurality of codebooks it is, and a state transition rule generation unit that generates a state transition rule indicating the output matching result and the destination codebook-specific matching state when a codeword included in the codebook indicated by the codebook information used to determine the codebook-specific matching state is input at the matching degree used to determine the codebook-specific matching state, using the one or more specified strings and the plurality of codebooks, using the one or more specified strings and the plurality of codebooks, and a matching unit that obtains matching results corresponding to the codewords constituting the input text data and generates the output matching result from the obtained matching results.

2. A string matching device according to claim 1, wherein the state transition rule generation unit generates an automaton for matching one or more specified strings, and generates the state transition rule using the state of the automaton as the degree of matching.

3. A string matching method comprising: a string matching device that generates an output matching result which is the result of matching one or more specified strings against input text data, wherein the input text data is a sequence of codewords encoded using a plurality of codebooks, the codebook-specific matching state is a state determined by a combination of a matching degree indicating how many characters of the one or more specified strings have been confirmed to match and codebook information indicating which of the plurality of codebooks it is, a state transition rule generation step in which the string matching device generates a state transition rule that indicates the output matching result and the destination codebook-specific matching state for each codebook-specific matching state, using the one or more specified strings and the plurality of codebooks, when a codeword included in the codebook indicated by the codebook information used to determine the codebook-specific matching state is input at the matching degree used to determine the codebook-specific matching state, and a matching step in which the string matching device obtains matching results corresponding to the codewords constituting the input text data using the state transition rule, and generates the output matching result from the obtained matching results.

4. A program for causing a computer to function as a string matching device according to claim 1 or 2.

Images