Full-Text Search Processor

Abstract

To provide a hard ware accelerator processor for full text retrieval.SOLUTION: A full text retrieval processor includes a character storage device for temporarily storing a text data of a retrieval target with allocating each byte thereof from a first address to a Nth address, a character detection circuit for receiving coded characters included in a retrieval keyword as comparison data for each byte and for sequentially detecting a storage position of all coded characters included in the retrieval keyword in the character storage device, a character string detection circuit for sequentially detecting positions in the character storage device of the encoded characters matched with an arrangement of all of the encoded characters included in the retrieval keyword, and a result output circuit for receiving a detection result of the character string detection circuit to output a head position or an end position of the character string matched with the retrieval keyword.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a full-text search processor that performs a full-text search using keywords on text data within a semiconductor device. [Background technology]

[0002] Generally, the process of finding specific document data from a large amount of document data (including sentences, literature, and texts) is called full-text search or keyword search, and is frequently used in a wide range of fields, including web searches, patent information searches, in-house document databases, and even PCs and smartphones.

[0003] Here, full-text search and keyword search are the basic information processing methods of natural language processing.

[0004] The full-text search process involves providing keywords (key characters or strings such as "search," "search," or "information") as search criteria, searching to see if these characters or strings are contained in documents, and identifying the data in the documents that contain them.

[0005] Conventional processors such as CPUs and GPUs are generally not good at searching or other information retrieval processes, and it takes a huge amount of time to read and search all document data without a heading (index). For this reason, an index called an inverted index is usually created in advance and used to speed up searches, which is the only way to speed up searches.

[0006] Here, the inverted index generally uses a method in which dictionary terms are used as entries (indexes) or a method in which character strings called N-grams are used as entries (indexes).

[0007] When dictionary terms are used as an index, English is easy to detect because it uses a method of creating sentences by inserting a space between each word (term), a method known as "word segmentation." However, in the case of Japanese or Chinese, this "word segmentation" rule cannot be applied.

[0008] Therefore, in the case of Japanese, a complex method of analysis called morpheme analysis is used to extract words (terms) in accordance with Japanese grammar.

[0009] Morphological indexing is characterized by a small number of indexes, and although it is good for forward matches, it is difficult to perform full-text searches for intermediate or backward matches, and it is difficult to handle new terms such as buzzwords.

[0010] On the other hand, the N-gram index was invented by Claude Elwood Shannon, the famous founder of information theory, for the purpose of natural language analysis.

[0011] Its distinctive feature is that it can handle forward matches, middle matches, backward matches, and full-text searches for new terms, but its disadvantage is that the number of indexes becomes enormous.

[0012] Given the above background, various methods have been developed, such as combining the advantages of morphological indexing and N-gram indexing.

[0013] Although using such indexes can speed up full-text searches and keyword searches, it has some major problems.

[0014] Even if text data of documents is prepared, it cannot be searched until an inverted index is created. Therefore, full-text search of stream information (data) and real-time processing of full-text search itself are difficult.

[0015] - The memory area and storage area are expanded due to the index, and it is difficult to expand because of the index.

[0016] Due to the various issues mentioned above, full-text search is difficult for anyone other than experts, and language differences make it difficult to standardize on a global level.

[0017] We will explain the prior art technology that implements full-text search in semiconductors.

[0018] US 2010 / 0185647 A1 is a semiconductor device designed to search for character data. When dealing with only 256 different characters, such as ASKII code, the XY matrix consisting of row decoders and feature cells can be as small as 256 x 256. However, when dealing with 3-byte or 4-byte structures such as Japanese characters in UTF-8 code, the XY matrix becomes enormous and difficult to implement.

[0019] Furthermore, this patent is intended for searching stream data, such as for detecting malware, and cannot be used for both stored data and stream data, as in the present application.

[0020] In order to solve the various problems of information detection described above, the inventor of the present application has made various inventions using in-memory computing, PIM (Process in Memory), and architecture, and has obtained patents as shown in Patent Documents 2 to 5 below.

[0021] However, none of the above inventions included an algorithm suitable for full-text search. [Prior art documents] [Patent documents]

[0022] [Patent Document 1] US 2010 / 0185647 A1 DEVICES, SYSTEMS, AND METHODS FOR COMMUNICATING PATTERN MATCHING RESULTS OF A PARALLEL PATTERN SEARCH ENGINE [Patent Document 2] Patent No. 4588114: Memory with information narrowing and detection function, method of use thereof, and device including this memory.

[0023] [Patent Document 3] Patent No. 5981666: Memory with information search function, its usage method, device, and information processing method.

[0024] [Patent Document 4] Patent No. 6014120: Memory with set operation function and set operation processing method using the same [Patent Document 5] Patent No. 6393852: Data comparison calculation processor and calculation method using the same. Summary of the Invention [Problem to be solved by the invention]

[0025] The purpose of this application is to provide a hardware accelerator processor for full-text search that eliminates the need to create indexes such as inverted indexes and yet has full-text search performance equivalent to that of systems that use N-gram inverted indexes, thereby fundamentally resolving the various issues that full-text search technology faces, improving natural language processing technology, and aiming to globally standardize full-text search. [Means for solving the problem]

[0026] In order to solve the above problems, according to a main aspect of the present invention, the following invention is provided.

[0027] (1) A full-text search processor consisting of a semiconductor device for full-text keyword search, A character memory element that receives the text data to be searched and temporarily stores the encoded character string contained in this text data by assigning each byte to addresses 1 to N; a character detection circuit that sequentially receives one or more coded characters included in a search keyword as comparison data byte by byte, compares each of the coded data with the coded character strings stored in the character storage element in N parallel sequences, and repeats this process for all coded characters included in the search keyword, thereby detecting the storage positions of all coded characters included in the search keyword in the character storage element; a character string detection circuit for detecting positions on the character storage element where all coded characters included in the search keyword are consecutive in the order in which they are included in the search keyword; a result output circuit that receives the detection result of the character string detection circuit and outputs the start position or end position of the continuous character string; A full-text search processor comprising:

[0028] (2) In the full-text search processor according to (1), The character detection circuit a set of N pairs of eight 1-bit match or mismatch calculation circuits connected to each address of the character storage element, for comparing a 1-byte / 8-bit code of the text data stored in each address with a 1-byte / 8-bit code constituting the comparison data, and detecting a match or mismatch for each bit; N logical operation (logical AND, logical OR, logical negation) circuits that receive the 8-bit result output from each set of 1-bit match or mismatch operation circuits and detect whether the coded characters of the text data stored in each address match or mismatch the coded characters that make up the comparison data; It has A full-text search processor comprising:

[0029] (3) In the full-text search processor according to (1), The character string detection circuit It consists of two circuits, an FG shift circuit and an FG winning circuit, to continuously determine the validity of the FG (flag) of the character memory position detected continuously by the character detection circuit and the FG (flag) of the detected character memory position. The FG shift circuit and the FG winning circuit refer to the sequence of coded characters included in the search keyword, and repeatedly determine the validity of the sequence of adjacent character codes from the character strings in the character storage elements allocated and stored at addresses 1 to N. These circuits detect, in N parallel rows, the start position (address) or end position (address) of the character string in the character storage elements that matches the character string specified by the search keyword. A full-text search processor characterized by:

[0030] (4) In the full-text search processor according to (1), The character string detection circuit N FG shift circuits that store the calculation results of the character detection circuit as flags and shift the stored FGs in N parallel ways; N FG winning circuits that perform a winning operation for the FG by shifting the stored FG in N parallel steps by the FG shift circuit and performing N parallel logical operations with the FG before the shift, and repeating this for all coded characters included in the search keyword; A full-text search processor comprising:

[0031] (5) In the full-text search processor according to (1), If the search keyword is composed of n bytes of coded characters, The number of shifts by the FG shift circuit and the number of corresponding winning calculations by the FG winning calculation circuit are n-1 times. A full-text search processor characterized by:

[0032] (6) In the full-text search processor according to (1), The FG winning circuit has a function that allows masking (ignoring) of operations, and enables full-text searches when wildcards are used in search keywords. A full-text search processor characterized by:

[0033] (7) In the full-text search processor according to (1), The FG winning circuit incorporates two sets of registers, enabling full-text searches that include character gaps in the search keywords. A full-text search processor characterized by:

[0034] (8) In the full-text search processor according to (1), A logical sum (OR) operation is performed on the N parallel (full parallel) detection results output by the result output circuit, and the presence or absence of a full-text detection result is output. A full-text search processor characterized by:

[0035] (9) In the full-text search processor according to (1), A full-text search processor characterized by transferring text data from a memory or storage external to the full-text search processor as batch data to a character memory element that temporarily stores the N bytes of character text data, and repeating the N-parallel (fully parallel) full-text search operation.

[0036] (10) In the full-text search processor according to (1), A full-text search processor characterized by transferring text data from the internal memory or storage of the full-text search processor as batch data to a character memory element that temporarily stores the N bytes of character text data, and repeating the N-parallel (fully parallel) full-text search operation.

[0037] (11) In the full-text search processor according to (1), It is characterized by being implemented on ASIC and FPGA. Full-text search processor.

[0038] (12) In the full-text search processor according to (1), A full-text search processor characterized by its built-in CPU.

[0039] (13) In the full-text search processor according to (1), A method for using a full-text search processor that enables full-text searches common to all languages ​​in the world by applying global standard character codes such as UTF-8.

[0040] (14) In the full-text search processor according to (1), A full-text search processor characterized by the fact that it either executes a full-text search operation or does not execute it if the string of search keywords provided from outside contains a character code specified in advance.

[0041] (15) In the full-text search processor according to (1), This full-text search processor is characterized by comparing a part of a sentence being created with a full-text search processor that has accumulated a large amount of text data containing the latest information, and determining that there is no precedent if no text is found.

[0042] (16) In the full-text search processor according to (1), This full-text search processor is characterized by improving the accuracy of speech recognition by selecting the optimal recognition result from multiple recognition candidates in speech recognition by comparing it with a full-text search processor that has accumulated a large amount of text data containing the latest information and selecting terms that produce a large number of hits. [Effects of the Invention]

[0043] Full-text search processing, such as web searches, patent searches, corporate data searches, and data searches on PCs and smartphones, is an indispensable form of information processing that is closely related to our work and lives.

[0044] However, current full-text search processing using computing has to rely on indexes such as inverted indexes, which makes real-time processing difficult, systems cannot be built by anyone other than experts, and language differences prevent progress in global standardization.

[0045] By using the full-text search processor of the present invention, it is not necessary to use an inverted index, and yet full-text search with performance comparable to that of a method using an inverted index becomes possible.

[0046] Therefore, the evolution of natural language processing (knowledge processing) technology will accelerate, and since it can be used in common for languages ​​of all countries, full-text search technology can be standardized globally. [Brief explanation of the drawings]

[0047] [Figure 1] FIG. 1 is a diagram illustrating a basic configuration of a full-text search processor according to one embodiment (first example) of the present invention.

[0048] [Figure 2] FIG. 10 is a diagram for explaining the detailed configuration of the character storage element and character detection circuit of the full-text search processor.

[0049] [Figure 3] FIG. 10 is a diagram for explaining the detailed configuration of a character string detection circuit and a result output circuit of the full-text search processor.

[0050] [Figure 4] FIG. 10 is a diagram for explaining a specific example of full-text search operation condition generation by the command generation circuit.

[0051] [Figure 5] FIG. 10 is a diagram for explaining data state transition-A (character detection process) in a standard full-text search of English text.

[0052] [Figure 6]Similarly, this is a diagram for explaining data state transition-B (character string detection process) in a standard full-text search of English text.

[0053] [Figure 7] FIG. 10 is a diagram for explaining data state transition-A (character detection process) in a standard full-text search of Japanese text.

[0054] [Figure 8] FIG. 10 is a diagram for explaining data state transition-B (character string detection process) in a standard full-text search of Japanese text.

[0055] [Figure 9] Similarly, this is a diagram for explaining data state transition-A (character detection process) in a full-text search in which wildcards are applied to English sentences.

[0056] [Figure 10] Similarly, this is a diagram for explaining data state transition-B (character string detection process) in a full-text search in which wildcards are applied to English text.

[0057] [Figure 11] Similarly, this is a diagram for explaining data state transition-A (character detection process) in full-text search that applies character gaps to English text.

[0058] [Figure 12] Similarly, this is a diagram for explaining data state transition-B (character string detection process) in full-text search that applies character gaps to English text.

[0059] [Figure 13] FIG. 10 is a diagram for explaining the overall configuration of a full-text search processor.

[0060] [Figure 14] FIG. 10 is a diagram illustrating the configuration of a full-text search processor when performing a refined search.

[0061] [Figure 15] FIG. 10 is a diagram for explaining an overview of an external memory type full-text search processor.

[0062] [Figure 16] FIG. 10 is a diagram for explaining an outline of data transfer between an external memory type full-text search processor and an external memory or storage.

[0063] [Figure 17] FIG. 10 is a diagram for explaining a time chart of batch processing of the external memory type full-text search processor.

[0064] [Figure 18] This is also a diagram summarizing the computing power of external memory type full-text search processors.

[0065] [Figure 19] FIG. 10 is a diagram for explaining an outline of an internal memory type full-text search processor.

[0066] [Figure 20] This is also a diagram summarizing the computing power of internal memory type full-text search processors.

[0067] [Figure 21] FIG. 10 is a diagram illustrating an example of a system configuration when a full-text search processor is used.

[0068] [Figure 22] FIG. 10 is a diagram illustrating the basic configuration of a full-text search processor according to a second example of an embodiment of the present invention.

[0069] [Figure 23] FIG. 10 is a diagram for explaining the detailed configuration of a character string detection circuit and a result output circuit of the full-text search processor in the second example.

[0070] [Figure 24]FIG. 10 is a diagram for explaining a specific example of full-text search operation condition generation by the command generation circuit in the second example.

[0071] [Figure 25] FIG. 10 is a diagram for explaining data state transition-A of each function of a standard full-text search for English text in the second example.

[0072] [Figure 26] FIG. 10 is a diagram for explaining data state transition-B of each function of a standard full-text search for English text in the second example.

[0073] [Figure 27] FIG. 10 is a diagram for explaining data state transition-A of each function of a standard full-text search for Japanese text in the second example.

[0074] [Figure 28] FIG. 10 is a diagram for explaining data state transition-B of each function of a standard full-text search for Japanese text in the second example.

[0075] [Figure 29] FIG. 10 is a diagram illustrating a data state transition-A of a full-text search that applies wildcards to English sentences in the second example.

[0076] [Figure 30] FIG. 10 is a diagram illustrating a data state transition-B of a full-text search that applies wildcards to English sentences in the second example.

[0077] [Figure 31] FIG. 10 is a diagram for explaining data state transition-A of a full-text search that adapts gaps in English sentences in the second example.

[0078] [Figure 32] FIG. 10 is a diagram for explaining data state transition-B of a full-text search that adapts gaps in English sentences in the second example. DETAILED DESCRIPTION OF THE INVENTION

[0079] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0080] The full-text search processor 101 according to the embodiment of the present invention can be used with any character code and provides a configuration that can realize an advanced and efficient full-text search.

[0081] Before explaining the configuration of this embodiment, the concept of full-text search implemented in the present invention will be explained.

[0082] (The concept of full-text search in this invention) First, character text data 132 contained in a document is expressed using various encoded characters, that is, character codes, such as ASCII (American Standard Code for Information Interchange), Shift JIS, and UTF-8 (UCS Transformation Format 8).

[0083] ASCII is 7 bits or 1 byte, Shift-JIS is 2 bytes, and the international standard UTF-8 is variable length.

[0084] In the case of UTF-8, most Japanese characters consist of three bytes.

[0085] Therefore, in general, to properly read character strings contained in document data, it is necessary to identify the character code and read any character string based on that.

[0086] Furthermore, in order to perform a high-speed full-text search with little search waiting time, it is necessary to create an inverted index based on the character text data 132 and perform a full-text search using this inverted index.

[0087] In contrast, in this embodiment, the character text data 132 to be searched is stored in a memory element for each byte (8 bits), and the ``characters'' and ``character sequences'' of the character string of the given search keyword 125 are compared in parallel for each byte to see if they match, and the position (address) of the character text data 132 that corresponds to the beginning or end of the character string of the given search keyword 125 is returned as the full-text search result.

[0088] This allows full-text searches to be performed with a simple circuit configuration regardless of the character code, and also enables high-speed full-text searches without creating an inverted index.

[0089] The specific configuration of this embodiment will be described below.

[0090] (Basic circuit configuration (first example)) Figure 1 shows the basic configuration of a full-text search processor.

[0091] This full-text search processor 101 is connected to a host computer (hereinafter referred to as "HOST"), and performs parallel full-text search operations on character text data 132 to be searched, which is provided by the HOST, using search keywords 125 also provided by the HOST as search conditions, and returns the location (address) of the character text data 132 detected as a result to the HOST.

[0092] To execute this process, the full-text search processor 101 has a configuration in which a full-text search circuit 103 and a command generation circuit 127 are connected to an input / output interface 115 connected to the HOST.

[0093] The full-text search circuit 103 includes a character storage element 102 that stores character text data 132 to be searched, a character detection circuit 105 that detects characters included in a search keyword 125 from the character text data 132 stored in the character storage element 102, a character string detection circuit 106 that identifies the position (address) of a character in the character text data 132 that corresponds to the first character or the last character of the character string of the search keyword 125 based on the character detection result, and a result output circuit 107 that outputs the detection result of the character string detection circuit 106 in a predetermined format.

[0094] As shown enlarged in FIG. 1, the command generation circuit 127 is made up of a system clock generation circuit 131 that generates a system clock 131, a comparison data generation circuit 123 that generates comparison data 123 to be given to the character detection circuit 105 based on the search keyword 125, a shift clock generation circuit 130 that determines the timing of giving the winning calculation condition 129 to the character string detection circuit 106 after character detection, and a winning calculation condition generation circuit 129 that generates the winning calculation condition 129 to be given to the character string detection circuit 106.

[0095] (Command generation circuit) The configurations of the full-text search circuit 103 and the command generation circuit 127 will be described in detail below, but for convenience of explanation, the command generation circuit 127 will be described first.

[0096] The system clock generation circuit 131 of the command generation circuit 127 generates a system clock 131 that is the basis for the full-text search processor 101 to perform full-text search operations at a predetermined operation timing, for example, a continuous clock every 10 nsec or 20 nsec, and the comparison data generation circuit 123, the shift clock generation circuit 130, and the knockout operation condition generation circuit 129 operate using (synchronizing with) this system clock 131.

[0097] The above-mentioned comparison data generation circuit 123, shift clock generation circuit 130, and tournament-winning calculation condition generation circuit 129 generate full-text search calculation conditions 114 consisting of three types of calculation conditions, namely, comparison data 123, shift clock 130, and tournament-winning calculation condition 129, to be given to the character detection circuit 105 and string detection circuit 106 based on the search keyword 125 set by the keyword setting function 128 of the HOST.

[0098] In this embodiment, the search keywords 125 include English keywords each consisting of one byte per character, Japanese keywords each consisting of three bytes per character, and other multilingual keywords.

[0099] As shown in Figure 2, for example, if the search keyword 125 is the English word "search," this keyword is composed of character codes of "s," "e," "a," "r," "c," and "h," each of which has one byte, for a total of six bytes.

[0100] Furthermore, if the search keyword 125 is the two Japanese characters "search," each kanji data is composed of three bytes, i.e., for "search," it is composed of "search:1 / 3," "search:2 / 3," and "search:3 / 3," and for "search," it is composed of "search:1 / 3," "search:2 / 3," and "search:3 / 3," resulting in a total of six bytes of character code.

[0101] (Comparison data generation circuit of command generation circuit) The comparison data generation circuit 123 of the command generation circuit 127 is configured to decompose the search keyword 125 into one byte, i.e., eight bits of data (0 or 1), generate comparison data 123 for each byte, and provide it to the character detection circuit 105, as shown in FIG. 2.

[0102] Specifically, in synchronization with the system clock 131 signal generated by the system clock generating circuit 131, one-byte character codes are extracted in order from the beginning or end of the search keyword 125, and are provided to the character detection circuit 105 as comparison data 123.

[0103] As will be explained later, when generating the comparison data 123, the comparison data generating circuit 123 uses special characters (wildcard symbols " ? ", gap (hereinafter also referred to as Gap) operator " * ", etc.), the special characters are ignored or replaced with a specified character code.

[0104] Details will be described later in the explanation of the operation, but for example, if a specific wildcard (e.g., " ? ") is included, a process is performed to mask (hereinafter referred to as "mask" or "ignore") the characters that correspond to this wildcard. Although not shown, this process according to the special character code can be performed by using, for example, a special character lookup table to distinguish between normal characters and special characters.

[0105] The comparison data generating circuit 123 is also configured to count the number of bytes of the character string that constitutes the search keyword 125 and pass the result to the winning calculation condition generating circuit 129 .

[0106] (Shift clock generation circuit of command generation circuit) Next, the shift clock generation circuit 130 generates a shift clock 130 that provides timing for providing specific calculation conditions for character string detection to the character string detection circuit 106. Specifically, this shift clock generation circuit 130 is a circuit that provides a shift clock 130 signal that indicates a predetermined shift timing in synchronization with the above-mentioned system clock 131 to the FG (flag) shift circuit 112 of the character string detection circuit 106 shown in FIG.

[0107] Specifically, if the search keyword 125 is 6 bytes, the number of shifts given to the FG shift circuit 112 is 5, which is 6 minus 1. This also applies when the search keyword 125 contains special characters such as a wildcard symbol or a gap operator.

[0108] The shift clock generating circuit 130 is also configured to determine the timing for providing the calculation conditions to the knockout calculation condition generating circuit 129 .

[0109] (Command generation circuit for generating winning operation conditions) The winning operation condition generating circuit 129 provides operation conditions to the character string detecting circuit 106 in conjunction with the shift clock generating circuit 130 described above.

[0110] Specifically, a predetermined winning operation condition 129 is given to the FG winning circuit 113 of the character string detection circuit 106 shown in Figure 3 by selecting from "direct input," "logical AND," "logical OR," "exclusive OR," "mask (ignore)," "gap operation," "logical negation," etc.

[0111] Details will be given later, but for example, if the search keyword is 6 bytes long and consists of normal characters that do not contain special characters such as wildcards, the calculation command "direct input" is selected when detecting characters (first step (1st step)), and the calculation command "logical AND" is selected when searching for strings (5th to 11th steps) and given to the FG winning circuit 113.

[0112] The selection of the calculation conditions can be determined by, for example, referring to a look-up table.

[0113] (Full-text search circuit) Next, the full-text search circuit 103 will be described.

[0114] (Character memory element of full-text search circuit) First, as shown in FIG. 2, the character memory element 102 of this full-text search circuit 103 is a register or memory such as a flip-flop that temporarily stores character text data 132 given from the HOST through the input / output interface 115 in 8-bit, or 1-byte units, and is configured to store N pieces of 1-byte data from address 1 to address N in total.

[0115] Here, character text data 132 refers to all text data such as web text data, text data such as novels, magazines, and papers, patent document text data, in-house document text data, and even email text data, Word, Excel, and so on for PCs and smartphones.

[0116] The character text data 132 varies in size from a few bytes to hundreds of thousands of bytes or more per item.

[0117] Furthermore, the character text data 132 is composed of a large number of document data (text data), ranging from a few items to tens of billions of items.

[0118] In this embodiment, a portion of the character text data 132, for example 32K bytes of character text data 132, from addresses 1 to N, is stored in the character storage element 102 from among the character text data 132 of various sizes as described above, and a full-text search is performed on the stored character text data 132.

[0119] 2 shows an example in which the character string "est..." after the "y" in the character string "yesterday" is stored in the character storage element 102. In this case, if address 126 is "e," address 2 is "s," address 3 is "t," and address N is "h," the UTF-8 character codes are stored in the character storage element 102 as "01100101" for address 1, "01110011" for address 2, "01110100" for address 3, and "01110100" for address N.

[0120] (Character detection circuit for full-text search circuit) Next, as shown in FIG. 2, the character detection circuit 105 of the full-text search circuit 103 is composed of a 1-bit match detection circuit 109 connected to the comparison data generation circuit 123 of the command generation circuit 127, and an 8-input AND circuit 110 connected to this 1-bit match detection circuit 109.

[0121] (1-bit match detection circuit for character detection circuit) 2, the one-bit match detection circuit 109 is provided corresponding to each of the eight memory element cells of one byte that make up the character memory element 102, with one input connected to each of those memory cells and the other input connected to each of the eight-bit data of one byte received as the comparison data 123. The one-bit match detection circuit 109 is configured to perform a match detection operation for each bit of both inputs and output the result to an eight-input AND circuit 110.

[0122] Therefore, the 1-bit match detection operation circuit 109 is made up of N×8 match circuits connected in parallel, and executes N parallel operations on the N-byte character text data 132 .

[0123] (8-input logical product circuit for character detection circuit) The eight-input AND circuit 110 is provided for each byte, and receives the outputs of one byte, that is, eight one-bit match detection circuits 109, performs a logical AND operation on these, and outputs the result.

[0124] Therefore, the 1-bit match detection circuit 109 and the 8-input AND circuit 110 perform a comparison of the 1-byte comparison data 123 provided from the command generation circuit 127 with the N-byte character text data 132 in parallel across all bytes.

[0125] 2, the 8-input AND circuit 110 corresponding to address 2 of the matching full-text search circuit address 126 outputs a flag (FG) of "1" indicating a match of 8 bits of data, and outputs "0" for addresses that do not match. Note that in this embodiment, character detection is performed based on match detection, but this can also be achieved by combining a mismatch detection circuit (XOR), an 8-input OR circuit 111, a logical NOT circuit, or other circuits.

[0126] (String detection circuit for full-text search circuit) Next, the character string detection circuit 106 will be described with reference to FIG.

[0127] The character string detection circuit 106 is composed of an FG shift circuit 112 and an FG winner circuit 113.

[0128] (FG shift circuit of character string detection circuit) First, the FG shift circuit 112 is composed of N shift registers with a preset function, corresponding to the number of the 8-input AND circuits 110.

[0129] In this example, the output of the 8-input AND circuit 110 is connected to the preset input "P" of this FG shift circuit 112. The data output "Q" of the shift register at the first address is connected to the data input "D" of the shift register at the second address, and the data output "D" of the shift register at the second address is connected to the data input "Q" of the shift register at the third address.

[0130] The subsequent shift registers are connected in the same manner up to the Nth address, and a shift clock 130 given from the command generation circuit 127 is connected to each shift register.

[0131] With this configuration, the character detection circuit 105 detects a character or a portion of a character (1-byte code) that matches the search keyword 125 from the character text data 132 as a match flag (FG), and then shifts the position of this flag sequentially by n-1 times (corresponding to the number of shift clocks) as the number of bytes in the character string of the search keyword 125, so that the FG winning circuit 113, which will be described next, can detect the position of consecutive flags for the number of bytes in the search keyword 125, i.e., the character sequence 122 (character string).

[0132] (FG winning circuit for character string detection circuit) As shown in Figure 3, the FG winning circuit 113 is an N-circuit provided corresponding to the FG shift circuit 112, and is composed of a group of logic circuits (or elements) that can perform logical negation, direct input, logical product, logical sum, exclusive logical sum, mask, and gap operations, a select circuit for selecting operation conditions, and a winning register for storing the operation results.

[0133] With this configuration, as will be described in detail later, it is possible to detect the head flag position of n consecutive flags, the number of bytes of the search keyword 125, from the match flag data stored in the FG shift circuit 112.

[0134] The calculation condition input of the selection circuit is connected to a winning calculation condition generation circuit 129 of the command generation circuit 127, and the winning calculation condition generation circuit 129 of the command generation circuit 127 determines which calculation condition to use: logical negation, direct input, logical product, logical sum, exclusive logical sum, mask (ignore), or gap calculation.

[0135] That is, as described above, the winning calculation conditions 129 determine the calculation conditions in the character string detection circuit 106 based on the character string and character type specified as the search keyword 125, and as a result, as will be described in detail later, a winning calculation process is performed according to the character string included in the search keyword 125, and each time the winning calculation is performed, the winning calculation result is stored as FG in the winning register.

[0136] Typically, the winning register that survives the winning calculation (n x 2 - 1 times, the number of bytes in the string of the search keyword 125) will be stored with a logic "1", and the others will be stored with a logic "0", and the final winning FG will be stored in the winning register.

[0137] (Full-text search circuit result output circuit) 3, the result output circuit 107 is a circuit that is provided with N circuits corresponding to the FG winning circuits 113 of the N circuits, and is configured to output the calculation result of "1" or "0" of the winning register of the FG winning circuit 113. In addition to a configuration that outputs "1" and "0" of all addresses, it may be configured to output only addresses where the winning register is "1", or, as will be described later, a configuration that outputs "calculation result available" if there is even one "1" among the N calculation results, or "no calculation result available" if there is no "1" - that is, only a "yes / no" result.

[0138] The output from the output circuit 107 is configured to be returned as the calculation result to the HOST via the input / output interface 115 as shown in FIG.

[0139] (Full-text search processor operation) The above configuration will be explained in more detail below through the operation of this device.

[0140] (Text data storage operation) First, the character text data 132 to be searched is stored in the character storage element 102 .

[0141] At this time, the character text data 132 is transferred to the character storage element 102 via the input / output interface 115 directly from the CPU of the host or by DMA (Direct Memory Access) method, and N bytes of character code is stored.

[0142] In most cases of ASCII English text, the entire character code can be expressed in one byte, so the character storage element 102 stores N characters.

[0143] On the other hand, in the case of Shift JIS Japanese characters, a code representing one character can be expressed with two bytes, so that the character storage element 102 stores N / 2 characters.

[0144] Furthermore, in the case of Japanese text in UTF-8, the entire code representing one character can be expressed using three bytes, so that the character storage element 102 stores approximately N / 3 characters.

[0145] It is also possible for the character storage element 102 to store a mixture of the above-mentioned multiple types of character codes.

[0146] (Character detection operation) Next, a character detection operation is performed based on the search keyword 125 provided by the host, and then character string detection is performed.

[0147] In the character detection operation, character detection is performed based on the comparison data 123 provided from the command generation circuit 127.

[0148] The character detection circuit 105 is composed of eight 1-bit match detection circuits 109 per byte connected to the output of each memory cell of the character storage element 102, and one 8-input logical product (AND) circuit 110 per byte connected to the output of these eight 1-bit match detection circuits 109.Therefore, if the character code of the specified comparison data 123 matches the character code of the character storage element 102, the output of the logical product circuit 110 at address 126 of the target full-text search circuit will be logical "1".

[0149] In the example of FIG. 2, the comparison data 123 given from the command generation circuit 127 as "s": "01110011" and address 2 of the full-text search circuit address 126 match in all eight 1-bit match detection circuits 109, indicating that the AND condition of the logical product circuit 110 is satisfied (8-bit match).

[0150] This result is stored as a character detection result for each byte in the FG shift circuit 112 described later as an FG (flag) with a logic value of "1", and since the addresses of the other full-text search circuits in address 126 do not match, they are stored as a character mismatch in the FG shift circuit 112 with a logic value of "0".

[0151] (String detection operation) Next, a character string detection operation is performed based on the result of the character detection.

[0152] At this time, the character string is detected based on the shift clock 130 and the command of the winning calculation condition 129 given from the command generation circuit 127 .

[0153] As described above, the character string detection circuit 106 is composed of the FG shift circuit 112 and the FG winning circuit 113.

[0154] The FG shift circuit 112 has the function of storing and data shifting the match results (FG) of the parallel character match detection by the character detection circuit 105, such as the aforementioned "s", "e", "a", "r", "c", and "h", as well as character match results such as "Search: 1 / 3", "Search: 2 / 3", "Search: 3 / 3", "Search: 1 / 3", "Search: 2 / 3", and "Search: 3 / 3".

[0155] The FG shift circuit 112 stores N preset FGs from the output of the character detection circuit 105 in full parallel (N parallel) and shifts the data every time a shift clock 130 for shift operation is generated from the command generation circuit 127. (Specific examples of this data shift will be described later with reference to FIGS. 5 to 12.) On the other hand, the FG winning circuit 113 has the function of determining whether or not the detected character code sequence such as "s", "e", "a", "r", "c", "h" or "Search:1 / 3", "Search:2 / 3", "Search:3 / 3", "Search:1 / 3", "Search:2 / 3", "Search:3 / 3" matches the character sequence 122 (character string) of the character code given as the search keyword 125 (whether or not the character arrangement is valid), and detecting the address 126 of the full-text search circuit that corresponds to the first address or last address of the character string of the search keyword 125.

[0156] This FG winning circuit 113 is composed of logical elements such as logical product and logical sum, as well as exclusive OR and logical negation, for performing advanced winning operations, a selection circuit for selecting the logical operation of the above logical elements, and a winning register for storing the operation results output from the selection circuit.

[0157] The selection circuit receives as input the operation conditions given by the knockout operation condition generation circuit 129 of the command generation circuit 127, and the selection signals for logical negation, logical product, logical sum, exclusive sum, mask (ignore), and gap operation.

[0158] In this embodiment, the tournament register is composed of two registers, a tournament main register and a tournament sub-register that holds the intermediate results of the tournament calculation, in order to improve the convenience and performance of the calculation, and the command generation circuit 127 controls which of these registers to use.

[0159] As an example, in the case of normal calculations, the winner main register is used, and in the case of special calculations such as gap calculations, both the winner main register and the winner sub-register can be operated.

[0160] Therefore, when simply referring to a winning register, it means a winning main register, and in the case of gap calculation, the main and sub registers will be distinguished in the description.

[0161] It should be noted that this circuit configuration shows the concept of a knockout operation, and therefore shows conceptual content such as the configuration of logic circuits and registers, and is not limited to this configuration.

[0162] A specific tournament calculation will be described later with reference to FIG. 4 and subsequent figures, but the general contents of the tournament calculation are as follows.

[0163] That is, when direct input is selected in the winning operation condition 129 as the first step of detecting a character string, the input received from the output of the FG shift circuit 112 is directly substituted into the winning register of the FG winning circuit 113 .

[0164] In this example, the above calculation results in a logic "1" at address 2 of address 126 of the full-text search circuit, and a logic "0" at the other addresses.

[0165] In the subsequent step, when a logical product is selected in the winning operation condition 129 in synchronization with one shift clock 130 for the shift operation given from the command generation circuit 127, the first (initial) winning operation is executed by performing a logical product operation on the output of the winning register and the input received from the FG shift circuit 112.

[0166] As a result of a predetermined number of clock shifts 130 and the winning operation, the winning register output of the FG winning circuit 113 at address 126 of the winning full-text search circuit becomes logic "1" and the other register outputs become logic "0". When "search" is the search keyword 125, the address 126 of the winning full-text search circuit stores the character code corresponding to the first address "s" of the character string of the search keyword 125 to be searched.

[0167] If the keyword "search" is found backwards, such as "hcraes," the last address "h" will be found.

[0168] The above is an outline of the operation for detecting a general character sequence 122 (character string).

[0169] A specific example of full-text search will be explained in detail with reference to FIG. 4 and FIGS. 5 to 12.

[0170] (Result output operation) The result output circuit 107 has the function of outputting to the HOST the output of the FG winning circuit 113 when a predetermined number of character strings 122 (character strings) are found as logic "1" and the other outputs as logic "0".

[0171] The logic state of the FG winning circuit 113 is output from the full-text search operation result output circuit 107 to the HOST via the input / output interface 115. The method of outputting the results may be any method, such as outputting the logical states of all addresses or outputting the numbers (addresses) of the winning addresses. As explained above, this full-text search processor 101 performs full-text search operations with a parallelism of N using N full-text search circuits 103, which are composed of character storage elements 102, character detection circuits 105, and character string detection circuits 106, thereby realizing advanced, efficient, and high-speed full-text searches.

[0172] The above configuration can be implemented using only general-purpose logic, so it can easily be realized using an ASIC (Application Specific Integrated Circuit) as well as an FPGA (Field Programmable Gate Array). More details will be given later.

[0173] (Command generation operation) Next, the command generation operation by the command generation circuit 127, that is, the full-text search operation condition generation operation, will be described with reference to FIG.

[0174] As described above, the command generation circuit 127 generates a predetermined command based on the search keyword 125 given by the host.

[0175] The command generation circuit 127 has the function of generating full-text search commands based on various search conditions, such as not only search keywords 125 such as "search" but also search keywords 125 including wildcards and character gaps (allowance for character position).

[0176] In this example, four types of commands (instructions) are shown as representative examples: (1) English standard (full-text search in case of), (2) Japanese standard (full-text search in case of), (3) English wildcard (full-text search in case of), and (4) English character gap (full-text search in case of).

[0177] The first line of each table shows the operation steps (order) of the full-text search operation, and indicates at which step what command (instruction) is given to the full-text search circuit 103. Each step is executed in synchronization with the system clock 131.

[0178] The "Comparison Data" 123 on the second line shows comparison data such as "s", "e", "a", "r", "c", "h" and "Search:1 / 3", "Search:2 / 3", "Search:3 / 3", "Search:1 / 3", "Search:2 / 3", "Search:3 / 3" given to the character detection circuit 105.

[0179] In the "shift clock" 130 on the third line, a "circle" indicates the clock feed given from the shift clock generation circuit 130 to the FG shift circuit 112. For example, in the example of "(1) English Standard," comparison data is given in the first six steps, and the shift operation by the FG shift circuit 112 is performed in the next steps 7 to 11.

[0180] The fourth line, "Winning Operation Conditions" 129, shows the winning operation conditions to be given to the FG winning circuit 113, such as "Direct Input", "Logical AND", "Mask (Ignore)", and "Gap Operation".

[0181] In Figure 4, (1) full-text search for standard English text and (2) full-text search for standard Japanese text are examples of standard full-text searches. In this example, the English text is "search" with 6 characters and 1 byte, and the Japanese text is "search" with 2 characters and 3 bytes (both 6 bytes).

[0182] On the other hand, the English wildcard in (3) is the "a" in "search" and the "r" in "search". ? "," ? In this case, the comparison data 123, the shift clock 130, and the winning calculation condition 129 are as shown in FIG. ?The wildcard represented by the special character " is ignored as comparison data 123 by referring to the special character lookup table mentioned above, and then a 6-byte shift operation is performed, so the total number of steps (9 in this example) is fewer by the number of wildcard characters (2 characters in this example) than the number of steps in the normal search of (1) and (2) above (11 in this example).

[0183] (4) Full-text search for gaps in English letters is performed when the letter "h" is in one of the three positions after "s", "e", or "a", e.g., * This is an example command for matching "seah", "sea*h", and "sea**h" using an operation condition that indicates a range of positions and special characters such as "3Gap".

[0184] As mentioned above, in this example, the command generation circuit 127 is explained as being placed inside the full-text search processor 101, but it is also possible to send the commands from the HOST CPU to the full-text search processor 101 step by step or all at once through software processing.

[0185] Below are the contents of full-text search operations for a full-text search in English standard, a full-text search in Japanese standard, a full-text search in English wildcards, and a full-text search in English character gaps.

[0186] (Standard full-text search behavior for English text) FIG. 5 illustrates data state transition A (character detection process) in a standard full-text search of English text.

[0187] FIG. 6 explains data state transition-B (string detection process) in a standard full-text search of English text.

[0188] The explanation will be given assuming that the commands introduced in FIG. 4(1) are sequentially given to the full-text search circuit 103 for each step, and the full-text search circuit 103 performs full-text search processing based on these commands.

[0189] In this embodiment, data state transitions in each configuration of 16 full-text search circuits 103 (16 addresses) that are part of N full-text search circuits 103 (N addresses) are shown.

[0190] In this embodiment, it is assumed that the 16-character English text data "full text search" in UTF-8 character code is stored in the character storage element 102 of the full-text search processor 101 in 16 bytes from the i+1 byte (hereinafter simply referred to as the relative address 1) to the i+16 byte (hereinafter simply referred to as the relative address 16) of the relative address 104 of the full-text search circuit of the character storage element 102.

[0191] In this embodiment, a full-text search is performed on this character text data 132 using the six characters "search" as a search keyword 125 with a byte count n of 6 (6 byte length).

[0192] Although omitted in FIGS. 5 to 12, in the initial state of step 0, all FG shift circuits 112 and FG winning circuits 113 are cleared and in a logic "0" state.

[0193] The processing from step 1 to step 11 is controlled by the full-text search circuit 103 in accordance with commands (comparison data 123, shift clock 130, and knockout calculation condition 129) given by the command generation circuit 127.

[0194] Steps 1 to 6 shown in Figure 5 correspond to steps 1 to 6 in (1) of Figure 4, and show the processing steps for detecting the character "search" from the stored character text data 132 of "full text search."

[0195] Parts to note are shown with the text inverted.

[0196] In step 1, the character "s" consisting of one byte is detected, and in this embodiment, the character detection circuit 105 at relative address 11 detects "s".

[0197] The detected FG is preset in the FG shift circuit 112, and the FG shift circuit 112 at relative address 11 becomes logic "1", while the others become logic "0". Furthermore, this FG is the first winning FG, and by selecting "direct input" as the calculation condition for the FG winning circuit 113, a logic value of "1" is set in (the register of) the FG winning circuit 113 at relative address 11.

[0198] In step 2, the character "e" consisting of one byte is detected. In this embodiment, the character detection circuits 105 at relative addresses 7 and 12 detect "e."

[0199] The detected FG is set in the FG shift circuit 112. At this time, as shown in FIG. 4(1), no calculation condition is given to the FG winning circuit 113, and the logical value "0" remains set in the corresponding winning circuit.

[0200] Steps 3 to 5 are the same as step 2 and are therefore omitted here.

[0201] In the final step 6, the character "h" consisting of one byte is detected, and in this embodiment, the character detection circuit 105 at relative address 16 detects "h".

[0202] A logical value of "1" is set in the FG shift circuit 112 as the detected FG.

[0203] In this example, all characters of the given search keyword 125 "search" are detected consecutively from relative address 11 in six steps, the number of which is the same as the number of bytes n of the character string of the given comparison data 123.

[0204] It is noteworthy that in step 6, the six consecutive FGs "1111111" from relative address 11 to relative address 16 of the FG shift circuit 112 and the FG at relative address 11 of the FG winning circuit 113 set in step 1 are "1".

[0205] FIG. 6 shows data state transition B (detection of continuous character sequences) after step 6.

[0206] Step 6 in FIG. 6 is the same as the end result of character detection described above.

[0207] Steps 7 to 11 correspond to (1) in FIG. 4, and represent a "character string detection" process for detecting the character string 122 (character string) of "search" detected in steps 1 to 6 above.

[0208] In step 7, one clock signal is given to the FG shift circuit 112 from the shift clock generation circuit 130, and the FG of the FG shift circuit 112 shown in step 6 is shifted one byte to the left, and a logical product (AND) operation is performed between the shifted FG and the FG of the FG winning circuit 113 by setting the operation condition of the FG winning circuit 113 to "logical product" (see Figure 4(1)).

[0209] In this step, there is an FG at relative address 11 in the FG winning circuit 113 and the FG shift circuit 112, and the logical AND condition is met, so the relative address 11 in the FG winning circuit 113 remains at step 6 (a winner), i.e., it remains at logical value "1".

[0210] Steps 8 to 10 are omitted because they involve the repetition of similar calculations.

[0211] In the final step 11, the FG of the FG shift circuit 112 shown in step 10 is further shifted one byte to the left (a total of five shifts to the left), and a logical product (AND) operation is performed between the shifted FG and the FG of the FG winner circuit 113.

[0212] In this step, both FGs exist at the relative address 11 of the FG winning circuit 113 and the FG shift circuit 112, and the logical AND condition is met, so the relative address 11 of the FG winning circuit 113 remains at step 10 (remains at logical value "1") and is the winner.

[0213] The shift operation and logical product (AND) winning operation are repeated n-1 times (6-1=5 times) for the number of bytes of the string of the comparison data 123 given above, and the relative address 11 that wins to the end becomes the final winning FG.

[0214] Through the above process, it is successively detected that the character sequence 122 (character string) from relative address 11 to relative address 16 is the same as the character sequence 122 (character string) of the given search keyword 125 condition.

[0215] The above explanation is about the calculation content within the range shown in the figure, but in actual processing, character strings that match (are identical to) the character string 122 (character string) of the given search keyword 125 condition are detected consecutively within the character string 122 (character string) from address 1 to address N in the character storage element 102.

[0216] As described above, in this method, the final winning FG is determined only when all detected characters are consecutive, so in step 2, two "e" characters are detected at relative addresses 7 and 11, but the "e" at relative address 7 does not form the character sequence 122 (character string), so it is treated as noise.

[0217] The result of this calculation is output from the result output circuit 107 to the HOST in step 12, which enables the HOST to know whether the character string "search" exists within addresses 1 to N of the target character text data 132, and also the position of the first address of the character string of the search keyword 125.

[0218] As will be described later, it is particularly important that the character sequence 122 (character string) can be detected by successively detecting characters in steps equal to the number of bytes n of the character string of the given comparison data 123, and successively repeating shift operations and logical product (AND) elimination operations equal to the number of bytes n-1 of the character string of the given comparison data 123.

[0219] When detecting the character string of the given search keyword 125, it is not necessary to detect the character string of the search keyword 125 from the beginning, such as "search", but it may be detected from the end, such as "hcraes".

[0220] In that case, the shift register shift operation is shifted to the right and the last address "h" is detected.

[0221] Such a shift operation can be easily realized by using a reversible shift register (a shift register that can rotate forward and backward).

[0222] (Standard full-text search behavior for Japanese text) FIG. 7 illustrates data state transition A (character detection process) in a standard full-text search of Japanese text.

[0223] FIG. 8 explains data state transition-B (string detection process) in a standard full-text search of Japanese text.

[0224] The commands (comparison data 123, shift clock 130, and knockout calculation condition 129) shown in FIG. 4(2) are sequentially given to the full-text search circuit 103 for each step, and the full-text search circuit 103 performs full-text search processing based on these commands.

[0225] In this embodiment, data state transitions in each configuration of 15 full-text search circuits 103, which are part of the N full-text search circuits 103, are shown.

[0226] As shown in Figure 7, in this embodiment, the character code of the five Japanese characters "full-text search is" in UTF-8 character code is stored in 15 bytes in the character storage element 102 of the full-text search processor 101, from the i+1th byte (hereinafter simply referred to as the relative address 1) to the i+15th byte (hereinafter simply referred to as the relative address 15) of the relative address 104 of the full-text search circuit of the character storage element 102.

[0227] This is an embodiment in the case of performing full-text search using two characters "Search" from the above character codes as a search keyword 125 with a byte count n of 6.

[0228] Similar to the above, the processing from step 1 to step 11 is that the full-text search circuit 103 performs a full-text search operation according to a command given by the command generation circuit 127.

[0229] Steps 1 to step 6 show the process of detecting the characters "examine" and "search" from the stored character text data 132 of "Full-text search".

[0230] The part to be noted shows the characters with black and white inversion.

[0231] In step 1, the first byte "Examine: 1 / 3" of the character code of "examine" composed of 3 bytes is detected. In this embodiment, the character detection circuit 105 at relative address 7 detects "Examine: 1 / 3".

[0232] The detected FG is preset to the FG shift circuit 112 as described above. Furthermore, this FG is set to the FG breakthrough circuit 113 as the first-time breakthrough FG as described above.

[0233] Since steps 2 to step 5 are repetitions of the same operation content, they are omitted.

[0234] In the last step 6, the third byte "Search: 3 / 3" of the character code of "search" composed of 3 bytes is detected. In this embodiment, the character detection circuit 105 at relative address 12 detects "Search: 3 / 3".

[0235] The detected FG is set to the FG shift circuit 112.

[0236] In six steps equal to the byte count n of the character string of the comparison data 123 given above, the characters "Search" of the given search keyword 125 are continuously detected. It is noteworthy that in step 6, the six consecutive FGs "1111111" from relative address 7 to relative address 12 of the FG shift circuit 112 and the FG at relative address 7 of the FG winning circuit 113 set in step 1 are "1". FIG. 8 explains data state transition-B (string detection process) in a standard full-text search of Japanese text.

[0237] Step 6 is the end result of the character detection explained above.

[0238] Steps 7 to 11 show the process of detecting the string of characters 122 (character string) of "search" detected above. Step 7 shows the result of shifting the FG of the FG shift circuit 112 shown in step 6 by one byte to the left, and performing a logical product (AND) operation between the shifted FG and the FG of the FG winner circuit 113, as described above.

[0239] In this step, since both FGs at relative addresses 7 and 16 of the FG winning circuit 113 and the FG shift circuit 112 exist and the logical AND condition is met, the relative address 7 of the FG winning circuit 113 remains in step 6 (the winner).

[0240] Steps 8 to 10 are the same as steps 10, but are omitted here.

[0241] The final step 11 is to shift the FG of the FG shift circuit 112 shown in step 10 by another byte to the left (a total of 5 shifts to the left), and perform a logical AND operation between the shifted FG and the FG of the FG winner circuit 113.

[0242] In this step, as described above, both FGs at relative address 7 of the FG winning circuit 113 and the FG shift circuit 112 exist and the logical product (AND) condition is met, so relative address 7 of the FG winning circuit 113 remains the winner in step 10.

[0243] The shift operation and logical product (AND) winning operation are repeated 6 steps - 1 time = 5 times, the same number as the number of bytes n of the string of the comparison data 123 given above, and the relative address 7 that wins to the end becomes the final winning FG.

[0244] Through the above process, it is successively detected that the character sequence 122 (character string) from relative address 7 to relative address 12 is the same as the character sequence 122 (character string) of the given search keyword 125 .

[0245] The above explanation is about the calculation content within the range shown in the figure, but in actual processing, it is continuously detected whether or not there is a character string in the character sequence 122 (character string) from address 1 to address N in the character storage element 102 that matches (is identical to) the character sequence 122 (character string) of the given search keyword 125.

[0246] By outputting the calculation results of this full-text search processor 101 to the HOST, the HOST can know whether the character string "search" exists within addresses 1 to N of the target character text data 132, and at what position (address) the first address of the character string exists.

[0247] As with the standard full-text search of English text described above, it is particularly important to be able to detect a sequence of characters 122 (character string) by successively detecting characters in steps equal to the number of bytes n of the given comparison data 123, and successively repeating shift operations and logical product (AND) operations the number of bytes n-1 times of the given comparison data 123.

[0248] Although the above are only two examples of English and Japanese, it shows that full-text searches are possible across all languages ​​around the world by using standard character codes such as UTF-8.

[0249] Next, we will show an example of application using the wildcard and gap functions, which are essential for advanced fuzzy full-text search.

[0250] (Full-text search behavior when wildcards are applied) FIG. 9 explains data state transition-A (character detection process) in a full-text search that applies wildcards to English sentences.

[0251] FIG. 10 explains data state transition-B (string detection process) in a full-text search that applies wildcards to English text.

[0252] The explanation will be given assuming that the commands (comparison data 123, shift clock 130, and knockout calculation condition 129) shown in FIG. 4(3) are sequentially given to the full-text search circuit 103 for each step, and the full-text search circuit 103 performs full-text search processing based on these commands.

[0253] Wildcards are used when you are unsure of the spelling.

[0254] In this example, the wildcard " is used in the third and fourth characters of the "search" string. ? This shows an example of applying "

[0255] Steps 1 and 2 of character detection are as explained above.

[0256] The third and fourth characters are special characters, wildcards " ? is applied, the command generation circuit 127 does not provide the comparison data 123 to the full-text search circuit 103, the third and fourth characters are skipped, and the character detection process is not executed.

[0257] Therefore, step 3 detects the letter "c" and step 4 detects the letter "h".

[0258] Character detection is completed through the above four steps, and the required characters have been detected in succession.

[0259] FIG. 10 explains data state transition-B (string detection process) in a full-text search that applies wildcards to English text.

[0260] Step 4 is the final result of the character detection explained above.

[0261] The character string detection steps are steps 5 to 9. In this case, since the search keyword 125 is 6 bytes including the wildcard, the character detection process is executed in five steps (steps 5 to 9) of 6-1.

[0262] However, for steps 6 and 7 designated as wildcards, "mask (ignore)" is selected as the calculation condition generated by the winning calculation condition generation circuit 129 of the command generation circuit 127 and given to the FG winning circuit 113. This causes the calculation of the FG winning circuit 113 to be ignored, and the logic state of the FG winning circuit 113 is not changed.

[0263] That is, since step 5 is not subject to the wildcard, the calculation of the FG winning circuit 113 is the AND calculation explained above.

[0264] Since steps 6 and 7 are steps for which wildcards are to be used, the AND operation of the FG winning circuit 113 is masked (ignored), and the FG winning circuits 113 of steps 6 and 7 are left as winners, and the process moves to step 8.

[0265] Steps 8 and 9 are not steps that are subject to wildcards, so a normal AND operation is performed, and in the final step 11, "s" at relative address 11 wins.

[0266] Through the above process, it is determined that the character sequence 122 (character string) from relative address 11 to relative address 16 matches (is identical to) the character sequence 122 (character string) of the given search keyword 125.

[0267] The above explanation is about the calculation content within the range shown in the figure, but in actual processing, character strings that match (are identical to) the character string 122 (character string) of the given keyword condition are detected consecutively within the character string 122 (character string) from address 1 to address N in the character storage element 102.

[0268] In this example, a wildcard is specified in the middle of a character string, but it is possible to use a wildcard not only in the middle of a character string, but also at any position such as the beginning or end.

[0269] As described above, when a wildcard is included in the character string of the search keyword 125 provided from the outside, this method enables wildcard processing by omitting processing that is the target of the wildcard.

[0270] (Full-text search with character gaps) FIG. 11 explains data state transition-A (character detection process) in full-text search that applies character gaps to English text.

[0271] FIG. 12 explains data state transition-B (string detection process) in full-text search that applies character gaps to English text.

[0272] The explanation will be given assuming that the commands introduced in (4) of FIG. 4 are sequentially given to the full-text search circuit 103 for each step, and the full-text search circuit 103 performs full-text search processing based on these commands.

[0273] Character position tolerances (gaps) are also used when spelling is uncertain. As mentioned above, * The special character and character position of "h" 3Gap indicate the allowable position of the "h" character, in this example, "sea" is certain, and then if it is assumed that there is an "h" character in three places with gaps from 0 to 2 (Gap 3), then it is a calculation method that will match any of "seah", "sea*h", and "sea**h". A specific example is shown below.

[0274] The detection of the character "sea" from step 1 to step 3 is as explained above.

[0275] In step 4, * A gap of 3 is specified for the special character "h".

[0276] Since "h" is specified, the FG shift circuit 112 at relative address 16 is set to logic "1".

[0277] This completes the character detection, and all the necessary characters have been detected in succession.

[0278] FIG. 12 shows the data state transition-B in the character string detection process.

[0279] Step 4 is the final result of the character detection explained above.

[0280] Step 5 of detecting character strings is the same as before.

[0281] In the case of a gap specification, if the FG winning circuit 113 that is the winner in step 6 detects "h" in any of steps 7, 8, and 9, a gap calculation is performed to make the FG winning circuit 113 the winner.

[0282] The gap calculation is performed using both the main winner register and the sub-winner register of the FG winner register.

[0283] Specifically, in step 6, the winning sub-register of the relative address that has been won so far is set to logic "1".

[0284] Therefore, in this example, the winning sub-register at relative address 11 becomes logic "1".

[0285] Step 7 is to find the sequence of letters 122 (string) of "seah".

[0286] The winning subregister remains at logic "1".

[0287] The output of the FG shift circuit 112 at relative address 11 becomes logic "0", the winning sub-register becomes logic "1", and the logical AND condition of both is not established, so the winning main register cannot be restored to "1".

[0288] Step 8 is to search for the character sequence 122 (string) of "sea*h".

[0289] The winning subregister remains at logic "1".

[0290] The output of the FG shift circuit 112 at relative address 11 becomes logic "0", the winner sub-register becomes logic "1", and the logical AND condition of both is not satisfied, so as before, the winner main register cannot be restored to "1".

[0291] Step 9 searches for the character sequence 122 (string) "sea**h".

[0292] The winning sub-register at relative address 11 is set to logic "1".

[0293] In this step, the FG shift circuit 112 at relative address 11 becomes logic "1". Since both logical AND conditions are met, the winning main register is restored to logic "1".

[0294] If the logical AND operation condition between the logic "1" of the FG shift circuit 112 at relative address 11 and the logic "1" of the winning sub-register is not met in any of steps 7, 8, and 9, the winning main register at relative address 11 will not survive.

[0295] Immediately following this step, in step 10, the result of the calculation is output by the result output circuit 107.

[0296] The above calculations and output enable full-text searches for strings that include gaps between characters, such as "seah," "sea*h," and "sea**h."

[0297] Although the explanation is omitted, gaps can be set at any position by detecting character strings from the rear.

[0298] The wildcard and gap functions described above make it possible to improve the convenience of full-text search operations.

[0299] As explained above, in this method, when the character string of the externally provided search keyword 125 includes character gap tolerance, gap tolerance processing becomes possible by incorporating two sets of registers into the winning circuit.

[0300] It is especially important that wildcard and gap functions can be processed in the same number of steps as a standard full-text search, or even fewer.

[0301] In this embodiment, the special characters that specify the wildcard function or gap function are " ? "or" * " but is not limited to this.

[0302] The greatest feature of this method is the full-text search calculation with a parallel degree of N, which enables not only forward matching, middle matching, and backward matching, which are essential for various full-text search needs (including keyword search), but also advanced full-text searches that apply character wildcard functions and gap functions.

[0303] Since it is not a morphological index, it can respond in real time to new terms such as buzzwords.

[0304] Furthermore, this method allows full-text searches regardless of the character code, as long as the character code is defined in units of an integer multiple of a byte (8 bits). Therefore, this method makes it possible to standardize full-text search systems across all languages ​​in the world.

[0305] (Novelty and high speed of the algorithm of this embodiment) The novelty and high speed of the algorithm of this embodiment will be described.

[0306] Patent Document 2, "Patent No. 4588114: Memory with information narrowing down detection function, its usage method, and device including this memory," discloses a pattern matching method using a shift register.

[0307] It has been confirmed that hardware pattern matching of images using this method can be 10,000 times faster than conventional software pattern matching using a CPU.

[0308] However, the pattern matching of this prior invention is primarily intended for image pattern matching, and therefore must satisfy various matching conditions associated with image processing, resulting in a configuration requiring many steps.

[0309] Therefore, when detecting a character string of n consecutive bytes using the conventional method, n character detection operations and 1+2+3+···(n-1) shift operations for character string detection are required.

[0310] In contrast, as shown in Figures 5 to 12, our method can perform full-text search pattern matching processing in n character detection operations plus (n-1) shift operations for character string detection, for a total of n x 2-1 operations.

[0311] Therefore, compared to the number of calculations required by the conventional method, for example, in the case of two Japanese characters with 6 bytes, the conventional method requires 6 (character detection) + 15 (5 + 4 + 3 + 2 + 1) (string detection) = 21 calculations, while the present method requires 6 (character detection) + 5 (string detection) = 11 calculations, so the number of calculations is reduced to 21 / 11 = approximately 1 / 2.

[0312] In the case of four Japanese characters, 12 bytes in length, the conventional method requires 12 (character detection) + 78 (11 + 10··· + 2 + 1) (string detection) = 90 calculations, whereas the new method requires 12 (character detection) + 11 (string detection) = 23 calculations, reducing the number of calculations to 90 / 23, or approximately 1 / 4.

[0313] In the case of 8 Japanese characters and 24 bytes, the conventional method requires 24 (character detection) + 300 (23 + 22···2 + 1) (string detection) = 324 calculations, while the new method requires 24 (character detection) + 23 (string detection) = 47 calculations, reducing the number of calculations to 324 / 47 = approximately 1 / 7.

[0314] As described above, this method is particularly advantageous when the character string of the search keyword 125 is long, and the calculations including the wildcard function and the gap function are simple and efficient.

[0315] Therefore, by using this algorithm, the performance of full-text searches can be significantly improved, and a high-speed full-text search system can be realized even without an index.

[0316] (Overall structure of full-text search processor) FIG. 13 shows an example of the overall configuration of a full-text search processor.

[0317] In this example, character text data 132 indicated by input 1 from the HOST is transferred directly from the CPU of the HOST via the input / output interface 115 or by DMA (Direct Memory Access) method, and N bytes of character code are stored in the character memory element 102.

[0318] The search keyword 125 for full-text search shown as input 2 from the host is given from the host through the input / output interface 115 .

[0319] This search keyword 125 is converted into full-text search operation conditions 114 by a command generation circuit 127 .

[0320] One of the full-text search operation conditions 114 is comparison data 123 generated by a comparison data generation circuit 123 , and this comparison data 123 is given to the input of the character detection circuit 105 .

[0321] The other full-text search operation condition 114 is a shift clock 130 generated by a shift clock generation circuit 130 and a tournament operation condition 129 generated by a tournament operation condition generation circuit 129, and these two signals are given to the input of the character string detection circuit.

[0322] In this example, the 8-bit data “s” of the comparison data 123: “01110011” is compared bit by bit as described above, and the result of the comparison is shown as the calculation result of the 1-bit match detection circuit 109 .

[0323] At address 2 of the full-text search circuit address 126, the operation result of the 1-byte 1-bit match detection circuit 109 is "11111111", so the output of the 8-input AND circuit 110 is logic "1".

[0324] This calculation result indicates a state in which the shift register of the FG shift circuit 112 is set as logic "1".

[0325] Furthermore, the register of the FG winning circuit 113 at address 2 of address 126 of the full-text search circuit has survived and is at logic "1", while addresses other than 2 are at logic "0".

[0326] The logic state of the FG winning circuit 113 is output from the full-text search operation result output circuit 107 to the HOST via the input / output interface 115. The method for outputting the results is as described above. An efficient method for outputting results will be described later.

[0327] (Example of a processor configuration for narrowed search and comprehensive output) FIG. 14 explains the configuration of a full-text search processor when performing a refined search. As an example, this is a block diagram for narrowing down a search using multiple keywords such as "search," "information," and "detection." The logical sum (OR) circuit 111 performs a logical sum (OR) of the outputs of all the result output circuits 107, and sets the total result output circuit 108 if there is a winner at any one point.

[0328] The refinement search operations are as follows:

[0329] If there is a winner in the initial "search" keyword search, the overall result output circuit 108 is set.

[0330] If there is a winner in the next keyword search for "information", the total result output circuit 108 will remain set.

[0331] If there is a winner in the next keyword search for "detection", the total result output circuit 108 will remain set.

[0332] If this overall result output is sent to the HOST, the HOST will know that all three character strings, "search," "information," and "detection," exist within addresses 1 to N of the character storage element 102.

[0333] If there is no winner in the next keyword search for "information" or the search after that for "detection", the total result output circuit 108 is cleared.

[0334] By sending this overall result output to the HOST, the HOST can know that the three character strings "search," "information," and "detection" do not all exist together within addresses 1 to N of the character memory element 102.

[0335] As explained above, the method of narrowing down a search using a plurality of search keywords 125, performing a logical sum (OR) operation on the detection results of the function that outputs the detection results of the detected character sequence 122 (character string) in N parallel byte-by-byte, and outputting the presence or absence of a full-text detection result to the HOST as an overall result output, significantly reduces the result output processing on the HOST side.

[0336] In this embodiment, a configuration is shown in which the logical sum 111 of all result outputs 1 to N is taken and output to the outside, but it is also possible to divide 1 to N into appropriate sizes and output the logical sum circuit 111 and the total result output circuit 108 to the outside for each divided range.

[0337] With the above configuration, if there is a lot of character text data 132 with character strings shorter than the divided ranges, by storing the character strings in each divided range and performing a full-text search operation, it is possible to obtain full-text search results for the character strings stored in each range. Also, if the data size is limited, it becomes easy for the host to find where the character strings are.

[0338] Furthermore, in the 1-bit operation with the FG shift circuit 112, the FG winning circuit 113 is configured to be able to perform various Boolean operations such as not only the direct input, logical product (AND) operation, logical sum (OR) operation, mask (ignore) operation, and gap operation explained so far, but also negation (NOT) operation, exclusive operation, and counter operation as needed, thereby enabling more advanced full-text searches.

[0339] As an example of a negation (NOT) operation, if the search keyword 125 is the character string "Kyoto," the character string "Tokyo" will be searched as noise. This is an effective function for finding a character string that does not include "Higashi" before "Kyoto" (logical negation of "Higashi").

[0340] Similarly, the overall result output circuit 108 can be configured to perform other necessary operations such as logical product (AND) operations, logical sum (OR) operations, negation (NOT) operations, winning sub-registers, and count operations, thereby enabling more convenient full-text searches, such as outputting the detection results of multiple batch operations (operations on long sentences) using batch processing, which will be described later, together at the end.

[0341] (Features of full-text search in this embodiment) The full-text search features of this full-text search processor 101 will be described below.

[0342] First, our method performs full-text search processing equivalent to an N-gram inverted index in hardware, and allows for inverted indexes of any character length.

[0343] N-gram inverted indexes are characterized by a wide variety of full-text search functions and a low number of missed searches.

[0344] However, N-gram inverted indexes tend to have a large number of indexes and require a large memory capacity for the indexes, but this method does not require the creation of indexes, so there is no need to consider the memory capacity for the indexes.

[0345] Secondly, when the full-text search processor 101 is used, complicated algorithms such as an inverted index are not required, so the level of expertise is reduced and experts are not required.

[0346] It also eliminates language barriers between countries and enables standardization of full-text searches.

[0347] Third, the full-text search algorithm of the full-text search processor 101 enables ultra-high speed full-text searches.

[0348] Moreover, not only can it perform prefix, middle, and suffix matches, but it can also perform advanced processing such as wildcard and gap functions with a minimum number of calculations. An example of the processing time for a full-text search using the full-text search processor 101 will be described later with reference to FIGS. An example of a configuration that utilizes various features of the full-text search processor 101 is shown below.

[0349] (Example of an external memory type full-text search processor) FIG. 15 illustrates an overview of an external memory type full-text search processor.

[0350] As mentioned above, in this full-text search processor 101, N full-text search circuits 103 perform operations with a parallelism of N (fully parallel), realizing efficient and high-speed string searches, but the number of N full-text search circuits 103 cannot be increased without limit.

[0351] Therefore, large-sized character text data 132 cannot be stored in the full-text search processor 101.

[0352] The batch processing method solves this problem.

[0353] The HOST computer is shown at the top of the full-text search processor 101 .

[0354] Details are shown in Figure 21, but this HOST computer has memory or storage.

[0355] The character text data 132 stored in these memories or storages can be transferred to the full-text search processor 101 via a standard interface 116 such as PCIe or USB.

[0356] In addition, the search keywords 125 from the HOST and the results output to the HOST are also communicated through the standard interface 116 .

[0357] Below, a description will be given of batch processing when N of the full-text search circuit 103 is 32K (32×1,024) and the amount of data in one batch is 32K bytes.

[0358] Incidentally, 32K is an invention in Patent Document 3, "Patent No. 5981666," and was based on the degree of parallelism of FPGA parallel processing that has been researched up to now.

[0359] First, the case where this memory is a DRAM memory will be described.

[0360] DRAM memory is the main memory of modern computers and is used in all types of computers, from servers and PCs to smartphones. These DRAM memories are rarely used alone, but are used as memory modules (DIMMs) that comply with standards such as JEDEC (Solid State Technology Association).

[0361] The currently mainstream DIMM (Dual Inline Memory Module) is a DRAM standard called DDR4, which has a memory capacity of around 8GB and a data transfer rate of 10GB / s to 40GB / s.

[0362] If 8GB is used with UTF-8 kanji 3-byte code, 2.6 billion Japanese characters can be stored at 8 billion bytes / 3 bytes.

[0363] (Example of how data is transferred) FIG. 16 is a diagram for explaining an outline of data transfer between the external memory type full-text search processor 101 and an external memory or storage.

[0364] In this example, the concept of writing text data from a DIMM memory or storage into the character storage element 102 of the full-text search processor 101 is shown.

[0365] In this case, the host manages where the character text data 132 of which document is stored or will be stored based on the FAT (File Allocation Table), in the same way as in normal information processing.

[0366] Similarly, reading from memory refers to the FAT, reads the text data of the target sentence from the DIMM memory, and writes a predetermined amount of character text data 132 to the full-text search processor 101 .

[0367] The character text data 132 written to the DIMM memory is usually burst transferred (written) to the character storage element 102 of the full-text search processor 101 in units of 64 bits (8 bytes).

[0368] Next, the concept of transferring data from an external memory to the full-text search processor 101 will be explained.

[0369] There are three cases when transferring text data from memory.

[0370] If the target character text data 132 is larger than 32 Kbytes, the character text data 132 can be divided and sent to the full-text search processor 101 for batch processing.

[0371] When transferring each batch of data, the last few dozen bytes (maximum search string length) of the previous transfer are transferred again in the next transfer, thereby preventing any missed searches.

[0372] If the target character text data 132 is smaller than or close to 32 Kbytes, one batch may be one file of character text data 132.

[0373] If the target character text data 132 is significantly smaller than 32 Kbytes, it is sufficient to write character text data 132 of a plurality of files in one batch.

[0374] The host knows what kind of file of character text data 132 has been batch processed, and therefore can associate the calculation results of the full-text search processor 101 with the character text data 132 .

[0375] (Example of batch processing method) FIG. 17 is a timing chart for explaining batch processing in an external memory type full-text search processor.

[0376] This shows a time chart for batch processing of the external memory and the full-text search processor 101, and shows the time chart for batch processing from 1 batch to X batch, performing Y consecutive searches from 1 to Y, and outputting the Yth search result to the HOST.

[0377] The following explanation will provide an overview of full-text search processing when the general-purpose DIMM memory explained above is used and the batch count X is set to 250,000 batches. When processing 8GB in 250,000 batches, each batch is 32KB.

[0378] First, let us consider data transfer capacity.

[0379] If you transfer all 8GB of data at 32GB / sec, it will take 250ms.

[0380] Even if the data is divided into batches and transferred, the total data transfer time is 250 ms.

[0381] If we define real-time search time as less than 1 second, then the remaining maximum of 750 milliseconds can be used for search calculations.

[0382] Conversely, the relationship between the data transfer time per batch and the search operation time will be explained.

[0383] If the parallelism N of the full-text search circuit 103 shown above is 32K and the data volume of one batch is 32K bytes, the number of 8G byte batches X will be 250,000, and if a full-text search is performed within one second, the processing time for one batch will be a maximum of 4 μs. With a data transfer capacity of 32GB / sec, the data transfer time for one batch of 32KB is 1μsec.

[0384] Therefore, the remaining 3 μsec can be used for the search time of the full-text search processor 101.

[0385] As mentioned above, the full-text search processor 101 can execute one detection process in several tens of steps (clocks). Details will be explained in Figure 18, but in a typical full-text search, keywords are 3 to 4 characters and of two or three types, so 50 steps are sufficient. If the system clock 131 is set to 10 ns and 50 steps per batch is processed 250,000 (250K) times (8 GB), the calculation processing time will be 125 ms. 250 ms (data transfer time) + 125 ms (computational processing time), totaling 375 ms, is a computational performance comparable to full-text searches using indexes.

[0386] Another important feature is that it can directly utilize the currently mainstream DIMM memory.

[0387] By taking advantage of the high accuracy and speed of the N-gram method, fuzzy searches and synonym searches become possible.

[0388] The HOST process only receives the results of whether or not there are search results, which are output for each batch, so the search burden is small, and power consumption of the entire system can be reduced.

[0389] Of course, as shown in FIG. 13, it is also possible to send the search results including the search byte position to the HOST for each search process.

[0390] (External memory method application example 1 (expansion)) The above explanation is for one 8GB DDR4 DIMM memory.

[0391] If the capacity needs to be increased, the number of full-text search processors 101 can be increased easily by using the same number of DIMMs as the required number (required capacity) and parallelizing the control from the host.

[0392] In this case, the full-text search processor 101 performs search processing independently, so the full-text search time remains the same even if the memory capacity increases.

[0393] (External memory application example 2 (non-volatile memory)) Since DRAM is a volatile memory, when the power is turned off, the stored data is erased and must be re-stored.

[0394] The case where non-volatile memory (storage) is used will be described.

[0395] In recent years, the data transfer capabilities of SSDs (Solid State Drives) have improved, with some offering high-speed data transfer speeds of up to 7 GB / s.

[0396] However, compared to the DRAM method explained earlier, which has a transfer capacity of 32GB / sec, it only has a fraction of that.

[0397] In such cases, by connecting multiple SSDs in a RAID (Redundant Arrays of Inexpensive Disks)-0 configuration, it becomes possible to realize a non-volatile system with data transfer capabilities similar to those of DRAM. The memory capacity of an SSD is 1TB per module, which is more than 100 times larger than the 8GB DRAM memory mentioned above.

[0398] Therefore, if the entire 1 Tbyte memory space is used as character text data 132, the number of batch processes will be 100 times or more, significantly slowing down search time.

[0399] It is advisable to use the 1T-byte memory space not only for character text data 132 but also for various data such as audio, video, log files, location information, and sensor information.

[0400] Since DRAM stores non-volatile data, the character text data 132 must be stored somewhere.

[0401] A major feature of this method is that the character text data 132 stored in the SSD can be used for full-text search in real time, as is, immediately after power is turned on.

[0402] The above-described batch-type full-text search processor 101 with external memory or storage can be realized not only by ASIC but also by FPGA. Since FPGA allows functions to be flexibly added or deleted, it is possible to realize a full-text search processor 101 that is optimal for the system.

[0403] (On the computing power of external memory-based full-text search processors) FIG. 18 summarizes the computing capabilities of the external memory type full-text search processors explained so far.

[0404] The performance of the present invention is determined by the batch data transfer capacity, the parallelism N of the full-text search circuit 103, the system clock 131 speed of the arithmetic function, and the number of batches X.

[0405] As an example, consider two sets of full-text searches using four Japanese characters: "information processing" and "full-text search."

[0406] In the case of Japanese UTF-8 character code, each character is approximately 3 bytes long, so each set consists of 12 bytes.

[0407] Therefore, detecting one set of four characters requires 12 × 2 - 1 = 23 steps, and for two sets of four characters, it requires approximately 50 steps, including processing such as outputting the results and clearing registers.

[0408] A keyword search for English text consisting of a total of 24 characters and 24 bytes, such as "full text search process," requires 24 steps to detect the characters and 23 steps to detect the character sequence 122 (string), for a total of approximately 50 steps including processing such as outputting the results and clearing registers.

[0409] Since the search conditions for a general search are as described above, the typical performance of the full-text search processor 101 explained so far will be summarized with 50 steps (clocks) as the number of steps (clocks) for a standard full-text search.

[0410] The external memory method-A (low speed) shown in the upper part of the chart is based on the full-text search processor 101 explained in FIG. 15, and in this example, as explained above, the number of processes per batch is 32K bytes (parallelism is 32K), data transfer with the external memory is 32G bytes / second (low speed), and the calculation time is shown as the calculation time per batch when the system clock 131 is 10n seconds (low speed) and 50 steps.

[0411] Based on the above conditions, the chart shows the number of batch processes, the amount of search data, the transfer time for transferring this data, the search calculation time, and the total processing time = data transfer time + search calculation time.

[0412] This method has a slower processing time than the internal memory method described below, but its advantage is that it can be immediately used with DRAM, SSDs, and FPGAs available on the market.

[0413] The part of the chart that says "250K batch processes, total processing time 375 ms" is the computing capacity shown in Figure 17 above.

[0414] The external memory method-B (high speed) shown in the bottom part of the chart summarizes the calculation time per batch when an ASIC is developed and HBM (High Band Memory) with high data transfer capacity is used.

[0415] As explained above, this example shows the calculation time per batch when the number of processes per batch is 32K bytes (parallelism is 32K), data transfer with external memory is 320G bytes / second (high speed), and the calculation time is 50 steps with the system clock 131 being 5ns (high speed).

[0416] It is 4.3 times faster than the external memory method-A (low speed).

[0417] (Internal memory type full-text search processor) FIG. 19 illustrates an overview of an internal memory type full-text search processor. In the external memory method explained so far, the memory or storage and the full-text search processor 101 are separated, and data transfer takes a long time due to the influence of a bus bottleneck.

[0418] The figure shows a full-text search processor 101 implemented as an ASIC, with an internal memory 120 or internal storage 121 having addresses from 1 to M and a data width of N, the same as the number of character storage elements 102, incorporated into the full-text search processor 101.

[0419] In the figure, an internal memory 120 or internal storage 121 having a data width equal to the number of data items in the character storage element 102, from addresses 1 to M, is built into the full-text search processor 101, and by selecting any address, data in the row direction can be assigned to the character storage element 102 in parallel.

[0420] With the above configuration, instead of transferring character text data 132 from an external device, data transfer is performed by selecting any address between 1 and M and assigning (accessing) it to the character storage element 102, thereby enabling full-text search processing that is faster than the batch processing described so far.

[0421] The internal memory 120 may be the DRAM or SRAM explained above, and the internal storage 121 may be not only NAND or NOR type SSD memory, but also spintronics memory or resistance change memory.

[0422] Needless to say, a faster access time is more advantageous.

[0423] In the case of non-volatile flash memory, NOR type SSDs can be expected to have faster access times than NAND type SSDs.

[0424] In semiconductor manufacturing technology, we can expect to see full-text search processors 101 that make full use of the latest semiconductor technologies, such as SoC (System-on-a-Chip), SiP (System in Package), WoW (Wafer on Wafer), and 3D packaging.

[0425] It is also possible to incorporate an internal memory 120 and an internal storage 121 capable of full-text search into the FPGA.

[0426] The batch processing and full-text search calculations are the same as those of the external memory type full-text search processor 101 explained so far. The performance of this system is explained below.

[0427] (Computational power of internal memory type full-text search processor) FIG. 20 summarizes the computing capabilities of the internal memory type full-text search processors explained so far.

[0428] The internal memory method-A (low speed) shown in the upper part of the chart is based on the full-text search processor 101 explained in Figure 19, and as before, the number of processes per batch is 32K bytes (parallelism is 32K), and the calculation time per batch is shown assuming a storage-type memory, the data transfer time of the internal memory 120 is 100n seconds (low speed), and the calculation time is 50 steps with the system clock 131 being 2n seconds (low speed).

[0429] Based on the above conditions, the chart shows the number of batch processes, the amount of search data, the transfer time for transferring this data, the search calculation time, and the total processing time = data transfer time + search calculation time.

[0430] This method has a processing time 7.5 times faster than the external memory method-A (low speed).

[0431] Compared to the external memory method-B (high speed), the processing time is about 1.75 times faster. The internal memory method-B (high speed) shown in the lower part of the chart assumes that the number of processes per batch is 32K bytes (parallelism is 32K) as before, and that a high-speed internal memory 120 such as a DRAM memory is used, and the data transfer time is 10n seconds (high speed), and the calculation time is 1n seconds (high speed) with the system clock 131 being 50 steps. This shows the calculation time per batch.

[0432] This method has a processing time 3.3 times faster than the internal memory method-A (low speed).

[0433] This method has a processing time 25 times faster than the external memory method-A (low speed).

[0434] Compared to the external memory method-B (high speed), the processing time is 5.8 times faster.

[0435] The amount of search data for both internal memory method-A (low speed) and internal memory method-B (high speed) will depend on the capacity of the memory integrated inside the full-text search processor 101, so memory implementation technology will be a future research topic.

[0436] If you expect a search time of less than one second, the external memory method-A (low speed) allows full-text search of 500K batches and 16GB of data. In the case of external memory method-B (high speed), full-text search of 2M batches and 64GB of data is possible.

[0437] In the case of internal memory method-A (low speed), full-text search of 4M batches and 128GB of data is possible.

[0438] In the case of the internal memory method-B (high speed), full-text search of 16M batches and 512GB of data is possible.

[0439] The performance shown in the chart is the performance of one full-text search processor 101, so by connecting multiple full-text search processors 101 in parallel, it becomes possible to increase the amount of data searched in the same time.

[0440] Furthermore, the specifications shown in the chart are based on estimated values ​​of the current semiconductor technology level, so that the full-text search processor 101 can be expected to become even more powerful as semiconductor technology improves in the future.

[0441] When developing an internal memory type full-text search processor 101, it is advisable to classify it into several types such as high speed / low capacity, medium speed / medium capacity, and low speed / high capacity, and select the optimal type taking into consideration heat generation, chip size, and economy.

[0442] Once the chips are developed, users can choose the most suitable chip based on the performance and functionality they require.

[0443] One of the advantages of this method is that it allows you to predict the full-text search time in advance, as described above.

[0444] As mentioned above, the memory used in the full-text search processor 101 is not limited to the DRAM, NAND type, or NOR type storage described above, and it is also possible to use new memory that is expected to be used in the future.

[0445] As mentioned above, it can be incorporated not only into ASIC but also into FPGA.

[0446] The computational capabilities shown in Figures 18 and 20 above do not guarantee feasibility. Also, since this is a theoretical value, a certain amount of overhead must be taken into account.

[0447] (Example of system configuration when using a full-text search processor) FIG. 21 illustrates an example of a system configuration when a full-text search processor is used.

[0448] There are various ways to use the external memory type full-text search processor 101 shown in FIG. 15 and the internal memory type full-text search processor 101 shown in FIG. 19, but two representative examples will be introduced below.

[0449] System configuration example-A is a system configuration example of the external memory type full-text search processor 101 shown in FIG. 15, and is an example in which the full-text search processor 101 is connected to the outside of the system board 124 and used.

[0450] This is an example of a case where data in a DRAM memory or storage is connected from a system board 124 to a standard interface 116 such as PCIe or USB for use.

[0451] The system board 124 is equipped with a DRAM memory and is connected to an external storage.

[0452] In this case, the full-text search processor 101 receives, via the standard interface 116, character text data 132 transferred from the DRAM memory or storage, search keywords 125 from the host, and a calculation result output signal to the host.

[0453] The current maximum transmission bandwidth for USB is 5Gbps (USB3.0).

[0454] On the other hand, PCIe has a wide range of broadband transmission capabilities, from tens of gigabytes per second to hundreds of gigabytes per second, so it is a good idea to use the PCIe standard that matches the performance of your system.

[0455] System configuration example B is a system configuration example of the internal memory type full-text search processor 101 shown in FIG. 19, in which the full-text search processor 101 with built-in memory and storage is embedded in the system board 124 and used.

[0456] In this example, a full-text search is performed using the full-text search processor interface 119 inside the system board 124.

[0457] The explanation so far has been based on the assumption that the HOST sends the search keyword 125 to the full-text search processor 101, the command generation circuit 127 sends command (control) signals for each step shown in Figures 5 to 12 to the full-text search circuit 103, the full-text search circuit 103 performs a full-text search operation, and the HOST receives the output of the operation results.

[0458] The HOST receives the search results for each batch sent from the full-text search processor 101, and if it knows which batches have a result output of "yes" from the overall result output circuit 108, it can check which part of the batch data contains the string to be searched.

[0459] When constructing a system, it goes without saying that an appropriate host and application software must be prepared depending on the required computing performance and functionality as well as the number of parallel full-text search processors 101 .

[0460] The following describes a system application embodiment of the full-text search processor 101.

[0461] (WEB Search System-1) Full-text searches on web search sites are extremely difficult.

[0462] In the case of a web search site, the data volume of the target character text data 132 is enormous, and furthermore, an extremely large number of people use the site regardless of the time.

[0463] If 50 million Japanese people search Japanese search sites an average of 10 times a day, that would mean 500M searches (50M people x 10 searches) / 86,400 seconds = approximately 5,787 searches per second.

[0464] Working backwards, the processing time per cycle is 1 / 5,787 = 173 μsec.

[0465] Therefore, it is necessary to complete the search process in at least half of this time.

[0466] In the above case, this corresponds to 60 μs for 1K batch (32 MB) in the internal memory method B shown above. While it is possible to use this type of processor, it is also possible to write the same data to 1,000 full-text search processors 101 with a 60 ms processing time for 1M batch (32 GB) and perform distributed processing.

[0467] When mounting semiconductor chips on a printed circuit board, it is more advantageous to use a larger chip than to use many small chips. Taking into consideration the multiple accesses described above, it is advisable to use a full-text search processor 101 with appropriate performance.

[0468] Currently, most web search sites use HDD-type storage systems in order to reduce system costs, but if the cost of SSD-type storage decreases in the future, it is expected that these will gradually be replaced by SSD-type storage.

[0469] In such cases, using this full-text search engine will free you from the various constraints of indexes, which is expected to reduce system operating costs and create a web search system with high real-time capabilities.

[0470] (WEB Search System-2) Replacing the web search sites described above with this system would likely require a considerable amount of time, including economic considerations such as implementation and operating costs.

[0471] We will introduce a method that is relatively easy to implement and makes good use of the features of this technology.

[0472] According to information from a major web search site, the number of different Japanese N-gram methods (maximum number of indexes) is published on the NET website as follows: https: / / japan.googleblog.com / 2007 / 11 / n-gram.html Total words: 255,198,240,937 (255 billion) Total sentences: 20,036,793,177 (20 billion) Number of different 1-grams: 2,565,424 Number of different 2-grams: 80,513,289 Number of different 3-grams: 394,482,216 Number of different 4-grams: 707,787,333 Number of different 5-grams: 776,378,943 Number of different hexagrams: 688,782,933 Number of different 7-grams: 570,204,252 The difference corresponds to the maximum number of indexes.

[0473] A 1-gram number indicates a single character, and includes not only commonly used kanji and English characters, but also characters and symbols used all over the world, as well as environmental characters, meaning that there are a total of 2.56 million different characters.

[0474] Even if an index has a low frequency of occurrence, ignoring it will result in missed searches.

[0475] Since it is necessary to have an index even for special characters and symbols that are used only once in a total of 20 billion sentences, creating an index using the N-gram method would require a huge total of 3.2 billion indexes.

[0476] It is important to consider that even though most of the 3.2 billion indexes mentioned above are rarely used, ignoring them will result in missed searches.

[0477] Therefore, it is necessary to use a complex index structure, such as a morphological index, in combination with other indexes.

[0478] Here's how to solve this perennial indexing conundrum.

[0479] For example, if the word contains even one of the characters "bara," "udon," "don," "kan," or "kankangakugaku," or a special symbol, which are rarely used, the full-text search processor 101 is used, and if it does not contain any of the characters, the full-text search is performed using the index method as before, thereby making the system significantly more efficient. Since the frequency of full-text searches that include rarely used characters is extremely low, the full-text search processor 101 should be selected to have a search speed that is suitable for that search frequency.

[0480] If indexes containing rarely used characters are not needed, the number of indexes can be reduced dramatically.

[0481] Furthermore, since the full-text search processor 101 only needs to store text data of a website that contains characters that are rarely used, the number of full-text search processors 101 can also be reduced.

[0482] For example, various applications are possible, such as determining the top 100,000 most frequently used indexes, performing full-text searches using these indexes, and leaving searches related to less frequently used indexes to the full-text search processor 101.

[0483] If the number of indexes, which has currently required up to 3.2 billion or more, can be reduced to 100,000 and additional indexes will no longer be necessary, the complexity of web search systems will change dramatically.

[0484] As described above, by combining the advantages of the index method with the advantages of this full-text search method, it is possible to reduce the number of indexes to the minimum without sacrificing search performance, thereby streamlining the web search system.

[0485] The problem of the number of indexes explained above is not limited to web search systems but is a common issue for full-text searches.

[0486] In the previous example, a system configuration was shown in which the full-text search processor 101 is used when the text contains at least one special symbol or rarely used character, but it is also possible to use the full-text search processor 101 in the opposite case, where a full-text search is not performed, or where the full-text search is performed for specific characters other than special symbols and rarely used characters.

[0487] It is a good idea to analyze the system's features and consider how to best use it.

[0488] (Local web search system) When searching for characters contained in papers, literature, etc., the search methods used on regular web search sites may not be sufficient.

[0489] In such cases, if necessary information is collected from the Web and the full-text search processor 101 of the present invention is used, full-text searches can be performed using advanced methods such as grep (global regular expression print).

[0490] (Detection of character and terminology errors, mistranslation detection) A corpus is a database used for natural language research.

[0491] Because a corpus is a database that uses indexes, it is regularly revised, just like a dictionary, and so often does not include the latest words such as "corona disaster."

[0492] By using the present invention, the corpus can always be made into text data containing the latest terms and information.

[0493] Another way to use it is to compare part of the text being created with a large amount of text data containing the latest information stored in internal or external memory, and if there is no matching text, it will be determined to be unprecedented (a mistake) and an alert will be issued.

[0494] (Large-scale infrastructure system, large-scale in-house search system) It goes without saying that the present invention is optimal for large-scale infrastructure systems such as patent searches and in-house search systems of large corporations, as well as full-text search systems for large organizations.

[0495] It is a good idea to build your system by referring to the contents of the web search site explained above.

[0496] (Unexplored areas of implementation) The present invention is expected to shed light on systems that have problems with full-text searches using indexes, and on fields where such searches have not yet been systematized. It is expected that this technology will be applied to full-text search of stream-type text data, which will find value in new information.

[0497] The frequency of characters used in text data flowing as stream data is essential for AI analysis.

[0498] It would be a good idea to focus on real-time processing such as natural language processing and statistical systems, which are difficult to handle with current methods.

[0499] For example, by comparing speech recognition candidates or translation candidates in translation with the full-text search processor 101, if there is no matching text, it can be determined to be unprecedented (a mistake) and an alert can be output, or the text with the largest matching text can be selected, thereby improving the accuracy of speech recognition and other intelligent information processing (AI field).

[0500] (Personal use) Many people who use PCs frequently use full-text searches on a daily basis, such as searching for emails and searching text data such as Word, Excel, and PowerPoint.

[0501] However, these full-text searches are performed using software that comes standard with PCs, and it is not possible to set advanced search conditions.

[0502] The full-text search according to the present invention does not require any expertise, and it is expected that the standardization of software with various functions will progress.

[0503] By using this software, even those who are not full-text search experts or software experts can use full-text search in their own way.

[0504] (Genome analysis) Although the explanation so far has been about searching character text data 132, the full-text search processor 101 can also be used for analyzing genomes.

[0505] The human genome contains approximately 6 billion base pairs of DNA in the nucleus.

[0506] Therefore, with a capacity of 8GB, it would be possible to store the entire human genome and analyze it all at once.

[0507] This method is ideal for analyzing the genome's "ATGC" base sequence, as wildcard and gap functions are essential.

[0508] Most current genome analysis uses indexes to speed up the process, but the way the indexes are created can lead to missed searches and inconsistent search results, which is a problem.

[0509] The time required to create the index is also a waiting time.

[0510] Use of this full-text search processor 101 enables high-speed genome analysis without indexing, ie, without waiting for index creation, without missed searches or variations in search results.

[0511] (Standardization of full-text search) The full-text search according to the present invention can be used for characters all over the world by using a global standard character code such as UTF-8.

[0512] By introducing rules for character pattern matching, such as regularized expressions like grep in SQL and NonSQL, it will be possible to standardize full-text search technology worldwide.

[0513] Standardization will enable the supply of ASIC chips for the full-text search processor 101 at low cost, and large demand is expected.

[0514] (Advantages of the present invention) The advantages of the full-text search processor 101 of the present invention and the systems and products that use it are listed below. No inverted index required Real-time processing and stream processing become possible No need to tune indexes No index memory or storage required No need to update indexes - System development becomes easier -Easy system expansion Accuracy / speed function ·Similar search accuracy and speed to N-gram Wildcards, gaps, in-between matches, and suffix matches are possible. (Character searches like SQL Like statements and grep (regularized expressions) are possible.) Versatility -Compatible with character codes such as ASCII, JIS, and UTF -Can also be used for genome analysis ·UTF-8 character encoding allows for global standardization of full-text search ·Available for SQL and Non-SQL databases Practicality -Practical use with FPGA · It can be incorporated into a full-text search system that uses indexes ·You can create a corpus containing the latest terms Contribution to information processing - Full-text search technology becomes less specialized, eliminating the need for experts Accelerate the evolution of natural language processing technologies such as speech recognition and translation · Global standardization of full-text search technology Many of the above are actual or potential needs for full-text search and natural language processing technology.

[0515] (Point of focus of the present invention) Finally, the focus of the present invention will be described.

[0516] Ideally, the full-text search circuit 103 would be used for each byte of the character text data 132, but this method would result in extremely poor cost performance for the full-text search semiconductor chip and the full-text search system.

[0517] The present invention aims to balance the performance of index-free full-text search with system costs by providing a large amount of character text data 132 to the full-text search circuit 103 in a time-division manner.

[0518] In this example, all circuits (functions) of the command generation circuit 127 are described as being provided inside the full-text search processor 101, but it is also possible to provide some or all of the circuits (functions) of the command generation circuit 127 on the HOST side, and have the full-text search processor 101 perform calculations based on calculation conditions for each step given from the HOST side, and notify the HOST side of an acknowledgement (ACK) for each step.

[0519] Furthermore, when the command generation circuit 127 is placed inside the full-text search processor 101, it is also possible to incorporate the CPU and memory for storing the program inside the full-text search processor 101, and generate the full-text search operation conditions 114 by software processing and control the full-text search circuit 103.

[0520] (Basic circuit configuration (second example)) Next, a second example of the full-text search processor according to the present invention will be described.

[0521] In this second example, similar to the first example above, the character text data to be searched is stored in a memory element one byte (8 bits) at a time, and the "characters" and "character sequences" of the character string of the given search keyword are compared in parallel one byte at a time to see if they match, and the position (address) of the character text data that corresponds to the beginning or end of the character string of the given search keyword 125 is returned as the full-text search result.

[0522] However, in the first example, the character detection circuit 105 completes character detection, and then the character string detection circuit 106 performs a shift operation. In contrast, in this second example, the character detection circuit 105 and the character string detection circuit 106 perform a shift operation alternately.

[0523] That is, in the first example, the character detection circuit 105 first detects, from the character text data 132, a character or a portion of a character (1-byte code) that matches all of the characters included in the search keyword 125 as a match flag (FG), and then the string detection circuit 106 shifts the position of this flag in order by n-1 times (corresponding to the number of shift clocks), which is the number of bytes in the string of the search keyword 125, to detect the positions of consecutive flags for the number of bytes in the search keyword 125, i.e., the character sequence 122 (string).

[0524] In contrast to this, in this second example, each time the character detection circuit 105 detects a character or a portion of a character (1-byte code) that matches a character in the search keyword 125 (match flag (FG)), the position of this flag is shifted by one, and the FG win circuit 113 detects, for each character code, whether the flag in question is consecutive to the position of the flag detected immediately before. Even with this method, it is possible to detect a character sequence 122 (character string). Hereinafter, processing using this second method will be referred to as "alternate processing."

[0525] Figure 22 shows the basic configuration of a full-text search processor for executing this second example. In this second example, a step condition generation circuit 133 for generating a step condition 133 (a signal (described later) that is logic "0" in step 1 and logic "1" in steps other than step 1) for realizing the above-mentioned alternating processing is added to the second example command generation circuit 127 (Figure 1).

[0526] 23 corresponds to FIG. 3 of the first example, and the character string detection circuit 106 in FIG. 3 is provided with a step condition-specific AND circuit 134 connected to the FG shift circuit 112.

[0527] The step condition generating circuit 133 and the step condition logical product circuit 134 enable alternate processing.

[0528] Although the step condition-specific AND circuit 134 is not limited to this configuration, it is composed of a two-input AND circuit, a three-input AND circuit, and a logical negation (NOT) circuit, as shown in Figure 23.

[0529] The alternating processing using these circuits will now be described in detail.

[0530] (Second Example Alternating Processing Operation: Command Generation Operation) FIG. 24 corresponds to FIG. 4 and shows specific examples (1) to (4) of full-text search calculation condition commands generated by the command generating circuit 127 in this second example.

[0531] In this second example, unlike the first example (FIG. 4), the command generation circuit 127 constructs commands for steps 1 to 11 so that character and character string detection calculations are performed alternately. To enable this alternating processing, the second example differs from the first example in the following points: 1) a fourth command 127, step condition 133, given by the step condition generation circuit 133, is added (in this example, a signal that is logic "0" in step 1 and logic "1" in other steps), 2) the flag is shifted right instead of left, and 3) all of the given winning calculation conditions are "direct input".

[0532] (Regarding the full-text search behavior for English text in the second example) Next, the data state transition of the full-text search in this second example will be described with reference to FIGS.

[0533] The search target is the same as in the first example (FIGS. 5 and 6), and the search character string "search" is searched for in the character text data 132 of "full text search." The number of processing steps is also 11, the same as in the first example.

[0534] The processing of each step (1 to 11) is executed by the full-text search circuit 103 in accordance with commands (comparison data 123, shift clock 130, step condition 133, knockout operation condition 129) given by the command generation circuit 127.

[0535] First, in step 1, the character "s" consisting of one byte is detected, and as in step 1 of the first example, the character detection circuit 105 at relative address 11 detects "s." This detection result is preset as a flag (FG) in the FG shift circuit 112. In the example of Figure 25, since "s" is only found at address 11 in the "full text search," only the FG shift circuit 112 at relative address 11 is set to logic "1," and the logic of the FG shift circuits 112 at the other addresses becomes "0."

[0536] This FG value (calculation result) is then processed by the step condition-specific AND circuit 134 and input to the FG winning circuit 113. In the case of step 1, the fourth command 127 and step condition 133 (a signal that is logical "0" in this example for step 1 and logical "1" for other steps) given by the step condition generation circuit 133 are "0", so the two-input AND circuit of the step condition-specific AND circuit 134 becomes valid, and the calculation result is input to the FG winning circuit 113 via the OR circuit.

[0537] As shown in Figure 24(1), since the calculation condition for the FG winning circuit 113 in step 1 is specified as "direct input", the FG winning circuit 113 (register) at relative address 11 is set to logic "1", and other addresses are set to logic "0" (see step 1 in Figure 25).

[0538] That is, in step 1 (first step), the calculation result of the character detection circuit 105 is set to the FG winning circuit 113 as is.

[0539] In step 2, all FGs set in the FG shift circuits 112 are shifted by one address to the right by the shift clock 130 signal, and the FGs are set in the FG shift circuits to the right (step 2 in FIG. 25). At this time, the logic state of the FG winner circuit 113 does not change.

[0540] As a result, character detection is performed in step 1, and flag shifting for character string detection is performed in step 2. From then on, character detection and shifting are performed alternately in this manner.

[0541] Next, in step 3, the second character, "e," is detected. In this example, as shown in Fig. 25, "e" is present at relative addresses 7 and 12, so these are detected.

[0542] The detection results at addresses 7 and 12 are input to the FG shift circuit 112. At this time, the flag (logic "1") shifted from address 11 in step 2 above has already been set in the FG shift circuit 112 at address 12, so the following calculation is executed.

[0543] That is, at address 12, both the logical value input from the character detection circuit 105 and the logical value preset in the FG shift circuit 112 are "1", and the fourth command 127 (a signal that is logical "0" in step 1 and logical "1" in other steps) given by the step condition generation circuit 133 is "1", so the three-input logical product (AND) circuit of the step condition-specific logical product circuit 134 becomes valid, and the operation result (logical "1") is input to the FG winning circuit 113 via the logical sum (OR) circuit. Then, as shown in Fig. 24 (1), a "direct input" command is given to the FG winning circuit 113, so in this example the logical value of relative address 12 set in the FG winning circuit 113 is "1" (see the value of address 12 in step 3 of Fig. 25).

[0544] On the other hand, for address 7, the logical value preset in the FG shift circuit 112 is "0", so as a result of the same calculation as above, the logical value set in the FG win circuit 113 becomes "0" and is treated as noise.

[0545] Furthermore, for addresses other than the above addresses 11 and 7, the logical value input from the character detection circuit 105 and the logical value preset in the FG shift circuit 112 are both "0", so as a result of the same calculation as above, the logical value set in the FG win circuit 113 becomes "0".

[0546] That is, the FG win circuit 113 at relative address 11, which had been set to logic "1" up to step 2, is cleared to "0" because the logical product (AND) condition of both the character detection circuit 105 and the FG shift circuit 112 is not satisfied.

[0547] In other words, in the steps after step 1 (initial step), in this example step 3, the result of the calculation by the character detection circuit 105 and the logical product (AND) condition of the FG shift circuit 112 are satisfied, and the winning FG is updated in the FG winning circuit 113 of the address, which is a major difference from the conventional method.

[0548] As for steps 4 to 9, the same calculations as above are repeated, so the explanation will be omitted.

[0549] Step 10 shows the state in which the FG of the FG shift circuit 112 shown in step 9 has been shifted another byte to the right (a total of 5 shifts to the right). In this step 10, as shown in Figure 26, the logical values ​​of addresses 11 and 16 of the FG shift circuit 112 are set to "1", and the relative address 15 of the FG winning circuit 113 becomes the winning address at this point.

[0550] In step 11, since both FGs exist in the character detection circuit 105 and the FG shift circuit 112 at relative address 16, the logical product (AND) condition is met, so the relative address 16 of the FG winner circuit 113 becomes "1" and the relative address 15 is rewritten to "0".

[0551] Therefore, relative address 16 will be the final remaining address.

[0552] Through the above process, it is detected that the character sequence 122 (character string) from relative address 11 to relative address 16 is the same as the character sequence 122 (character string) of the given search keyword 125 condition, and an address that matches the character code at the end of the given search keyword 125 has been detected.

[0553] Unlike the first example, this second example can determine the validity of the array for each detected character code, for each character code. On the other hand, the number of calculation steps to be performed is n x 2-1 times the number of bytes in the character string of the given comparison data 123, which is 11 steps in this example, the same as the first example. In other words, compared to the first example, more accurate calculations can be performed with the same number of steps. In other words, this second example is a method that updates the surviving FG every time a new character is detected, so it can guarantee the validity of the array for all searched character strings, and is characterized by high search accuracy and fast search speed.

[0554] In step 12, the above calculation results are output from the result output circuit 107, but unlike the first example, what is output is not the first address but the last address.

[0555] However, when detecting the character string of the given search keyword 125, it is not necessary to detect the character string of the search keyword 125 from the beginning, such as "search," and it is also possible to detect it from the end, such as "hcraes." In that case, the shift operation of the shift register is shifted left to detect the starting address "s."

[0556] (Standard full-text search behavior for Japanese text in the second example) 27 and 28 illustrate the data state transitions of each function of a standard full-text search for Japanese text in the second example, which corresponds to FIGS. 7 and 8 of the first example.

[0557] In this process, the command 127 (comparison data 123, step condition 133, shift clock 130, and knockout operation condition 129) shown in FIG. 24(2) is sequentially given to the full-text search circuit 103 for each step, and the full-text search circuit 103 performs full-text search processing based on this command.

[0558] Detailed explanation of the processing based on the above command will be omitted, but in the case of Japanese text, one character is multiple bytes (3 bytes in this example), but the operation is the same as for English text.

[0559] Steps 1 to 5 are used to detect the character codes "Search:1 / 3", "Search:2 / 3", and "Search:3 / 3" from the stored character text data 132 of "Full-text search is", and step 6 shows a shift operation as a preprocessing step to detect the character code of "Search:1 / 3".

[0560] In the final step 11, by giving the comparison condition "search:3 / 3", as mentioned above, FG exists in both the character detection circuit 105 and the relative address 12 of the FG shift circuit 112, and the logical product (AND) condition is met, so the relative address 11 of the FG winning circuit 113 becomes "1", and the relative address 11 becomes the final winning address.

[0561] Through the above process, it is detected that the character sequence 122 (character string) from relative address 7 to relative address 12 matches the character sequence 122 (character string) of the given search keyword 125, and the address 126 of the full-text search circuit that matches the character code at the end of the given search keyword 125 is output as the search result.

[0562] (Full-text search behavior when wildcards are applied in the second example) 29 and 30 illustrate data state transitions in a full-text search that applies wildcards to English sentences according to a second example corresponding to FIGS. 9 and 10 of the first example.

[0563] In this process, the command 127 (comparison data 123, step condition 133, shift clock 130, and knockout operation condition 129) shown in FIG. 24(3) is sequentially given to the full-text search circuit 103 for each step, and the full-text search circuit 103 performs full-text search processing based on this command.

[0564] Detailed explanation of the processing based on the above command will be omitted, but in this example, the wildcard " ? ", but the operation is basically the same as the previous two examples.

[0565] That is, steps 1 to 4 of character detection are the same as those explained above.

[0566] If steps 5 and 7 are wildcards, the FG winning circuit 113 is given a Mask (ignore) calculation condition as shown in Figure 24, and the FG winning circuit 113 performs the same processing as if the character detection circuit 105 had matched, regardless of the character, and thereby performs the specified winning calculation.

[0567] The subsequent processing is the same as that explained above, so the explanation will be omitted, but in this example too, an address corresponding to the character code at the end of the given search keyword 125 is detected.

[0568] In this example, a wildcard is specified in the middle of a character string, but it is possible to use a wildcard not only in the middle of a character string, but also at any position such as the beginning or end.

[0569] As described above, in this method, when a wildcard is included in the string of search keywords 125 provided from the outside, the processing that is the target of the wildcard is configured so that the knockout condition is met regardless of the characters, thereby making wildcard processing possible.

[0570] (Full-text search with character gaps) 31 and 32 illustrate data state transitions of a full-text search that adapts gaps in English text according to a second example, which corresponds to FIGS. 11 and 12 of the first example.

[0571] In this process, the command 127 (comparison data 123, step condition 133, shift clock 130, and knockout operation condition 129) shown in FIG. 24(4) is sequentially given to the full-text search circuit 103 for each step, and the full-text search circuit 103 performs full-text search processing based on this command.

[0572] We will not go into detailed explanation of the processing based on the above command, but in this example, if "sea" is confirmed, and then it is assumed that the letter "h" is present in three places (Gap3) with Gap from 0 to 2, that is, if any of Gap0: "seah", Gap1: "sea*h", and Gap2: "sea**h" is considered a match, the rest of the calculations are the same as the previous three examples.

[0573] That is, the detection of the character "sea" from step 1 to step 6 is as explained above.

[0574] Steps 7, 9, and 11 are * Gap 0, 1, 2, and Gap 3 are specified as special characters for "h." Gap processing determines that if the character "h" is found in any of steps 7, 9, and 11, the string is valid and a match occurs. In order to perform such calculations, in this second example, a sub-register of the FG winning circuit 113 is used.

[0575] That is, when "h" is specified in step 7 as shown in FIG. 24, the FG winning circuit 113 is given a Gap operation condition, and just like a wildcard, it processes so that the winning operation is successful no matter what character is at relative address 14 in this example.

[0576] Furthermore, the subregister is operated, and the subregister at relative address 16, which is two addresses ahead of the remaining relative address 14, remembers that relative address 14 was not "h." Therefore, the subregister at relative address 16 is set to "0."

[0577] The same applies if "h" is specified in step 9. In this step, relative address 15 is not "h", so "0" is written to the subregister at relative address 16, which is one address after the winning relative address 15, and it remains at logic "0".

[0578] If "h" is specified in step 11, the logic of the character detection circuit 105 at relative address 16 becomes "1", and the main register at relative address 16 of the FG winning circuit 113 becomes "1".

[0579] Furthermore, a "1" is written to the sub-register at relative address 16, making the logic "1", and the main register is also "1".

[0580] Since the logic of both the main and sub registers is "1", it is determined that at least one of the three gap designations contains the letter "h", and relative address 16 becomes the final remaining address.

[0581] As in the previous explanations, in this example too, the address 126 of the full-text search circuit that corresponds to the character code at the end of the given search keyword 125 is detected.

[0582] The above explanation was based on the condition being met in the final step of Gap 2, but it goes without saying that if the sub-register is set to "1" in Gap 0 (i.e., step 7) or Gap 1 (i.e., step 9), it is also possible to perform calculations to ensure that the winner remains.

[0583] The above calculations and output enable full-text searches for strings that contain gaps between characters, such as "seah," "sea*h," and "sea**h."

[0584] Although the explanation is omitted, it is possible to set a gap at any position by detecting the character string from the rear.

[0585] The wildcard and Gap functions described above make it possible to improve the convenience of full-text search operations.

[0586] It is particularly important that advanced full-text searches, such as wildcard and gap functions, can be performed in the same number of steps as standard full-text searches.

[0587] The present invention is not limited to the above two embodiments, and various modifications are possible within the scope of the present invention. [Explanation of symbols]

[0588] 101...Full-text search processor 102...Character memory element 103...Full-text search circuit 104...Relative address of full-text search circuit 105...Character detection circuit 106...String detection circuit 107...Result output circuit 108...Comprehensive result output circuit 109...1-bit match detection circuit 110...Logical product (AND) circuit 111...Logical sum (OR) circuit 112...FG (flag) shift circuit 113...FG (Flag) Winning Circuit 114...Full-text search operation conditions 115...Input / output interface 116...Standard interface 117...DRAM interface 118...Storage interface 119...Interface for full-text search processor 120...Internal memory 121...Internal storage (including SSD) 122...Character sequence 123...Comparison data (generation circuit) 124...System board 125...Search Keywords 126...Full-text search circuit address 127...Command generation circuit 128...Keyword setting function 129...Winning operation conditions (generation circuit) 130...Shift clock (generation circuit) 131...System clock (generation circuit) 132...Character text data 133...Step condition (generation circuit) 134...Step conditional AND circuit

Claims

1. a search processor comprising a semiconductor device; a character memory element that temporarily stores the encoded character string to be searched for by allocating it to addresses 1 to N in units of a predetermined number of bits; a character detection circuit that sequentially receives one or more coded characters included in a search keyword as comparison data in units of the predetermined number of bits, compares each of the coded characters with the coded character string to be searched stored in the character storage element in N parallel sequences, and repeats this process for all coded characters included in the search keyword, thereby detecting the storage positions of all coded characters included in the search keyword in the character storage element; a character string detection circuit for detecting positions on the character storage element where all coded characters included in the search keyword are consecutive in the order of the sequence included in the search keyword; a result output circuit that receives the detection result of the character string detection circuit and outputs the start position or end position of the continuous character string; The character string detection circuit It is composed of two circuits, an FG shift circuit and an FG winning circuit, for determining the validity of the FG (flag) of the character storage position detected by the character detection circuit and the arrangement of the FG (flag) of the detected character storage position. The FG shift circuit and the FG winning circuit refer to the sequence of coded characters included in the search keyword, and repeatedly determine the validity of the sequence of adjacent coded characters in the coded character string to be searched for in the character storage elements allocated and stored at addresses 1 to N, thereby detecting in N parallel the start position (address) or end position (address) in the character storage elements of the coded character string to be searched for that matches the sequence of coded characters of the search keyword.

1. A search processor comprising:

2. 2. The search processor of claim 1, the predetermined bit unit for temporarily storing the encoded character string in the character storage element is 1 byte / 8 bits; The character detection circuit a set of N pairs of eight 1-bit match or mismatch calculation circuits connected to each address of the character storage element, for comparing a 1-byte / 8-bit code of the encoded character string to be searched stored in each address with a 1-byte / 8-bit code constituting the comparison data, and detecting a match or mismatch for each bit; N logical operation (logical product (AND), logical sum (OR), logical negation (NOT)) circuits that receive 8-bit result outputs from each set of 1-bit match or mismatch operation circuits and detect whether the sequence of the encoded character string to be searched stored in each address matches or mismatches with the sequence of the encoded character that constitutes the comparison data; It has 1. A search processor comprising:

3. 2. The search processor of claim 1, The character string detection circuit a number N of FG shift circuits for storing the calculation results of the character detection circuit as flags and for shifting the stored FGs in N parallel ways; N FG winning circuits that perform N parallel logical operations with the FG before shifting while shifting the stored FG in N parallel ways by the FG shift circuit, and repeat this for all coded characters included in the search keyword, thereby performing a winning operation for the FG; 1. A search processor comprising:

4. 2. The search processor of claim 1, When the search keyword is composed of n bytes of coded characters, The number of shifts performed by the FG shift circuit and the number of corresponding win-win calculations performed by the FG win-win calculation circuit are n×2−1 times.

1. A search processor comprising:

5. 2. The search processor of claim 1, The FG winning circuit has a function that allows masking (ignoring) of operations, and enables searches when wildcards are used in search keywords.

1. A search processor comprising:

6. 2. The search processor of claim 1, The FG winning circuit incorporates two sets of registers, enabling searches that include character gaps in the search keyword.

1. A search processor comprising:

7. 2. The search processor of claim 1, A logical sum (OR) operation is performed on the N parallel (all parallel) detection results output by the result output circuit, and the presence or absence of a detection result is output.

1. A search processor comprising:

8. 2. The search processor of claim 1, A search processor characterized by transferring text data from a memory or storage external to the search processor as batch data to a character memory element that temporarily stores an N-byte encoded string of characters to be searched, and repeating the search operation in N parallel (fully parallel) fashion.

9. 2. The search processor of claim 1, A search processor characterized by transferring text data from the internal memory or storage of the search processor as batch data to a character memory element that temporarily stores N bytes of encoded character strings to be searched, and repeating the search operation in N parallel (fully parallel) fashion.

10. 2. The search processor of claim 1, It is characterized by being implemented in ASIC and FPGA. Search processor.

11. 2. The search processor of claim 1, A search processor characterized by having a built-in CPU.

Images