Full-text retrieval method and device based on hash slot storage structure, equipment and medium

Abstract

The invention provides a full-text retrieval method and device based on a hash slot storage structure, equipment and a medium, and relates to the technical field of full-text retrieval. The method comprises the following steps: receiving a search word, and performing N-Gram word segmentation on the search word to obtain a series of entries; for each vocabulary entry, performing hash modulo operation on the vocabulary entry to obtain a hash value corresponding to the vocabulary entry; matching the Hash value with a pre-established slot block chain quick access array, determining a slot block chain corresponding to the Hash value, mapping each piece of slot data on the slot block chain to a memory one by one from a pre-established storage file system by using read mapping, restoring a document ID corresponding to each piece of slot data through an incremental coding (FOR) restoration algorithm, and storing the document ID in the memory. Obtaining an inverted list corresponding to the vocabulary entry; the storage file system is established based on a hash slot file storage structure; and outputting a retrieval result according to the inverted lists corresponding to all the entries of the retrieval word. According to the method and the device, the full-text retrieval on the precise external storage based on the general relational database is realized, and the enterprise-level full-text retrieval scene is met.

Description

Technical Field

[0001] The present application relates to the technical field of full-text retrieval, and in particular to a full-text retrieval method, device, equipment and medium based on a hash slot storage structure. Background Art

[0002] With the rapid development of the Internet and computer technology, data within enterprises is also growing exponentially. For rapidly developing enterprises, human and time costs are extremely precious. An efficient full-text search system within an enterprise, that is, a system that quickly screens and retrieves valuable data that meets the search criteria within the enterprise under limited information, accelerates information acquisition, reduces human decision-making costs, and promotes productivity development. It is an indispensable product.

[0003] With the rapid development of the Internet, full-text search technology has gradually been valued and widely used, and is now relatively mature. The full-text search backend persistent data storage structure usually uses an inverted index, which uses a certain word segmentation algorithm to split the document content into several entries, and creates an inverted index information from the entry to the document number on the disk. This is a solution that trades space for time, and from the actual effect, the benefits it brings are very significant.

[0004] In enterprise server applications, business data is generally stored in relational databases such as MySQL and Oracle. Such databases usually use B+ trees to quickly index data on disk. However, B+ tree indexes are not well adapted to scenarios where the search terms are ambiguous. The query process in such scenarios will degenerate into a full table scan, which is also the scenario where full-text search is born. For vertical search of enterprise information, among the existing full-text search technologies, taking ElasticSearch, a leading open source solution, as an example, its main shortcomings are:

[0005] 1. The configuration and management are complex. 2. There is a lot of internal cache information, and the actual deployment model has high memory requirements. 3. Semantic word segmentation is not well compatible with the situation where the user input search terms are meaningless or even the key columns that need to be indexed are meaningless, resulting in missing search results. 4. Using LSMTree as the backend storage structure of its inverted index, the read operation may need to query multiple shard files, resulting in read amplification, thus affecting the read performance. The full name of LSMTree is LogStructuredMergeTree, which is a tree that uses log append writing, has a certain structure, and can be merged.

[0006] For enterprise-level B-side products that need to compress and optimize resource consumption as much as possible and do not expect deviations in search results, the above-mentioned full-text search engine is not conducive to product promotion, nor is it the first choice, and cannot meet enterprise-level full-text search scenarios. Summary of the invention

[0007] The purpose of the present application is to provide a full-text retrieval method, device, electronic device and storage medium based on a hash slot storage structure to solve at least one problem existing in the above-mentioned prior art.

[0008] In a first aspect, an embodiment of the present application provides a full-text search method based on a hash slot storage structure, comprising:

[0009] Receiving a search term, performing N-Gram segmentation on the search term to obtain a series of terms;

[0010] For each of the entries, a hash modulo operation is performed on the entry to obtain a hash value corresponding to the entry;

[0011] Match the hash value with the pre-established slot block chain fast access array, determine the slot block chain corresponding to the hash value, use read mapping to map each slot data on the slot block chain to the memory one by one from the pre-established storage file system, restore the document ID corresponding to each slot data through the incremental encoding FOR restoration algorithm, and obtain the inverted list corresponding to the entry; the storage file system is established based on the hash slot file storage structure;

[0012] The search results are output according to the inverted list corresponding to all entries of the search term.

[0013] In a possible implementation, the creation process of the hash slot file storage structure is as follows:

[0014] An initialized storage file system is established based on a hash slot file storage structure, the hash slot file storage structure is composed of multiple slots, the data area of ​​each slot is composed of multiple frames, and the header of each slot is used to record the basic information of the slot, the basic information includes: the last write position, the number of documents contained in the last frame, the data bit width of the last frame, the last frame offset, and the last document ID in the slot, recorded as LastDocId, as well as the forward slot offset and the backward slot offset;

[0015] Establishing a slot block chain fast access array for the storage file system;

[0016] Establishing a read mapping and a write mapping of the storage file system to the memory;

[0017] Based on a preset index creation process, key column information is read from the database in batches to establish an inverted index, and the inverted index is stored in the storage file system.

[0018] In a possible implementation, the key column information includes a DocId column and a Text column for which a full-text search needs to be established.

[0019] In a possible implementation, the index creation process is as follows:

[0020] Incrementally read a batch of key column information to be used to create inverted indexes from the database according to the index creation schedule;

[0021] Process the retrieved batch data one by one. First, use the N-Gram word segmentation algorithm to segment the Text column to obtain a series of entries. If the length of the Text column content is less than 1, retain the entry.

[0022] Perform hashing modulo operation on the series of terms one by one to obtain the slot block chain number of each term stored in the hash slot file storage structure, use the slot block chain fast access array to directly access the last slot of the slot block chain with the number, use write mapping to write the corresponding slot to the memory, read the LastDocId recorded in the basic information of the slot, and allow writing when the LastDocId is greater than the current DocId to be written;

[0023] Calculate the remaining bit space in the slot, calculate the difference between the current DocId to be written and the LastDocId, and calculate the minimum bit width required to write the difference based on the difference;

[0024] Writing the difference into the storage file system according to the remaining bit space, the minimum bit width and the basic information of the slot;

[0025] Repeat the above process until all data processing of the current batch is completed, and finally update LastDocId to the index creation progress table to mark the completion of the current batch index creation.

[0026] In a possible implementation, writing the difference into a storage file system according to the remaining bit space, the minimum bit width, and the basic information of the slot includes:

[0027] If the last frame data bit width recorded in the basic information of the slot is greater than or equal to the minimum bit width, and the remaining bit space is greater than or equal to the minimum bit width, it is determined that the remaining space of the slot is sufficient, and the difference is written in accordance with the last frame data bit width through the FOR algorithm.

[0028] In a possible implementation, writing the difference into a storage file system according to the remaining bit space, the minimum bit width, and the basic information of the slot includes:

[0029] If the data bit width of the last frame recorded in the basic information of the slot is greater than or equal to the minimum bit width and the remaining bit space is less than the minimum bit width, it is determined that the remaining space of the slot is insufficient. Then, an attempt is made to apply for slot space from the free slot or expand the storage file system to create a new free slot, establish a two-way link with the current full slot, initialize the free slot space information obtained, and after switching to a new frame, write the current DocId as the initial value into the frame in the newly created slot through the FOR algorithm.

[0030] In a possible implementation, the writing the difference into the storage file system according to the remaining bit space, the minimum bit width, and the basic information of the slot includes:

[0031] If the data bit width data_bits of the last frame recorded in the basic information of the slot is less than the minimum bit width least_bits, enter the rewrite logic of the current frame:

[0032] Read the entire frame again, restore the inverted list sequence, and obtain the number of elements n in the frame;

[0033] Calculate (1 + n) * least_bits to obtain the required bit width need_bits. If need_bits < remain_bits, it is determined that the remaining space of the slot is sufficient, directly use least_bits as the frame bit width, and write the currently restored inverted list and the current DocId into the current frame through the FOR algorithm;

[0034] If need_bits >= remain_bits, it is determined that the remaining space of the slot is insufficient. Apply for slot space from the free slot or expand the storage file system to create a new free slot, establish a two-way link with the current full slot, initialize the free slot space information obtained, and after switching to a new frame, write the current DocId as the initial value into the frame in the newly created slot through the FOR algorithm.

[0035] In a second aspect, an embodiment of the present application provides a full-text retrieval device based on a hash slot storage structure, including:

[0036] A word segmentation module, configured to receive a search term and perform N-Gram word segmentation on the search term to obtain a series of entries;

[0037] A hash calculation module, configured to perform a hash modulo operation on each of the entries to obtain a hash value corresponding to the entry;

[0038] A retrieval module is used to match the hash value with a pre-established slot block chain fast access array, determine the slot block chain corresponding to the hash value, and use a read mapping to map each slot data on the slot block chain to the memory one by one from a pre-established storage file system, and restore the document ID corresponding to each slot data by an incremental encoding FOR restoration algorithm to obtain the inverted list corresponding to the entry; the storage file system is established based on the hash slot file storage structure;

[0039] The output module is used to output the search results according to the inverted lists corresponding to all entries of the search term.

[0040] In a third aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions can be executed by a processor to implement the method described in the first aspect.

[0042] Compared with the prior art, the full-text search method based on the hash slot storage structure provided by the present application receives the search term, performs N-Gram word segmentation on the search term to obtain a series of entries; for each of the entries, performs hash modulo operation on the entry to obtain the hash value corresponding to the entry; matches the hash value with the pre-established slot block chain fast access array, determines the slot block chain corresponding to the hash value, and uses read mapping to map each slot data on the slot block chain to the memory one by one from the pre-established storage file system, and restores the document ID corresponding to each slot data through the incremental encoding FOR restoration algorithm to obtain the inverted list corresponding to the entry; the storage file system is established based on the hash slot file storage structure; and outputs the search results according to the inverted list corresponding to all the entries of the search term. Compared with the prior art, the present application establishes a storage file system based on the hash slot file storage structure on the external memory for storing the inverted list in the full-text search, and the slot space in the hash slot file can be expanded to form a slot block chain structure, which can realize the accurate full-text search on the external memory based on the general relational database, and meet the enterprise-level full-text search scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0044] Figure 1 A flowchart of a full-text search method based on a hash slot storage structure provided in an embodiment of the present application is shown;

[0045] Figure 2 A schematic diagram of the search process provided in the embodiment of the present application is shown;

[0046] Figure 3 A schematic diagram showing a hash slot file storage structure provided in an embodiment of the present application is shown;

[0047] Figure 4 A schematic diagram of the initialization process of the storage file system provided in an embodiment of the present application is shown;

[0048] Figure 5 A schematic diagram of the index creation process provided in an embodiment of the present application is shown;

[0049] Figure 6 A schematic diagram of a full-text search device based on a hash slot storage structure provided in an embodiment of the present application is shown;

[0050] Figure 7 A schematic diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0052] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0053] In addition, the terms "first" and "second" etc. are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0054] First, we introduce the basic term definitions in the field and this application:

[0055] Document (Doc): In the full-text search system, it refers to all the content to be searched in the database. It is not limited to the content in the document, but can also be any data that needs to establish a full-text index, such as email information and file names.

[0056] Document ID (DocId): The unique identifier of a document, usually the row number ID or RowId in the database.

[0057] Term: The indexed documents are cut according to certain rules to obtain terms.

[0058] Term Dictionary: A dictionary consists of many terms. Usually, a dictionary is an index structure, and each index item in the dictionary points to the corresponding data area content.

[0059] Posting list: The simplest posting list structure can only contain monotonically increasing DocIds, which are also the data area pointed to by the index items in the dictionary.

[0060] N-Gram word segmentation: A word segmentation algorithm that agrees that each step forward is 1 visible character long, and extracts a word of length N, and repeats this operation until the end of the text is extracted. N can be any integer, which leads to 1-Gram (or Uni-Gram), 2-Gram (or Bi-Gram), and 3-Gram (or Tri-Gram).

[0061] Taking Chinese as an example, assuming the string "I love to eat carrots", the following entries will be obtained through Bi-Gram processing: "I love", "love to eat", "eat carrots", "carrots", and "radish".

[0062] Slot: Similar to the page in the B-tree file system, it is used to store indexes or data. When a slot data area is full, it can be connected to the free slots using a bidirectional linked list, pointing to each other, which is equivalent to the structure of a bidirectional block chain.

[0063] Frame: The data area in the slot is composed of multiple Frames. The device of the example of this application stipulates that each Frame can accommodate a maximum of 256 DocIds.

[0064] FOR (Frame of Reference incremental encoding): A difference encoding technology for integer sequences. For example, if there is an integer sequence [73, 300, 302, 332, 343, 372], the incremental value of each element relative to the previous element (the previous element of the first element is 0 by default, and the incremental value is itself) is [73, 227, 2, 30, 11, 29]. In this new list, all ids are less than 255, so each element only needs one byte to store.

[0065] The following is a detailed introduction to this application.

[0066] Please refer to Figure 1 , Figure 1 1 is a flowchart of a full-text search method based on a hash slot storage structure provided in an embodiment of the present application, comprising the following steps S101 to S104:

[0067] S101, receiving a search term, performing N-Gram segmentation on the search term to obtain a series of entries;

[0068] In actual applications, the front end provides a search term input box. After the user presses the search button, the search parameters are passed to the back end, and the back end performs Bi-Gram segmentation on the input search term to obtain a series of entries.

[0069] S102, for each of the entries, performing a hash modulo operation on the entry to obtain a hash value corresponding to the entry;

[0070] In this step, hash modulo operations are performed on the obtained entries respectively to obtain hash values ​​corresponding to each entry, and the slot block chain number corresponding to each entry can be further obtained according to the hash value.

[0071] S103, matching the hash value with a pre-established slot block chain fast access array, determining the slot block chain corresponding to the hash value, using a read mapping to map each slot data on the slot block chain to the memory one by one from a pre-established storage file system, restoring the document ID corresponding to each slot data by an incremental encoding FOR restoration algorithm, and obtaining a posting list corresponding to the entry;

[0072] The storage file system of the present application is established based on the hash slot file storage structure, and the specific establishment process is introduced later.

[0073] S104: Output the search results according to the posting list corresponding to all entries of the search term.

[0074] Read out the Posting lists corresponding to all entries, and intersect the obtained Posting list sets in pairs to finally obtain a thin Posing list. Use the SQL in operation to retrieve the corresponding row information of the Posting list from the database and temporarily store it in a container with DocId as the key. Organize the data in the form of Text column, separator, DocId, and line break and send it to the regular matching system for matching. Only the DocId row data that has been successfully matched is retained in the temporary storage container, and the data is organized in the form agreed with the front end and returned to the front end for display.

[0075] It can be seen that this application realizes accurate full-text retrieval on external storage based on a general relational database through hashing ideas and regular expression systems.

[0076] The above is the entire search process provided by this application. For ease of understanding, this application provides the following Figure 2 The retrieval process diagram is shown.

[0077] The following is a detailed introduction to the creation process of the storage file system provided by this application.

[0078] The storage file system of the present application is established based on the hash slot file storage structure. The creation process of the storage file system is as follows:

[0079] An initialized storage file system is established based on a hash slot file storage structure, the hash slot file storage structure is composed of multiple slots, the data area of ​​each slot is composed of multiple frames, and the header of each slot is used to record the basic information of the slot, the basic information includes: the last write position, the number of documents contained in the last frame, the data bit width of the last frame, the last frame offset, and the last document ID in the slot, recorded as LastDocId, as well as the forward slot offset and the backward slot offset;

[0080] Establishing a slot block chain fast access array for the storage file system;

[0081] Establishing a read mapping and a write mapping of the storage file system to the memory;

[0082] Based on a preset index creation process, key column information is read from the database in batches to establish an inverted index, and the inverted index is stored in the storage file system.

[0083] Specifically, the key column information includes a DocId column and a Text column for which full-text search needs to be established. Text is a column type in a MySQL database table that is used to store relatively long text data.

[0084] like Figure 3The figure shows a schematic diagram of the hash slot file storage structure provided by the present application. The hash slot file storage structure consists of 65536 slot spaces, and the header of each slot space records the basic information of each slot. Through these basic information, the fast writing of the inverted index and the fast encoding of the FOR algorithm can be realized, thereby achieving data security and improving the reading and writing efficiency. The storage structure on the external memory is the first in the full-text retrieval technology and is also the cornerstone for realizing the full-text retrieval of the present application.

[0085] Apply for two 64kb mmap spaces, one for read mapping and one for write mapping, to ensure read-write separation. Each slot space is 4KB in size, that is, one mmap mapping page space can accommodate 16 slots. If the target read-write slot is already in the mapping page, there is no need to change the mapping address. In addition, the storage file system consists of 65536 slot spaces, but in the actual operation process, if any slot space data area is full, the system will dynamically expand the file system size, apply for a batch of new slot spaces, mark them as free slots, take out a slot space from the free slot space, and record the address offsets with the full slot space, forming a bidirectional linked list structure logically, which can also be called a slot block chain. Assuming that the number of DocIds recorded in the Frame, that is, the data block in the slot space reaches 256, a new Frame or Slot will be created for writing. It should be known that subsequent reading and writing of the storage file system will follow these rules.

[0086] like Figure 4 The figure shows the initialization process of the storage file system. If the storage file system does not exist, it is the first initialization. The storage file system will be established first, and then all slot spaces will be initialized. If the storage file system exists, it is not the first initialization. All slot spaces will be traversed and the basic information of each slot space header will be verified. The address space from the last write offset position to the current slot space will be cleared. If there is data in this address space, it means that it is dirty data that has not been successfully written to the disk and can be erased.

[0087] Regardless of the initialization traversal or the subsequent non-initialization traversal, an array of 65536 of the same size as the basic slot space will be maintained in the memory to record the offset of the last slot space of each slot block chain, avoiding the disk arm movement caused by the linked list traversal during write and improving the writing efficiency.

[0088] For ease of understanding, this application provides Figure 5 The index creation process diagram shown in the figure is as follows. For details, please refer to Figure 5 , the index creation process is as follows:

[0089] Incrementally read a batch of key column information to be used to create inverted indexes from the database according to the index creation schedule;

[0090] The retrieved batch data are processed one by one. First, the N-Gram word segmentation algorithm is used to segment the Text column to obtain a series of entries. If the length of the Text column content is less than 1, the entry is retained. Specifically, the N-Gram word segmentation algorithm can adopt the Bi-Gram word segmentation algorithm. Different from the traditional Bi-Gram, if the length of the Text column content is less than 1, the application will retain the entry.

[0091] Perform hashing modulo operation on the series of terms one by one to obtain the slot block chain number of each term stored in the hash slot file storage structure, use the slot block chain fast access array to directly access the last slot of the slot block chain with the number, use write mapping to write the corresponding slot to the memory, read the LastDocId recorded in the basic information of the slot, and allow writing when the LastDocId is greater than the current DocId to be written;

[0092] Calculate the remaining bit space remaining_bits in the slot, calculate the difference delta between the current DocId to be written and LastDocId, and calculate the minimum bit width least_bits required to write the difference through the difference;

[0093] The difference is written into the storage file system according to the remaining bit space, the minimum bit width and the basic information of the slot.

[0094] Specifically, the above-mentioned writing the difference into the storage file system according to the remaining bit space, the minimum bit width and the basic information of the slot specifically includes:

[0095] If the last frame data bit width data_bits recorded in the basic information of the slot is greater than or equal to the minimum bit width least_bits, and the remaining bit space remaining_bits is greater than or equal to the minimum bit width least_bits, it is determined that the remaining space of the slot is sufficient, and the difference is written in accordance with the last frame data bit width through the FOR algorithm.

[0096] If the last frame data bit width data_bits recorded in the basic information of the slot is greater than or equal to the minimum bit width least_bits, and the remaining bit space remaining_bits is less than the minimum bit width least_bits, it is determined that the remaining space of the slot is insufficient, and an attempt is made to apply for slot space from a free slot or to expand the storage file system to create a new free slot to establish a bidirectional link with the current full slot, initialize the applied free slot space information, and after switching to a new frame, write the current DocId as the initial value into the frame in the new slot through the FOR algorithm.

[0097] If the data bit width data_bits of the last frame recorded in the basic information of the slot is less than the minimum bit width least_bits, enter the rewrite logic of the current frame:

[0098] Read the entire frame again, restore the inverted list sequence, and obtain the number of elements n in the frame;

[0099] Calculate (1 + n) * least_bits to obtain the required bit width need_bits. If need_bits < remain_bits, it is determined that the remaining space of the slot is sufficient, directly use least_bits as the frame bit width, and write the currently restored inverted list and the current DocId into the current frame through the FOR algorithm;

[0100] If need_bits >= remain_bits, it is determined that the remaining space of the slot is insufficient. Apply for slot space from the free slot or expand the storage file system to create a new free slot, establish a two-way link with the current full slot, initialize the free slot space information applied for, and after switching to the new frame, write the current DocId as the initial value into the frame in the newly created slot through the FOR algorithm.

[0101] Repeat the above process until all data in the current batch is processed. Finally, update LastDocId to the index creation progress table and mark the completion of the index creation for the current batch.

[0102] It should be noted that at the end of each write of DocId, the number of DocIds in the Frame will be judged. When the number of DocIds in the Frame reaches the agreed 256, frame offset is performed to ensure that the above complex write logic does not need to be coupled with the Frame switching process. The Frame switching process is to calculate the remaining bit space remain_bits of the current slot. If remain_bits is greater than or equal to 32 (i.e., the maximum bit width in the FOR algorithm), the address of the next frame is set to the next byte address of the current read / write offset and updated to the head of the slot space. The next time this slot is operated, it can be directly offset and mapped for use. If remain_bits is less than 32, it is necessary to try to apply for slot space from the free slot or expand the storage file system to create a new free slot space, establish a two-way link with the current full slot, and initialize the free slot space information and Frame information applied for.

[0103] The full-text retrieval method based on the hash slot storage structure provided by the embodiments of the present application can solve problems such as high system complexity, high machine resource requirements, and possible omissions in retrieval results in the prior art. Furthermore, it can solve the full-text retrieval scenarios that cannot be satisfied by the conventional relational databases of enterprise-level B-side server products, so as to accelerate the rapid screening and acquisition of valuable data within the enterprise, reduce the time spent on manual decision-making, and promote the internal development of the enterprise.

[0104] In the above-mentioned embodiment, a full-text retrieval method based on a hash slot storage structure is provided. Correspondingly, the present application also provides a full-text retrieval device based on a hash slot storage structure. The full-text retrieval device based on a hash slot storage structure provided in the embodiment of the present application can implement the above-mentioned full-text retrieval method based on a hash slot storage structure. The full-text retrieval device based on a hash slot storage structure can be implemented by software, hardware, or a combination of software and hardware. For example, the full-text retrieval device based on a hash slot storage structure can include integrated or separate functional modules or units to execute the corresponding steps in the above-mentioned methods. Please refer to Figure 6 As shown, the full-text search device 10 based on the hash slot storage structure of the present application includes:

[0105] The word segmentation module 101 is used to receive a search word and perform N-Gram word segmentation on the search word to obtain a series of entries;

[0106] A hash calculation module 102 is used to perform a hash modulo operation on each of the terms to obtain a hash value corresponding to the term;

[0107] The retrieval module 103 is used to match the hash value with the pre-established slot block chain fast access array, determine the slot block chain corresponding to the hash value, use read mapping to map each slot data on the slot block chain to the memory one by one from the pre-established storage file system, restore the document ID corresponding to each slot data by incremental coding FOR restoration algorithm, and obtain the inverted list corresponding to the entry; the storage file system is established based on the hash slot file storage structure;

[0108] The output module 104 is used to output the search results according to the posting lists corresponding to all entries of the search term.

[0109] In a possible implementation, the device further includes:

[0110] A creation module is used to create the hash slot file storage structure. The process is as follows:

[0111] An initialized storage file system is established based on a hash slot file storage structure, the hash slot file storage structure is composed of multiple slots, the data area of ​​each slot is composed of multiple frames, and the header of each slot is used to record the basic information of the slot, the basic information includes: the last write position, the number of documents contained in the last frame, the data bit width of the last frame, the last frame offset, and the last document ID in the slot, recorded as LastDocId, as well as the forward slot offset and the backward slot offset;

[0112] Establishing a slot block chain fast access array for the storage file system;

[0113] Establishing a read mapping and a write mapping of the storage file system to the memory;

[0114] Based on a preset index creation process, key column information is read from the database in batches to establish an inverted index, and the inverted index is stored in the storage file system.

[0115] In a possible implementation, the key column information includes a DocId column and a Text column for which a full-text search needs to be established.

[0116] In a possible implementation, the index creation process is as follows:

[0117] Incrementally read a batch of key column information to be used to create inverted indexes from the database according to the index creation schedule;

[0118] Process the retrieved batch data one by one. First, use the N-Gram word segmentation algorithm to segment the Text column to obtain a series of entries. If the length of the Text column content is less than 1, retain the entry.

[0119] Perform hashing modulo operation on the series of terms one by one to obtain the slot block chain number of each term stored in the hash slot file storage structure, use the slot block chain fast access array to directly access the last slot of the slot block chain with the number, use write mapping to write the corresponding slot to the memory, read the LastDocId recorded in the basic information of the slot, and allow writing when the LastDocId is greater than the current DocId to be written;

[0120] Calculate the remaining bit space in the slot, calculate the difference between the current DocId to be written and the LastDocId, and calculate the minimum bit width required to write the difference based on the difference;

[0121] Writing the difference into the storage file system according to the remaining bit space, the minimum bit width and the basic information of the slot;

[0122] Repeat the above process until all data processing of the current batch is completed, and finally update LastDocId to the index creation progress table to mark the completion of the current batch index creation.

[0123] In a possible implementation, the creation module is specifically used to:

[0124] If the last frame data bit width recorded in the basic information of the slot is greater than or equal to the minimum bit width, and the remaining bit space is greater than or equal to the minimum bit width, it is determined that the remaining space of the slot is sufficient, and the difference is written in accordance with the last frame data bit width through the FOR algorithm.

[0125] In a possible implementation, the creation module is specifically used to:

[0126] If the data bit width of the last frame recorded in the basic information of the slot is greater than or equal to the minimum bit width and the remaining bit space is less than the minimum bit width, it is determined that the remaining space of the slot is insufficient. Then, an attempt is made to apply for slot space from an idle slot or expand the storage file system to create a new idle slot, establish a two-way link with the current full slot, initialize the information of the applied idle slot space, and after switching to a new frame, write the current DocId as the initial value into the frame in the newly created slot through the FOR algorithm.

[0127] In a possible implementation manner, the creating module is specifically configured to:

[0128] If the data bit width data_bits of the last frame recorded in the basic information of the slot is less than the minimum bit width least_bits, enter the rewrite logic of the current frame:

[0129] Read the entire frame again, restore the inverted list sequence, and obtain the number of elements n in the frame;

[0130] Calculate (1 + n) * least_bits to obtain the required bit width need_bits. If need_bits < remain_bits, it is determined that the remaining space of the slot is sufficient, directly use least_bits as the frame bit width, and write the currently restored inverted list and the current DocId into the current frame through the FOR algorithm;

[0131] If need_bits >= remain_bits, it is determined that the remaining space of the slot is insufficient. Apply for slot space from an idle slot or expand the storage file system to create a new idle slot, establish a two-way link with the current full slot, initialize the information of the applied idle slot space, and after switching to a new frame, write the current DocId as the initial value into the frame in the newly created slot through the FOR algorithm.

[0132] The full-text retrieval device based on the hash slot storage structure provided by the embodiments of the present application and the full-text retrieval method based on the hash slot storage structure provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run, or implemented by them.

[0133] The embodiments of the present application also provide an electronic device corresponding to the method provided in the foregoing embodiments. The electronic device may be an electronic device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the above-mentioned full-text retrieval method based on the hash slot storage structure.

[0134] Please refer to Figure 7 which shows a schematic diagram of an electronic device provided by some embodiments of the present application. As Figure 7As shown, the electronic device 20 includes: a processor 200, a memory 201, a bus 202 and a communication interface 203, and the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and when the processor 200 runs the computer program, it executes the full-text retrieval method based on the hash slot storage structure provided in any of the aforementioned embodiments of the present application.

[0135] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0136] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving the execution instruction. The full-text retrieval method based on the hash slot storage structure disclosed in any implementation of the embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.

[0137] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

[0138] The electronic device provided in the embodiment of the present application and the full-text retrieval method based on the hash slot storage structure provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.

[0139] An embodiment of the present application also provides a computer-readable storage medium corresponding to the full-text retrieval method based on the hash slot storage structure provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the full-text retrieval method based on the hash slot storage structure provided in any of the aforementioned embodiments.

[0140] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0141] The computer-readable storage medium provided by the above-mentioned embodiments of the present application and the full-text retrieval method based on the hash slot storage structure provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs they store.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application.

Claims

1. A full-text retrieval method based on a hash slot storage structure, characterized in that: include: Receiving a search term, performing N-Gram segmentation on the search term to obtain a series of terms; For each of the entries, a hash modulo operation is performed on the entry to obtain a hash value corresponding to the entry; Match the hash value with the pre-established slot block chain fast access array, determine the slot block chain corresponding to the hash value, use read mapping to map each slot data on the slot block chain to the memory one by one from the pre-established storage file system, restore the document ID corresponding to each slot data through the incremental encoding FOR restoration algorithm, and obtain the inverted list corresponding to the entry; The storage file system is established based on the hash slot file storage structure; The search results are output according to the inverted list corresponding to all entries of the search term.

2. The method according to claim 1, characterized in that The process of creating the storage file system is as follows: An initialized storage file system is established based on a hash slot file storage structure, the hash slot file storage structure is composed of multiple slots, the data area of ​​each slot is composed of multiple frames, and the header of each slot is used to record the basic information of the slot, the basic information includes: the last write position, the number of documents contained in the last frame, the data bit width of the last frame, the last frame offset, and the last document ID in the slot, recorded as LastDocId, as well as the forward slot offset and the backward slot offset; Establishing a slot block chain fast access array for the storage file system; Establishing a read mapping and a write mapping of the storage file system to the memory; Based on a preset index creation process, key column information is read from the database in batches to establish an inverted index, and the inverted index is stored in the storage file system.

3. The method according to claim 2, characterized in that The key column information includes the DocId column and the Text column for which full-text search needs to be established.

4. The method according to claim 3, characterized in that: The index creation process is as follows: Incrementally read a batch of key column information to be used to create inverted indexes from the database according to the index creation schedule; Process the retrieved batch data one by one. First, use the N-Gram word segmentation algorithm to segment the Text column to obtain a series of entries. If the length of the Text column content is less than 1, retain the entry. Perform hashing modulo operation on the series of terms one by one to obtain the slot block chain number of each term stored in the hash slot file storage structure, use the slot block chain fast access array to directly access the last slot of the slot block chain with the number, use write mapping to write the corresponding slot to the memory, read the LastDocId recorded in the basic information of the slot, and allow writing when the LastDocId is greater than the current DocId to be written; Calculate the remaining bit space in the slot, calculate the difference between the current DocId to be written and the LastDocId, and calculate the minimum bit width required to write the difference based on the difference; Writing the difference into the storage file system according to the remaining bit space, the minimum bit width and the basic information of the slot; Repeat the above process until all data processing of the current batch is completed, and finally update LastDocId to the index creation progress table to mark the completion of the current batch index creation.

5. The method according to claim 4, characterized in that The step of writing the difference into a storage file system according to the remaining bit space, the minimum bit width and the basic information of the slot includes: If the data bit width of the last frame recorded in the basic information of the slot is greater than or equal to the minimum bit width, and the remaining bit space is greater than or equal to the minimum bit width, it is determined that the remaining space of the slot is sufficient, then follow the data bit width of the last frame and write the difference through the FOR algorithm.

6. The method according to claim 4, characterized in that Writing the difference into the storage file system according to the remaining bit space, the minimum bit width, and the basic information of the slot includes: If the data bit width of the last frame recorded in the basic information of the slot is greater than or equal to the minimum bit width, and the remaining bit space is less than the minimum bit width, it is determined that the remaining space of the slot is insufficient, then attempt to apply for slot space from an idle slot or expand the storage file system to create a new idle slot, establish a two-way link with the current full slot, initialize the space information of the applied idle slot, and after switching to a new frame, write the current DocId as the initial value into the frame in the newly created slot through the FOR algorithm.

7. The method according to claim 4, characterized in that Writing the difference into the storage file system according to the remaining bit space, the minimum bit width, and the basic information of the slot includes: If the data bit width data_bits of the last frame recorded in the basic information of the slot is less than the minimum bit width least_bits, enter the rewrite logic of the current frame: Read the entire frame again, restore the inverted list sequence and obtain the number of elements n in the frame; Calculate (1 + n) * least_bits to get the required bit width need_bits. If need_bits < remain_bits, where remain_bits is the remaining bit space, it is determined that the remaining space of the slot is sufficient, directly use least_bits as the frame bit width, and write the currently restored inverted list and the current DocId into the current frame through the FOR algorithm; If need_bits >= remain_bits, it is determined that the remaining space of the slot is insufficient, apply for slot space from an idle slot or expand the storage file system to create a new idle slot, establish a two-way link with the current full slot, initialize the space information of the applied idle slot, and after switching to a new frame, write the current DocId as the initial value into the frame in the newly created slot through the FOR algorithm.

8. A full-text search device based on a hash slot storage structure, characterized in that: Including: A word segmentation module, which is used to receive a search term and perform N-Gram word segmentation on the search term to obtain a series of terms; A hash calculation module, which is used to perform a hash modulo operation on each of the terms to obtain the hash value corresponding to the term; A retrieval module, which is used to match the hash value with a pre-established slot block chain fast access array to determine the slot block chain corresponding to the hash value, map each slot data on the slot block chain to the memory one by one using a read mapping from a pre-created storage file system, and restore the document ID corresponding to each slot data through an incremental encoding FOR restoration algorithm to obtain the inverted list corresponding to the term; The storage file system is established based on a hash slot file storage structure; An output module, which is used to output a retrieval result according to the inverted lists corresponding to all terms of the search term.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1 to 7.

Images