A method and device for implementing incremental inverted index data storage

Abstract

The present invention discloses a method and device for implementing incremental inverted index data storage, and relates to the field of computer technology. A specific implementation of the method includes: determining the required memory space for the incremental inverted index data to be stored; judging whether the required memory space for the incremental inverted index data to be stored is greater than the remaining memory space of the pre-allocated memory pool; if so, applying for a new memory pool, and storing the incremental inverted index data to be stored in the pre-allocated memory pool and the new memory pool, and marking the pre-allocated memory pool after the storage operation as a memory pool to be compressed; otherwise, storing the incremental inverted index data to be stored in the pre-allocated memory pool; compressing the inverted index data in the memory pool to be compressed. This method greatly improves the performance of incremental inverted index data storage when the incremental data is uncertain. In addition, it is conducive to effective data compression and improves space utilization.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for implementing incremental inverted index data storage. Background Art

[0002] An inverted index, also often called an inverted index, an inverted file, or an inverted file, is an indexing method used to store the mapping of a word's storage location in a document or a group of documents under full-text search (the mapping relationship between words and documents). It is the most commonly used data structure in document retrieval systems. Through the inverted index, you can quickly obtain a list of documents containing the word based on the word. Based on the real-time nature and flexibility of the data, the inverted index is divided into a full inverted index and an incremental inverted index. The full inverted index is generally built offline, has an immutable structure, and is slow to load and update, so it cannot be built frequently. However, in a product search system, merchants or products sometimes need to change frequently, so an incremental inverted index appears. The incremental inverted index has the characteristics of online real-time construction and high timeliness.

[0003] The basic structure of an incremental inverted index is: the correspondence between an inverted term and a set of document identifiers (docids). This correspondence allows us to know in which documents a term appears. The document identifier can be the internal ID of a product in a search engine, and the docid set is called a doclist. In the prior art, whenever incremental inverted index data needs to be stored, the required storage space is requested in real time to complete the storage. However, the incremental inverted index data is uncertain, and memory space may be requested frequently, which will result in reduced system performance. Moreover, the stored data are all data fragments, and the compression of ordered integers is only effective when the number reaches a certain level. Therefore, the prior art is not conducive to effective data compression, which in turn wastes a lot of memory space. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for implementing incremental inverted index data storage, which can significantly improve the performance of incremental inverted index data storage when the incremental data is uncertain. Furthermore, it facilitates effective data compression and improves space utilization.

[0005] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for implementing incremental inverted index data storage is provided.

[0006] The method for implementing incremental inverted index data storage in an embodiment of the present invention includes: determining the required memory space for the incremental inverted index data to be stored; judging whether the required memory space for the incremental inverted index data to be stored is greater than the remaining memory space of the pre-allocated memory pool; if so, applying for a new memory pool, and storing the incremental inverted index data to be stored in the pre-allocated memory pool and the new memory pool, and marking the pre-allocated memory pool after the storage operation as a memory pool to be compressed; otherwise, storing the incremental inverted index data to be stored in the pre-allocated memory pool; and compressing the inverted index data in the memory pool to be compressed.

[0007] Optionally, the step of storing the incremental inverted index data to be stored in the pre-allocated memory pool and the new memory pool includes: dividing the incremental inverted index data to be stored into first incremental inverted index data and second incremental inverted index data according to the remaining memory space; wherein the required memory space of the first incremental inverted index data is equal to the remaining memory space; storing the first incremental inverted index data in the pre-allocated memory pool; and storing the second incremental inverted index data in the new memory pool.

[0008] Optionally, the step of compressing the inverted index data in the memory pool to be compressed includes: for each inverted word in the inverted index data in the memory pool to be compressed, aggregating the document identification list of the inverted word to form a data block of the inverted word; and compressing the data block by using the PForDelta algorithm.

[0009] Optionally, after marking the pre-allocated memory pool after the storage operation as a memory pool to be compressed, and before compressing the inverted index data in the memory pool to be compressed, the method further includes: sending information of the memory pool to be compressed to a reconstruction queue;

[0010] The step of compressing the inverted index data in the memory pool to be compressed includes: rebuilding the memory pool to be compressed based on the consumed messages in the reconstruction queue; and compressing the inverted index data in the memory pool to be compressed during the reconstruction process.

[0011] To achieve the above-mentioned objective, according to another aspect of an embodiment of the present invention, a device for implementing incremental inverted index data storage is provided.

[0012] An apparatus for implementing incremental inverted index data storage in an embodiment of the present invention includes:

[0013] A required memory space determination module is used to determine the required memory space for storing incremental inverted index data;

[0014] A judgment module, configured to judge whether the memory space required to store the incremental inverted index data is greater than the remaining memory space in the pre-allocated memory pool;

[0015] A storage module is configured to apply for a new memory pool, store the incremental inverted index data to be stored in the pre-allocated memory pool and the new memory pool, mark the pre-allocated memory pool after the storage operation as a memory pool to be compressed, and store the incremental inverted index data to be stored in the pre-allocated memory pool;

[0016] The compression module is used to compress the inverted index data in the memory pool to be compressed.

[0017] Optionally, the storage module is also used to divide the incremental inverted index data to be stored into first incremental inverted index data and second incremental inverted index data according to the remaining memory space; wherein the required memory space of the first incremental inverted index data is equal to the remaining memory space; the first incremental inverted index data is stored in the pre-allocated memory pool; and the second incremental inverted index data is stored in the new memory pool.

[0018] Optionally, the compression module is further configured to, for each inverted word in the inverted index data in the memory pool to be compressed, summarize a document identification list of the inverted word to form a data block of the inverted word; and compress the data block using a PForDelta algorithm.

[0019] Optionally, the apparatus for implementing incremental inverted index data storage in an embodiment of the present invention further includes a sending module, configured to send information of the memory pool to be compressed to a reconstruction queue;

[0020] The compression module reconstructs the memory pool to be compressed based on the consumed messages in the reconstruction queue; and compresses the inverted index data in the memory pool to be compressed during the reconstruction process.

[0021] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device is provided.

[0022] An electronic device according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned methods for implementing incremental inverted index data storage.

[0023] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, any of the above-mentioned methods for realizing incremental inverted index data storage is implemented.

[0024] One embodiment of the above invention has the following advantages or beneficial effects: Because online incremental inverted index data is stored irregularly, a larger space (memory pool) can be pre-allocated, reducing performance losses caused by frequent requests for small memory blocks and reducing memory fragmentation, facilitating effective compression of inverted index data. This solves the existing problems of frequent memory allocation and low space utilization when large amounts of incremental inverted index data are present.

[0025] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0027] Figure 1 is a schematic diagram of the main process of a method for implementing incremental inverted index data storage according to an embodiment of the present invention;

[0028] Figure 2 is a schematic diagram of a document identification list summarizing inverted words according to an embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of a method for implementing incremental inverted index data storage according to an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of compressing a data block using a PForDelta algorithm according to an embodiment of the present invention;

[0031] Figure 5 is a schematic diagram of an inverted index data query according to an embodiment of the present invention;

[0032] Figure 6 is a schematic diagram of main modules of an apparatus for implementing incremental inverted index data storage according to an embodiment of the present invention;

[0033] Figure 7 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0034] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0036] Figure 1 is a schematic diagram of the main process of a method for implementing incremental inverted index data storage according to an embodiment of the present invention; Figure 2 4 is a schematic diagram of a document identification list of summarized inverted words according to an embodiment of the present invention.

[0037] like Figure 1 As shown, the method for implementing incremental inverted index data storage in an embodiment of the present invention mainly includes:

[0038] Step S101: determining the memory space required for storing the incremental inverted index data to be stored. The required memory space refers to the memory space required for storing the incremental inverted index data to be stored.

[0039] Step S102: Determine whether the memory space required to store the incremental inverted index data is greater than the remaining memory space in the pre-allocated memory pool. If so, execute step S103; otherwise, execute step S105. The pre-allocated memory pool is a memory pool that pre-allocates a certain amount of memory space. In an embodiment of the present invention, each time incremental inverted index data is constructed online in real time, it is preferentially stored in the pre-allocated memory pool. If the pre-allocated memory pool is full or the remaining memory space is insufficient, a new memory pool is requested.

[0040] Step S103: applying for a new memory pool, and storing the incremental inverted index data to be stored in the pre-allocated memory pool and the new memory pool, and marking the pre-allocated memory pool after the storage operation as a memory pool to be compressed.

[0041] Optionally, based on the remaining memory space, the incremental inverted index data to be stored is divided into the first incremental inverted index data and the second incremental inverted index data. The required memory space of the first incremental inverted index data is equal to the remaining memory space. The first incremental inverted index data is stored in a pre-allocated memory pool; the second incremental inverted index data is stored in a new memory pool. Through the above steps, the currently available pre-allocated memory pool can be filled first, and then a new memory pool can be applied for. Moreover, after the new memory pool is stored, a new memory pool is applied for again, thereby maximizing space utilization as much as possible.

[0042] Step S104: compress the inverted index data in the memory pool to be compressed.

[0043] Optionally, for each inverted word in the inverted index data in the memory pool to be compressed, the document identifier list of the inverted word is summarized to form a data block of the inverted word. Specifically, a hash function is used to convert the inverted word term to be created into a 64-bit integer termid (inverted word identifier), and the id of the doc where each term word is located is inserted into the corresponding doclist. Figure 2 As shown, for the inverted terms "mobile phone," "sneakers," and "eyes," which can also be product brands, after hashing, their termids are determined to be 1, 3, and 2, respectively. Furthermore, the doclists corresponding to termids 1, 3, and 2 are {12, 13, 30, 101}, {86, 93}, and {25, 36, 89}, respectively. In this embodiment of the present invention, the document identifier (docid) can be identification information generated by the system based on timestamp information and is generally continuous. Therefore, for each term's doclist, the included docids are arranged sequentially (from largest to smallest or vice versa).

[0044] The data blocks are then compressed using the PForDelta algorithm. The PForDelta algorithm (p4delta algorithm), first proposed by Heman in 2005 (Heman et al. ICDE 2006), allows for the simultaneous compression of entire chunks of data. The basic idea is that for a sequence of chunks, the majority (x%) of the data (e.g., 90%) takes up less space, while the remaining 1-x% (e.g., 10%) are the outliers that cause the data to be too large. Therefore, the x% of small data is uniformly stored using a smaller number (b bits), while the remaining 1-x% of data is stored separately.

[0045] After marking the pre-allocated memory pool after the storage operation as the memory pool to be compressed, and before compressing the inverted index data in the memory pool to be compressed, the method further includes: sending information about the memory pool to be compressed to a reconstruction queue. During the process of compressing the inverted index data in the memory pool to be compressed, the memory pool to be compressed is rebuilt based on the consumed messages in the reconstruction queue. During the reconstruction process, the inverted index data in the memory pool to be compressed is compressed.

[0046] Step S105: storing the incremental inverted index data to be stored in a pre-allocated memory pool.

[0047] In the embodiments of the present invention, since online incremental inverted index data is stored irregularly, a larger space (memory pool) can be pre-allocated, reducing the performance loss caused by frequent requests for small blocks of memory space and reducing memory fragmentation, facilitating effective compression of inverted index data. This solves the existing problems of frequent memory allocation and low space utilization when large amounts of incremental inverted index data are present.

[0048] Figure 3 is a schematic diagram of a method for implementing incremental inverted index data storage according to an embodiment of the present invention; Figure 4 is a schematic diagram of compressing a data block using a PForDelta algorithm according to an embodiment of the present invention; Figure 5 2 is a schematic diagram of inverted index data query according to an embodiment of the present invention.

[0049] In some application scenarios, such as determining a product list based on product query terms, the uncertainty of merchants listing products, as products can be added or removed at any time, leads to uncertainty in the incremental inverted index data. Existing technologies only allow for partial accumulation of incremental inverted index data, and when constructing inverted indexes for terms in this batch of incremental data, only the document IDs in that batch can be used to construct inverted indexes. Because memory is continuous, when the next batch of incremental data arrives, new space must be allocated to construct inverted indexes for the terms. If a new term appears in a previous inverted index, the same term will inevitably correspond to multiple, discontinuous inverted indexes. Compression of ordered integer doclists is only effective when the number reaches a certain level, and short chains have poor compression effects. Since the data in each inverted index is limited, it is not conducive to effective data compression.

[0050] To address the above problems, the embodiment of the present invention pre-allocates a larger space (memory pool), which not only reduces the performance loss caused by frequent application of small memory space, but also reduces the generation of memory fragmentation, thereby facilitating the effective compression of inverted index data. Figure 3 As shown, the method for implementing incremental inverted index data storage in an embodiment of the present invention includes:

[0051] Step S301: construct incremental inverted index data to be stored.

[0052] Step S302: Determine whether the incremental inverted index data to be stored can be stored in the current memory pool IncSegment. If yes, execute step S303; otherwise, execute step S306.

[0053] Step S303: insert the incremental inverted index data to be stored into the current IncSegment.

[0054] Step S304: If the IncSegment is full, it is placed in the rebuild queue and waits for the rebuild operation to be executed. To achieve memory compression, the data in the full IncSegment needs to be rebuilt regularly. In addition, for the full IncSegment, to facilitate the subsequent rebuild operation, it is set to read-only and will not be modified. To distinguish between unrebuilt data and rebuilt data, headers can be added to the data blocks, namely AliveBlockHeader and SolidBlockHeader.

[0055] Step S305: consume the messages in the rebuild queue, perform the rebuild operation and obtain the compressed memory pool. When rebuilding, the memory is accurately allocated according to the IncSegment to be rebuilt, avoiding memory gaps and achieving compact storage. Among them, each IncSegment has a hash_dict (dictionary) that records the inverted word term and the offset address of the BlockHeader in the IncSegment. The BlockHeader records the offset of the next BlockHeader of the term, and so on, in the form of a linked list. In this way, several short doclists corresponding to the same term in the IncSegment can be merged into one doclist, and p4delta compression is performed on it. Accumulate the length of all compressed data and record the compressed data in the temporary space. The total compressed data length can be directly allocated, and then copied from the temporary space in sequence.

[0056] The above operations, such as Figure 4 As shown, specifically including:

[0057] 1. Traverse all terms in the full IncSegment and aggregate the doclists corresponding to the same term. Use a hash function to convert the inverted terms to be created into 64-bit integer termids (inverted term identifiers). Insert the ID of the doc containing each term into the corresponding doclist.

[0058] 2. Re-allocate a block of memory. For each term's doclist, replace the AliveBlockHeader with a SolidBlockHeader and compress the data block using the p4delta algorithm. For example, compress the differences of 128 docids using the p4delta algorithm and write the maximum (or minimum) of these 128 docids into the corresponding BlockHeader. In this implementation, instead of compressing each 128 docids individually, the differences are calculated sequentially from back to front, resulting in 128 differences for compression. (The rightmost difference is 0, and the differences between two docids are calculated to the left.) The maximum docid in the BlockHeader can then be used to recover all the docids in the block.

[0059] 3. Repeat the above steps until all terms are rebuilt.

[0060] 4. Replace the original IncSegment pointer.

[0061] The docids in the block are arranged in ascending order. Since compressed data cannot be read directly, in order to accurately find the block where the specified docid is located and decompress it, the largest docid in the current block is recorded in the BlockHeader. The purpose of this operation is to find the first docid that is not less than the query docid. The search is performed using a jump method with a step size of 2 n (n>=1), where the input is docid, curr_header refers to the current header position, end_header refers to the position after the last header, step refers to the step size, and n refers to the calculation step size power. The input docid is the value to be queried, and curr_header is the BlockHeader currently traversed. By comparing the input docid with curr_header.max_docid, the subsequent jump direction and step size are determined. Figure 5 As shown, the search method is as follows:

[0062] 1. Initially n=1, step=0.

[0063] 2. As long as curr_header has not reached end_header and the maximum docid of curr_header (curr_header.max_docid) is smaller than the input docid, proceed to step 3; otherwise, end.

[0064] 3.step=2 nn is incremented by 1; the current curr_header moves backward by step. If the curr_header has not reached the end_header or the maximum docid of the curr_header is greater than the docid, proceed to step 4.

[0065] 4. curr_header moves forward by step - 1; set n to 1.

[0066] 5. Return to step 2.

[0067] The code is implemented as follows:

[0068]

[0069]

[0070] Step S306: Apply for a new IncSegment, and store the incremental inverted index data to be stored in the existing IncSegment and the new IncSegment.

[0071] Figure 6 FIG. 1 is a schematic diagram of main modules of an apparatus for storing incremental inverted index data according to an embodiment of the present invention. Figure 6 As shown, the apparatus 600 for implementing incremental inverted index data storage according to the embodiment of the present invention includes a required memory space determination module 601 , a judgment module 602 , a storage module 603 , and a compression module 604 .

[0072] The required memory space determining module 601 is used to determine the required memory space for storing incremental inverted index data.

[0073] The judgment module 602 is used to judge whether the memory space required for storing the incremental inverted index data is greater than the remaining memory space in the pre-allocated memory pool.

[0074] The storage module 603 is used to apply for a new memory pool after the judgment module 602 determines that the required memory space for the incremental inverted index data to be stored is greater than the remaining memory space in the pre-allocated memory pool. The storage module 603 also stores the incremental inverted index data to be stored in the pre-allocated memory pool and the new memory pool, and marks the pre-allocated memory pool after the storage operation as a memory pool to be compressed. After the judgment module 602 determines that the required memory space for the incremental inverted index data to be stored is not greater than the remaining memory space in the pre-allocated memory pool, the storage module 603 stores the incremental inverted index data to be stored in the pre-allocated memory pool.

[0075] The storage module is further configured to, based on the remaining memory space, divide the incremental inverted index data to be stored into first incremental inverted index data and second incremental inverted index data. The required memory space for the first incremental inverted index data is equal to the remaining memory space. The storage module is further configured to store the first incremental inverted index data in a pre-allocated memory pool and store the second incremental inverted index data in a new memory pool.

[0076] The compression module 604 is configured to compress the inverted index data in the memory pool to be compressed. The compression module is further configured to, for each inverted word in the inverted index data in the memory pool to be compressed, compile a list of document identifiers for the inverted word to form a data block for the inverted word. Furthermore, the compression module compresses the data block using the PForDelta algorithm.

[0077] The apparatus for storing incremental inverted index data in an embodiment of the present invention further includes a sending module configured to send information about the memory pool to be compressed to a reconstruction queue. The compression module rebuilds the memory pool to be compressed based on consumed messages in the reconstruction queue; during the reconstruction process, the inverted index data in the memory pool to be compressed is compressed.

[0078] In the embodiments of the present invention, since online incremental inverted index data is stored irregularly, a larger space (memory pool) can be pre-allocated, reducing the performance loss caused by frequent requests for small blocks of memory space and reducing memory fragmentation, facilitating effective compression of inverted index data. This solves the existing problems of frequent memory allocation and low space utilization when large amounts of incremental inverted index data are present.

[0079] Figure 7 An exemplary system architecture 700 is shown to which a method for implementing incremental inverted index data storage or an apparatus for implementing incremental inverted index data storage according to an embodiment of the present invention may be applied.

[0080] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, a network 704, and a server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0081] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0082] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0083] Server 705 may be a server that provides various services, such as a backend management server (for example only) that supports shopping websites browsed by users using terminal devices 701, 702, and 703. The backend management server may analyze and process received data such as product information query requests, and feed back the processing results to the terminal device.

[0084] It should be noted that the method for implementing incremental inverted index data storage provided by the embodiment of the present invention is generally executed by the server 705 , and accordingly, the device for implementing incremental inverted index data storage is generally provided in the server 705 .

[0085] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0086] Reference below Figure 8 , which shows a schematic structural diagram of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0087] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the system 800 are also stored in the RAM 803. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0088] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, and the like; an output section 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or a modem. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 810 as needed, so that computer programs read therefrom can be installed into the storage section 808 as needed.

[0089] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from a removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are performed.

[0090] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0092] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor including a required memory space determination module, a judgment module, a storage module, and a compression module. The names of these modules do not, in some cases, constitute a limitation on the modules themselves. For example, the required memory space determination module may also be described as a "module for determining the required memory space for storing incremental inverted index data."

[0093] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: determining the required memory space for storing the incremental inverted index data; judging whether the required memory space for storing the incremental inverted index data is greater than the remaining memory space of the pre-allocated memory pool; if so, applying for a new memory pool, and storing the incremental inverted index data to be stored in the pre-allocated memory pool and the new memory pool, and marking the pre-allocated memory pool after the storage operation as a memory pool to be compressed; otherwise, storing the incremental inverted index data to be stored in the pre-allocated memory pool; and compressing the inverted index data in the memory pool to be compressed.

[0094] In the embodiments of the present invention, since online incremental inverted index data is stored irregularly, a larger space (memory pool) can be pre-allocated, reducing the performance loss caused by frequent requests for small blocks of memory space and reducing memory fragmentation, facilitating effective compression of inverted index data. This solves the existing problems of frequent memory allocation and low space utilization when large amounts of incremental inverted index data are present.

[0095] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for implementing incremental inverted index data storage, characterized in that: include: Determine the required memory space to store the incremental inverted index data; Determine whether the memory space required to store the incremental inverted index data is greater than the remaining memory space in the pre-allocated memory pool; The pre-allocated memory pool is a memory pool that pre-allocates a certain amount of memory space; the memory pool is IncSegment, and each IncSegment has a dictionary that records inverted words and the offset address of the inverted words in the BlockHeader of the IncSegment; If yes, apply for a new memory pool, store the incremental inverted index data to be stored in the pre-allocated memory pool and the new memory pool, mark the pre-allocated memory pool after the storage operation as a memory pool to be compressed, and send the message of the memory pool to be compressed to the reconstruction queue; otherwise, store the incremental inverted index data to be stored in the pre-allocated memory pool; The memory pool to be compressed is rebuilt based on the consumed messages in the reconstruction queue; during the reconstruction process, the inverted index data in the memory pool to be compressed is compressed.

2. The method according to claim 1, characterized in that The step of storing the incremental inverted index data to be stored in the pre-allocated memory pool and the new memory pool includes: According to the remaining memory space, the incremental inverted index data to be stored is divided into first incremental inverted index data and second incremental inverted index data; wherein the required memory space of the first incremental inverted index data is equal to the remaining memory space; storing the first incremental inverted index data in the pre-allocated memory pool; The second incremental inverted index data is stored in the new memory pool.

3. The method according to claim 1, characterized in that The step of compressing the inverted index data in the memory pool to be compressed includes: For each inverted word in the inverted index data in the memory pool to be compressed, aggregating a document identification list of the inverted word to form a data block of the inverted word; The data block is compressed using a PForDelta algorithm.

4. A device for implementing incremental inverted index data storage, characterized in that: include: A required memory space determination module is used to determine the required memory space for storing incremental inverted index data; A judgment module, configured to judge whether the memory space required to store the incremental inverted index data is greater than the remaining memory space in the pre-allocated memory pool; The pre-allocated memory pool is a memory pool that pre-allocates a certain amount of memory space; the memory pool is IncSegment, and each IncSegment has a dictionary that records inverted words and the offset address of the inverted words in the BlockHeader of the IncSegment; A storage module is configured to apply for a new memory pool, store the incremental inverted index data to be stored in the pre-allocated memory pool and the new memory pool, mark the pre-allocated memory pool after the storage operation as a memory pool to be compressed, and send a message about the memory pool to be compressed to a reconstruction queue; storing the incremental inverted index data to be stored in the pre-allocated memory pool; A compression module is used to rebuild the memory pool to be compressed based on the consumed messages in the reconstruction queue; during the reconstruction process, compress the inverted index data in the memory pool to be compressed.

5. The device according to claim 4, characterized in that The storage module is also used to, according to the remaining memory space, divide the incremental inverted index data to be stored into first incremental inverted index data and second incremental inverted index data; wherein the required memory space of the first incremental inverted index data is equal to the remaining memory space; store the first incremental inverted index data in the pre-allocated memory pool; and store the second incremental inverted index data in the new memory pool.

6. The device according to claim 4, characterized in that The compression module is further configured to, for each inverted word in the inverted index data in the memory pool to be compressed, summarize a document identification list of the inverted word to form a data block of the inverted word; and compress the data block using a PForDelta algorithm.

7. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

8. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images