Decompression method, system, medium and electronic device of search engine

Abstract

The application provides a decompression method and system of a search engine, a medium and an electronic device, and relates to the technical field of decompression; the method comprises the following steps: when a target file is decompressed by a search engine, the target file is split into multiple sub-blocks; a first preset number of sub-areas are set in the memory, the sub-areas are independent of each other, and a dictionary for decompression is stored in each sub-area; the sub-blocks are decompressed based on a zstd algorithm and the dictionary stored in the sub-areas; the dictionary stored in each sub-area corresponds to the decompression of a sub-block; the application provides a brand-new compression algorithm based on the zstd algorithm, and the compression rate and speed are effectively improved compared with the existing LZ4 algorithm and Deflate algorithm, which greatly relieves the contradiction of the search engine in the storage of massive text data, thereby better meeting the needs of users.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of decompression, and particularly relates to a decompression method and system of a search engine, a medium and an electronic device. BACKGROUND

[0002] Full-text search engine technology and related products (Lucene) are widely used in big data scenarios such as log, time series storage and analysis, and these scenarios generally have the following characteristics:

[0003] 1) Large data size: PB level.

[0004] 2) Long storage period: 7-30 days.

[0005] 3) High cost of storage medium: SSD to meet high-performance reading requirements.

[0006] 4) Low query latency: millisecond-level query delay after storage.

[0007] With the development of user business, users hope to store more data and have a longer storage period. However, this will lead to a sharp increase in storage costs. In this case, conventional operation and maintenance methods are usually used, such as cleaning up old indexes, reducing the number of replicas, expanding, and using less expensive SATA disks or object storage to meet longer storage periods. This cannot fundamentally solve the contradiction of user storage costs and also brings additional operation and maintenance costs and risks.

[0008] The contradiction between cost and efficiency of storage is, from a bottom-up technical point of view, whether there is a way to store various types of data files more efficiently by using more efficient compression encoding. The representative search engine product (Lucene) in the industry generally uses either the LZ4 algorithm that sacrifices high compression ratio for faster compression speed or the Deflate algorithm that sacrifices decompression speed for extreme compression ratio. These two algorithms cannot meet the user's needs in terms of performance and storage cost.

[0009] LZ4 is a lossless compression algorithm that provides a compression speed of more than 500MB / s per core (more than 0.15 Bytes / cycle). Its characteristic is extremely fast decoding speed, with a speed of more than GB / s per core (about 1 Bytes / cycle). However, the compression ratio is not high, and the space after compression is 46% more than that of high-compression-ratio algorithms such as Deflate, which means more data transmission time and disk space occupation. Even the largest compression ratio is not high, only from 1.79 to 2.11, but the time consumption is from 5s to 51s.

[0010] Deflate is a data lossless compression algorithm, which is widely used in zip file compression and png image compression. Deflate combines huffman coding and LZ77 coding. Although he has a very high compression ratio, the performance is one order of magnitude slower than other fast compression algorithms, and the use of system CPU resources cannot be optimized. SUMMARY

[0011] In view of the above-mentioned disadvantages of the prior art, the purpose of the present application is to provide a search engine decompression method, system, medium and electronic equipment, which is used to solve the problem that the existing LZ4 compression algorithm and Deflate compression algorithm cannot meet the user's demand in terms of compression performance and storage cost.

[0012] To achieve the above-mentioned purpose and other related purposes, the present application provides a search engine decompression method, which comprises the following steps: when decompressing a target file of a search engine, the target file is split into a plurality of sub-blocks; a first preset number of sub-regions are set in the memory, and the dictionaries for decompression are stored in each sub-region; the sub-blocks are decompressed based on the zstd algorithm and the dictionaries stored in the sub-regions; the dictionaries stored in each sub-region correspond to the decompression of a sub-block.

[0013] In an embodiment of the present application, before the step of decompressing the sub-blocks based on the zstd algorithm and the dictionaries stored in the sub-regions, the method further comprises the following steps: extending the underlying storage compression encoding of the search engine using the zstd algorithm; after extending the underlying storage compression encoding of the search engine using the zstd algorithm, the compression algorithm corresponding to the underlying storage compression encoding at least includes: the zstd algorithm; when the compression algorithm also includes other algorithms in addition to the zstd algorithm, the priority of the zstd algorithm is the highest.

[0014] In an embodiment of the present application, the step of splitting the target file into a plurality of sub-blocks comprises the following steps: uniformly splitting the target file into a plurality of sub-blocks of a preset size; or splitting the target file into a second preset number of sub-blocks.

[0015] In an embodiment of the present application, the step of setting a first preset number of sub-regions in the memory comprises the following steps: based on the number of sub-blocks, a first preset number of sub-regions are set in the memory; the first preset number is not less than the number of sub-blocks.

[0016] In an embodiment of the present application, before the step of decompressing the sub-blocks based on the zstd algorithm and the dictionaries stored in the sub-regions, the method further comprises the steps of: judging whether the first preset number is less than the number of the sub-blocks; when the first preset number is less than the number of the sub-blocks, setting a third preset number of sub-regions in the memory independently of each other, and storing the dictionaries in each of the sub-regions respectively; and the sum of the third preset number and the first preset number is not less than the number of the sub-blocks.

[0017] In an embodiment of the present application, the step of decompressing the sub-blocks based on the zstd algorithm and the dictionaries stored in the sub-regions comprises the step of: decompressing each of the sub-blocks in parallel based on the zstd algorithm and the dictionaries stored in each of the sub-regions.

[0018] In an embodiment of the present application, the step of decompressing the sub-blocks based on the zstd algorithm and the dictionaries stored in the sub-regions comprises the steps of: when the dictionaries stored in the sub-regions cannot meet the decompression requirement for at least one of the sub-blocks, obtaining a new dictionary; decompressing the sub-blocks based on the zstd algorithm, the dictionaries stored in the sub-regions and the new dictionary; and the method further comprises the step of: updating the sub-regions by using the new dictionary.

[0019] The present application provides a decompression system of a search engine, comprising a splitting module, a setting module and a decompression module; the splitting module is used for splitting a target file into a plurality of sub-blocks when the target file is decompressed by the search engine; the setting module is used for setting a first preset number of sub-regions in the memory independently of each other, and storing dictionaries for decompression in each of the sub-regions; and the decompression module is used for decompressing the sub-blocks based on a zstd algorithm and the dictionaries stored in the sub-regions; and the dictionaries stored in each of the sub-regions correspond to decompression of one of the sub-blocks.

[0020] The present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the decompression method of the search engine when executing the computer program.

[0021] The present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the decompression method of the search engine when executed by a processor.

[0022] As described above, the decompression method, system, medium and electronic device of the search engine of the present application have the following beneficial effects:

[0023] (1) Compared with the prior art, the application provides a brand-new compression algorithm based on the zstd algorithm, and the compression rate and speed are effectively improved compared with the existing LZ4 algorithm and Deflate algorithm, greatly relieving the contradiction of search engines in mass text data storage, thereby better meeting the needs of users.

[0024] (2) The zstd algorithm provided by the application is the best in the comprehensive performance of compression speed and compression ratio compared with the existing LZ4 algorithm and Deflate algorithm, and the zstd algorithm can be said to take into account the advantages of both LZ4 and Deflate.

[0025] (3) The application develops a new Lucene codec compression framework, which is based on the C language lib of zstd at the bottom, and in the general framework layer of Lucene compression algorithm, the data to be decompressed is divided into multiple chunk sub-blocks, and then these sub-blocks are decompressed in parallel, so that the speed is doubled; for each sub-block, a separate sub-area is opened in the memory to store the dictionary used for decompression, and the dictionary supports continuous optimization and enhancement in training mode, so that in the continuous decompression process, based on the continuously enhanced dictionary content, the decompression speed will also become faster, thereby further improving the decompression performance.

[0026] (4) The application supports index-level decompression algorithm customization for the Lucene outer framework, such as Elasticsearch, and users only need to specify the new encoding method when creating an index to have zstd algorithm support. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A structural schematic diagram of a terminal of the application in an embodiment is shown.

[0028] Figure 2 A flowchart of a decompression method of a search engine of the application in an embodiment is shown.

[0029] Figure 3 A flowchart of decompression of a sub-block based on the zstd algorithm and the dictionary stored in the sub-area of the application in an embodiment is shown.

[0030] Figure 4 A schematic diagram of a sub-area of the application in an embodiment is shown.

[0031] Figure 5 A statistical comparison chart of the compression ratio and compression speed performance of various compression algorithms of the application in an embodiment is shown.

[0032] Figure 6A trend comparison chart showing compression ratio and compression speed performance of various compression algorithms of the present application in one embodiment.

[0033] Figure 7 A schematic diagram showing file storage of Lucene of the present application in one embodiment.

[0034] Figure 8 A schematic diagram showing Lucene base file format of the present application in one embodiment.

[0035] Figure 9 A schematic diagram showing test results of performance test on LZ4, Deflate and zstd algorithms of the present application in one embodiment.

[0036] Figure 10 A structural schematic diagram showing decompression system of search engine of the present application in one embodiment.

[0037] Label Explanation

[0038] 1 terminal

[0039] 11 processing unit

[0040] 12 memory

[0041] 121 random access memory

[0042] 122 cache memory

[0043] 123 storage system

[0044] 124 program / utility

[0045] 1241 program module

[0046] 13 bus

[0047] 14 input / output interface

[0048] 15 network adapter

[0049] 2 external device

[0050] 3 display

[0051] 101 splitting module

[0052] 102 setting module

[0053] 103 decompression module

[0054] S10-S30 steps

[0055] S301-S302 steps DETAILED DESCRIPTION

[0056] Following, the advantages and effects of the present application can be easily understood by those skilled in the art from the description. The present application can also be implemented or applied by different specific embodiments, and the details in the description can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0057] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and only the components related to the present application are shown in the diagrams, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be a random change, and the component layout may be more complex.

[0058] Compared with the prior art, the decompression method, system, medium and electronic device of the search engine of the present application provide a new compression algorithm based on the zstd algorithm, and the compression rate and speed are effectively improved compared with the existing LZ4 algorithm and Deflate algorithm, greatly relieving the contradiction of the search engine in the storage of massive text data, thereby better meeting the needs of users; the zstd algorithm provided by the present application is the best in the comprehensive performance of compression speed and compression ratio compared with the existing LZ4 algorithm and Deflate algorithm, and the zstd algorithm can be said to take into account the advantages of both LZ4 and Deflate; the present application develops a new Lucene codec compression framework, which is based on the C language lib of zstd at the bottom, and in the general framework layer of Lucene compression algorithm, the data to be decompressed is divided into multiple chunk sub-blocks, and then these sub-blocks are decompressed in parallel, so that the speed is doubled; for each sub-block, a separate sub-area is opened in the memory to store the dictionary used for decompression, and the dictionary supports continuous optimization and enhancement in training mode, so that in the continuous decompression process, based on the continuously strengthened dictionary content, the decompression speed will also become faster, thereby further improving the decompression performance; the present application supports customizing the decompression algorithm at the index level for the Lucene outer framework, such as Elasticsearch, and users only need to specify the new encoding method when creating an index to have zstd algorithm support.

[0059] The computer readable storage medium of the present application stores a computer program, which, when executed by a processor, implements the steps of the decompression method of the search engine described below. The storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk, a U disk, a memory card, or an optical disk, and various media that can store program codes.

[0060] Any combination of one or more storage media can be employed. The storage media can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0061] The computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, in which a computer-readable program code is carried. Such a propagated data signal can take on many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium that can send, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0062] The program code contained on the computer-readable medium can be transmitted in any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0063] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0064] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0065] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0066] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0067] The electronic device of the present application comprises a memory, a processor and a computer program stored on the memory and executable on the processor.

[0068] Preferably, the memory includes: ROM, RAM, disk, U disk, memory card or optical disc and various media that can store program codes.

[0069] The processor is connected with the memory, and is used for executing the computer program stored in the memory, so that the electronic device executes the steps of the decompression method of the search engine.

[0070] Preferably, the processor can be a general processor, including a central processing unit (CPU), a network processor (NP) and the like; and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0071] In an embodiment, the electronic device includes a terminal and / or a server.

[0072] Figure 1 A block diagram of an exemplary terminal 1 suitable for implementing embodiments of the present application is shown.

[0073] Figure 1 The terminal 1 shown is merely an example and should not limit the function and scope of use of embodiments of the present application.

[0074] As shown in Figure 1 The terminal 1 is in the form of a general computing device. The components of the terminal 1 can include, but are not limited to, one or more processors or processing units 11, a memory 12, and a bus 13 connecting different system components, including the memory 12 and the processing unit 11.

[0075] Bus 13 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0076] Terminal 1 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by Terminal 1, including volatile and non-volatile media, removable and non-removable media.

[0077] Memory 12 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 121 and / or cache memory 122. Terminal 1 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 123 may be used to read and write non-removable, non-volatile magnetic media (… Figure 1 Not shown; usually referred to as a "hard drive"). Although Figure 1 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 13 via one or more data media interfaces. Memory 12 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0078] A program / utility 124 having a set (at least one) of program modules 1241 may be stored, for example, in memory 12. Such program modules 1241 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 1241 typically perform the functions and / or methods described in the embodiments of the present invention.

[0079] Terminal 1 can also communicate with one or more external devices 2 (e.g., keyboard, pointing device, display 3, etc.), and with one or more devices that enable a user to interact with terminal 1, and / or with any device that enables terminal 1 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 14. Furthermore, terminal 1 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 15. Figure 1 As shown, network adapter 15 communicates with other modules of terminal 1 via bus 13. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with terminal 1, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0080] This invention provides a decompression method for a search engine, which effectively solves the problem that existing LZ4 compression algorithms and Deflate compression algorithms cannot meet user needs in terms of compression performance and storage cost.

[0081] like Figure 2 As shown, in one embodiment, the decompression method of the search engine of the present invention includes the following steps:

[0082] Step S10: When the search engine decompresses the target file, the target file is split into multiple sub-blocks.

[0083] In one embodiment, splitting the target file (i.e., the file to be decompressed) into multiple sub-blocks includes the following steps: splitting the target file into multiple sub-blocks according to a preset splitting rule.

[0084] It should be noted that the preset splitting rule is a pre-set file splitting rule. The specific content of the splitting rule is not a limitation of the present invention and can be set according to the specific application scenario.

[0085] In one embodiment, the preset splitting rule is set as follows: the target file is evenly split into multiple sub-blocks of a preset size.

[0086] Specifically, a preset size (e.g., 60K) is set in advance. When a target file needs to be decompressed, the target file is evenly divided into sub-blocks of the preset size, that is, the size of each sub-block is the preset size (e.g., ...). Figure 4 (As shown).

[0087] It should be noted that in this embodiment, the size of the sub-block is fixed, but for different target files, the size may be the same or different, and the number of sub-blocks split out for different target files will also be different.

[0088] For example, if one target file is 1000K in size and another target file is 2000K in size, with a preset size of 50K, then in this embodiment, the 1000K target file will be split into 20 sub-blocks, each of which is 50K in size; and the 2000K target file will be split into 40 sub-blocks, each of which is also 50K in size.

[0089] Furthermore, the preset size is pre-set, and its specific setting is not a limitation of the present invention. In practical applications, it can be set according to the specific application scenario.

[0090] In one embodiment, the preset splitting rule is set as follows: split the target file into a second preset number of sub-blocks.

[0091] Specifically, after obtaining a target file to be decompressed, the target file is split into a second preset number of sub-blocks.

[0092] Furthermore, in the second preset number of sub-blocks, each sub-block is of equal size.

[0093] It should be noted that the size of each sub-block in the second preset number of sub-blocks may also be different, depending on the setting of the preset splitting rule.

[0094] It should be noted that in this embodiment, the target file is split into a second preset number of sub-blocks. That is, after a target file is split into sub-blocks, the number of sub-blocks obtained is fixed (i.e., the second preset number). However, considering that different target files may have the same or different sizes, when target files of different sizes are split into the same number (the second preset number) of sub-blocks, the sizes of the corresponding sub-blocks are different.

[0095] For example, in one embodiment, one target file is 1000K in size and another target file is 2000K in size, with a preset number of 50, and each sub-block generated after splitting the two target files is of equal size. In this embodiment, the 1000K target file will be split into 50 sub-blocks, each of which is 20K in size; and the 2000K target file will be split into 50 sub-blocks, each of which is 40K in size.

[0096] It should be noted that the second preset quantity is pre-set, and its specific setting is not a limitation of the present invention. In practical applications, it can be set according to the specific application scenario.

[0097] Furthermore, the above only lists two preset splitting rules, but in actual applications, it is not limited to only these two preset splitting rules.

[0098] Step S20: Set a first preset number of independent sub-regions in memory, and store the dictionary for decompression in each of the sub-regions.

[0099] In one embodiment, setting a first preset number of mutually independent sub-regions in memory includes the following steps: setting a first preset number of mutually independent sub-regions in memory based on the number of sub-blocks.

[0100] It should be noted that, in this embodiment, the first preset number is not less than the number of sub-blocks, that is, the number of sub-regions is not less than the number of sub-blocks in step S10.

[0101] Specifically, for each sub-block in step S10, an independent sub-region is allocated in memory to store the dictionary used for decompression. Thus, in subsequent decompression processes, the corresponding sub-block is decompressed based on the dictionary stored in one sub-region (e.g., ...). Figure 4 (As shown).

[0102] It should be noted that the dictionary stored in each sub-region corresponds to the decompression of one sub-block. That is, in the subsequent decompression process, one sub-block corresponds to one sub-region, and each sub-block decompresses the corresponding sub-block by using the dictionary stored in a sub-region.

[0103] It should be noted that the dictionary content stored in each sub-region is the same.

[0104] In one embodiment, before the step of decompressing the sub-block based on the zstd algorithm and the dictionary stored in the sub-region, i.e. before the subsequent step S30, the method further includes the following steps:

[0105] Step 1: Determine whether the first preset quantity is less than the quantity of the sub-blocks;

[0106] Step 2: When the first preset number is less than the number of sub-blocks, set a third preset number of independent sub-regions in the memory, and store the dictionary in each of the sub-regions respectively.

[0107] It should be noted that the sum of the third preset quantity and the first preset quantity is not less than the number of sub-blocks.

[0108] Specifically, when the number of sub-regions set in memory in step S20 is less than the number of sub-blocks split from the target file in step S10, one or more new sub-regions can be set in memory directly. These new sub-regions are independent of the existing sub-regions in memory. Then, the dictionary content stored in one of the existing sub-regions in memory can be copied to the new sub-region.

[0109] It should be noted that the execution order of step S20 only needs to ensure that it is executed before the subsequent step S30, that is, it can be executed before step S10 or simultaneously with step S10.

[0110] Furthermore, when step S20 is executed after step S10, that is, when step S20 is executed, it is already known how many sub-blocks the target file has been split into by step S10. At this time, a corresponding number of sub-regions can be set based on the number of sub-blocks split in step S10. This ensures that the number of sub-regions set in step S20 meets the subsequent decompression requirements of the sub-blocks. Of course, if it is executed before step S10 or simultaneously with step S10, it means that before setting the sub-regions in memory in step S20, it is not known how many sub-blocks the target file to be decompressed has been split into by step S10. In this case, a certain number of sub-regions can be set first. After the number of sub-blocks is determined in step S10, if the number of sub-regions does not meet the requirements, new sub-regions can be set in memory.

[0111] It should be noted that executing step S20 before step S10 (or simultaneously with step S10) can effectively shorten the time of the entire decompression process. This means that the process of setting up sub-regions in memory in step S20 is excluded from the decompression process. By setting up sub-regions in memory in advance, the sub-blocks can be decompressed directly using the dictionary in the sub-regions during the decompression operation, without having to set up sub-regions during the decompression process.

[0112] Step S30: Decompress the sub-block based on the zstd algorithm and the dictionary stored in the sub-region.

[0113] Currently, search engine products, such as Lucene, only support two compression algorithms for their underlying storage: LZ4 compression, which has fast compression characteristics, and Deflate compression, which has a high compression ratio. The background section introduced the shortcomings and deficiencies of these two algorithms. Therefore, a third compression algorithm is needed to address these shortcomings and deficiencies, and this third compression algorithm should be able to demonstrate good performance in both compression speed and compression ratio.

[0114] After reviewing all current compression algorithms in the industry and conducting extensive performance benchmark tests, this invention proposes the zstd algorithm. The zstd algorithm boasts the best overall performance in terms of compression speed and compression ratio. It can be said that the zstd algorithm combines the advantages of both LZ4 and Deflate: its performance is comparable to LZ4 (in tests on log data, the zstd algorithm was found to be slightly better than LZ4), while its compression ratio is slightly weaker than Deflate (see details). Figure 5 and Figure 6 ).

[0115] It should be noted that the zstd (full name zstandard) algorithm is a new lossless compression algorithm open-sourced by Facebook in 2016. It is a fast, real-time lossless data compression algorithm and tool that provides a high compression ratio developed by Facebook. The algorithm is designed for real-time compression and high compression ratio scenarios at the zlib level. The underlying algorithm is based on the Huff0 and FSE libraries. Compared with other compression algorithms in the industry (such as Gzip, Snappy, and Zlib), its characteristic is that it can exchange compression speed for a higher compression ratio when needed (the trade-off between compression speed and compression ratio can be configured in small increments).

[0116] Specifically, after splitting the target file to be decompressed into multiple sub-blocks in step S10, the multiple sub-blocks are decompressed based on the zstd algorithm and the dictionary in step S20.

[0117] It should be noted that the decompression of the file to be decompressed (referring to the sub-blocks) using a compression algorithm (here referring to the zstd algorithm) and a dictionary is a conventional technique in the field. The specific decompression principle is not a limitation of this invention, and therefore will not be described in detail here.

[0118] In one embodiment, the decompression of the sub-block based on the zstd algorithm and the dictionary stored in the sub-region includes the following steps: decompressing each sub-block in parallel based on the zstd algorithm and the dictionary stored in each sub-region.

[0119] Specifically, when decompressing the multiple sub-blocks, the multiple sub-blocks are decompressed simultaneously in parallel, thus changing the original process of decompressing the entire target file to decompressing each sub-block; in terms of time, the time spent decompressing the entire target file is reduced to the time spent decompressing a single sub-block.

[0120] It should be noted that by decompressing these multiple sub-blocks in parallel, the time spent on decompression is greatly reduced compared to the existing method of decompressing the entire target file, thus effectively improving the decompression speed.

[0121] In one embodiment, before the step of decompressing the sub-block based on the zstd algorithm and the dictionary stored in the sub-region, i.e. before step S30, the method further includes the step of extending the underlying storage compression encoding of the search engine using the zstd algorithm.

[0122] Specifically, the zstd algorithm is introduced into the underlying storage compression algorithm framework of the search engine.

[0123] Taking Lucene as an example, if a new zstd algorithm can be implemented in Lucene's underlying storage compression encoding, it will bring comprehensive improvements in both storage cost and storage performance. Figure 7 As shown, the biggest overhead in Lucene's file storage is in the index files, which account for more than 80% of the storage.

[0124] In one embodiment, the underlying storage compression encoding includes, but is not limited to, an index file.

[0125] like Figure 7 As shown, in one embodiment, the index file includes at least, but is not limited to, any one of the following: row storage file (fdt), index dictionary file (tim), and column storage file (dvd).

[0126] In this embodiment, by extending the underlying storage compression encoding of the search engine using the zstd algorithm, the relevant compression encoding frameworks of row-based files, index dictionary files, and column-based files are reimplemented using the zstd algorithm, enabling the underlying storage compression encoding of the search engine to have zstd algorithm functionality, thereby effectively improving the storage performance of the search engine and reducing its storage costs.

[0127] It should be noted that after extending the underlying storage compression encoding of the search engine using the zstd algorithm, the compression algorithm corresponding to the underlying storage compression encoding includes, but is not limited to, the zstd algorithm (e.g., ...). Figure 8 (As shown).

[0128] Furthermore, when the compression algorithm includes other algorithms besides the zstd algorithm, the zstd algorithm has the highest priority.

[0129] like Figure 8 As shown, in one embodiment, when the underlying general compression algorithm also includes the LZ4 algorithm and / or the Deflate algorithm (but is not limited to these two algorithms), the zstd algorithm has a higher priority than the LZ4 algorithm and / or the Deflate algorithm. That is, when a decompression operation needs to be performed, the zstd algorithm is used first to perform the decompression operation.

[0130] like Figure 3 As shown, in one embodiment, the decompression of the sub-block based on the zstd algorithm and the dictionary stored in the sub-region includes the following steps:

[0131] Step S301: When the dictionary stored in the sub-region cannot meet the decompression requirements for at least one of the sub-blocks, obtain a new dictionary.

[0132] It should be noted that for different target files, after being split into multiple sub-blocks in step S10, the content of each sub-block is also different. Therefore, when decompressing each sub-block separately using the dictionary stored in the sub-region in step S30, the required dictionary content may also be different. At this time, when decompressing one sub-block, the dictionary content in the corresponding sub-region may not meet the decompression requirements of that sub-block. Therefore, it is necessary to obtain new dictionary content to achieve the decompression of that sub-block.

[0133] In one embodiment, the server stores a large number of dictionaries required for decompression. When the decompression process in step S30 requires a new dictionary (i.e., the dictionary stored in the sub-region cannot meet the decompression requirements for at least one of the sub-blocks), the required dictionary can be downloaded from the server.

[0134] It should be noted that obtaining a new dictionary from the server is only one implementation method for obtaining a new dictionary and is not intended to limit the scope of the present invention.

[0135] Step S302: Decompress the sub-block based on the zstd algorithm, the dictionary stored in the sub-region, and the new dictionary.

[0136] Specifically, based on the zstd algorithm described above, the dictionary stored in the sub-region in step S20, and the new dictionary obtained in step S301, each sub-block in step S10 is decompressed.

[0137] In one embodiment, the method further includes the step of updating the sub-region using the new dictionary.

[0138] Specifically, after obtaining the new dictionary in step S301, the sub-region in step S20 is updated using the new dictionary.

[0139] It should be noted that the dictionary stored in the sub-region in step S20 above can be continuously optimized and enhanced in training mode. That is, in the continuous decompression process, the dictionary stored in the sub-region is continuously updated to strengthen the dictionary content stored in the sub-region, thereby making the subsequent decompression speed faster and faster.

[0140] It should be noted that after a decompression process, once new dictionary content is obtained in step S301, the dictionary content stored in the sub-region in step S20 is updated based on step S302. If no new dictionary content is obtained after a decompression process, the above update operation is not required.

[0141] Furthermore, assuming that new dictionary content is obtained in step S301 during this decompression process, i.e., the above update operation needs to be performed, after this update, in the next decompression process (corresponding to step S30), the sub-block will be decompressed based on the updated dictionary in the sub-region.

[0142] The following specific embodiments further verify the decompression method of the search engine of the present invention.

[0143] In one embodiment, with the support of the new zstd algorithm framework, performance tests were conducted on three compression algorithms (LZ4, Deflate, and zstd) (the tests were based on a massive log storage and retrieval scenario). The test results are as follows: Figure 9 As shown in the image.

[0144] Depend on Figure 9 It can be seen that the compression ratio of the zstd algorithm is 35% higher than that of LZ4, and its performance is comparable to that of LZ4; the speed of the zstd algorithm is more than 20% higher than that of the Deflate algorithm. The new zstd algorithm in this solution greatly alleviates the contradiction of search engines in storing massive amounts of text data and better meets the needs of users.

[0145] It should be noted that the scope of protection of the decompression method of the search engine described in this invention is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this invention is included within the scope of protection of this invention.

[0146] like Figure 10 As shown, in one embodiment, the decompression system of the search engine of the present invention includes a splitting module 101, a setting module 102, and a decompression module 103.

[0147] The splitting module 101 is used to split the target file into multiple sub-blocks when the search engine decompresses the target file.

[0148] The setting module 102 is used to set a first preset number of independent sub-regions in memory, and to store the dictionary for decompression in each of the sub-regions.

[0149] The decompression module 103 is used to decompress the sub-block based on the zstd algorithm and the dictionary stored in the sub-region; each dictionary stored in the sub-region corresponds to the decompression of one sub-block.

[0150] It should be noted that the structure and principle of the splitting module 101, the setting module 102 and the decompression module 103 correspond one-to-one with the steps (steps S10 to S30) in the decompression method of the above-mentioned search engine, so they will not be described again here.

[0151] It should be noted that the division of the various modules in the above system is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software through processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, module x can be a separate processing element, or it can be integrated into a chip within the system. Alternatively, it can be stored as program code in the system's memory, and its function can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0152] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a System-On-a-Chip (SOC).

[0153] It should be noted that the decompression system of the search engine of the present invention can implement the decompression method of the search engine of the present invention, but the implementation device of the decompression method of the search engine of the present invention includes, but is not limited to, the structure of the decompression system of the search engine listed in this embodiment. All structural modifications and substitutions of the prior art made according to the principle of the present invention are included within the protection scope of the present invention.

[0154] In summary, the decompression method, system, medium, and electronic device of the search engine of this invention, compared with the prior art, provide a novel compression algorithm based on the zstd algorithm. Its compression ratio and speed are significantly improved compared to the existing LZ4 and Deflate algorithms, greatly alleviating the contradiction of search engines storing massive amounts of text data, thus better meeting user needs. The zstd algorithm provided by this invention outperforms the existing LZ4 and Deflate algorithms in terms of overall performance in compression speed and compression ratio, combining the advantages of both LZ4 and Deflate. This invention also develops a new Lucene... The codec compression framework, based on the zstd C language library, improves speed significantly by splitting the data to be decompressed into multiple chunks and decompressing them in parallel at the Lucene compression algorithm framework level. For each chunk, a dedicated memory area stores the dictionary used for decompression. This dictionary can be continuously optimized and improved through training, resulting in increasingly faster decompression speeds during continuous decompression, further enhancing performance. This invention also supports custom decompression algorithms at the index level for Lucene outer frameworks such as Elasticsearch. Users only need to specify a new encoding method when creating the index to have zstd algorithm support. Therefore, this invention effectively overcomes the shortcomings of existing technologies and has high industrial application value.

[0155] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A decompression method for a search engine, characterized in that, Includes the following steps: When the search engine decompresses the target file, it splits the target file into multiple sub-blocks; A first preset number of independent sub-regions are set in memory, and a dictionary for decompression is stored in each of the sub-regions; Determine whether the first preset quantity is less than the number of sub-blocks; When the first preset number is less than the number of sub-blocks, a third preset number of independent sub-regions are set in the memory, and the dictionary is stored in each of the sub-regions respectively; the sum of the third preset number and the first preset number is not less than the number of sub-blocks; Based on the zstd algorithm and the dictionary stored in each sub-region, each sub-block is decompressed in parallel; the dictionary stored in each sub-region corresponds to the decompression of one sub-block. If the dictionary stored in the sub-region cannot meet the decompression requirements for at least one of the sub-blocks, a new dictionary is obtained; The sub-block is decompressed based on the zstd algorithm, the dictionary stored in the sub-region, and the new dictionary; the sub-region is then updated using the new dictionary.

2. The decompression method for a search engine according to claim 1, characterized in that, Prior to the step of decompressing the sub-block based on the zstd algorithm and a dictionary stored in the sub-region, the method further includes the following steps: The underlying storage compression encoding of the search engine is extended using the zstd algorithm; after extending the underlying storage compression encoding of the search engine using the zstd algorithm, the compression algorithm corresponding to the underlying storage compression encoding includes at least the zstd algorithm; when the compression algorithm also includes other algorithms besides the zstd algorithm, the zstd algorithm has the highest priority.

3. The decompression method for a search engine according to claim 1, characterized in that, The step of splitting the target file into multiple sub-blocks includes the following steps: splitting the target file evenly into multiple sub-blocks of a preset size; or splitting the target file into a second preset number of sub-blocks.

4. A decompression system for a search engine, characterized in that, include: Split module, settings module, and decompression module; The splitting module is used to split the target file into multiple sub-blocks when the search engine decompresses the target file; The setting module is used to set a first preset number of independent sub-regions in memory, store the dictionary for decompression in each of the sub-regions, and determine whether the first preset number is less than the number of sub-blocks; when the first preset number is less than the number of sub-blocks, a third preset number of independent sub-regions is set in memory, and the dictionary is stored in each of the sub-regions respectively; the sum of the third preset number and the first preset number is not less than the number of sub-blocks; The decompression module is used to decompress each sub-block in parallel based on the zstd algorithm and the dictionary stored in each sub-region; each dictionary stored in each sub-region decompresses one sub-block; when the dictionary stored in the sub-region cannot meet the decompression requirements for at least one sub-block, a new dictionary is obtained; the sub-block is decompressed based on the zstd algorithm, the dictionary stored in the sub-region, and the new dictionary; and the sub-region is updated using the new dictionary.

5. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the steps of the decompression method of the search engine according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the decompression method of the search engine according to any one of claims 1 to 3.

Images