Massive data retrieval method, system, terminal and storage medium based on HASH mapping

Abstract

The present invention provides a massive data retrieval method, system, terminal and storage medium based on HASH mapping, including: decomposing cache layer data into data segments in advance in the data storage stage, calculating characteristic hash value combinations of data segments using a hash algorithm group, and saving characteristic hash value combinations of all data segments to a filtering layer; extracting keywords from user request data, and calculating hash value combinations of each keyword using the hash algorithm group; searching for matching data segments of keywords from the filtering layer by matching hash value combinations; if all keywords of the user request data have matching data segments, querying the requested data from the cache layer, otherwise querying the requested data from the persistence layer. The present invention filters a large number of invalid search fields, protects the stability and efficiency of the service cluster, and improves the retrieval efficiency of massive data. It effectively avoids server performance degradation and downtime caused by cache penetration and cache breakdown.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a mass data retrieval method, system, terminal and storage medium based on HASH mapping. Background Art

[0002] In the current era of big data, enterprise-level data is often measured in TB or even PB. With such a large amount of data, data retrieval has undoubtedly become a big problem. Especially in high-concurrency scenarios, the difficulty of efficient data retrieval increases exponentially. In actual enterprise application scenarios, a common storage solution is REDIS+MYSQL. When storing data, first persist the data to the MYSQL database (or other persistence layer database products), and then synchronize the hot data to the REDIS cache layer. When retrieving data, first access the REDIS cache layer. If the cache layer has the required data, just retrieve it and return it; if the cache layer does not have the data to be retrieved, cache penetration occurs at this time, and the retrieval request passes through the REDIS cache layer and directly accesses the persistence layer database to perform IO operations. If the persistence layer does not have the data to be retrieved, cache penetration occurs at this time. In high-concurrency scenarios, too many invalid queries cause a large number of IO operations to go directly to the database, which will cause the database performance to drop sharply or even crash. Summary of the invention

[0003] In view of the problem in the prior art that too many invalid queries lead to a large number of IO operations directly reaching the database, causing database performance to degrade or even crash, the present invention provides a massive data retrieval method, system, terminal and storage medium based on HASH mapping to solve the above technical problem.

[0004] In a first aspect, the present invention provides a method for retrieving massive data based on HASH mapping, comprising:

[0005] In advance, in the data storage stage, the cache layer data is decomposed into data segments, the characteristic hash value combinations of the data segments are calculated using the hash algorithm group, and the characteristic hash value combinations of all data segments are saved to the filtering layer;

[0006] Extracting keywords from the user request data, and calculating the hash value combination of each keyword using the hash algorithm group;

[0007] Find the matching data fragment of the keyword from the filtering layer by matching the hash value combination;

[0008] If there are matching data fragments for all the keywords in the user's requested data, the requested data is queried from the cache layer, otherwise the requested data is queried from the persistence layer.

[0009] Furthermore, the cache layer data is decomposed into data segments in advance in the data storage stage, the characteristic hash value combinations of the data segments are calculated using the hash algorithm group, and the characteristic hash value combinations of all data segments are saved to the filtering layer, including:

[0010] After decomposing each piece of data in the cache layer into data fragments, all data fragments are deduplicated to obtain a data fragment set;

[0011] The data fragments are calculated using each hash algorithm in the hash algorithm group to obtain multiple characteristic hash values, and the multiple characteristic hash values ​​are saved as a characteristic hash value combination of the data fragment.

[0012] Furthermore, the hash algorithm group includes a left shift operation, a right shift operation and an information digest operation.

[0013] Further, keyword extraction is performed on the user request data, and a hash value combination of each keyword is calculated using the hash algorithm group, including:

[0014] Extract keywords from the request data sent by the user using an unsupervised keyword extraction method;

[0015] Multiple hash value calculation threads are established, each thread uses each hash algorithm in the hash algorithm group to calculate the keyword to obtain multiple hash values, and saves the multiple feature hash values ​​as the hash value combination of the keyword.

[0016] Further, the matching data segments of the keywords are searched from the filtering layer by matching the hash value combination, including:

[0017] Determine whether there is a feature hash value combination that is the same as the keyword hash value combination in the filter layer:

[0018] If yes, it is determined that the keyword has a matching data segment in the filtering layer;

[0019] If not, it is determined that there is no matching data segment for the keyword in the filtering layer.

[0020] In a second aspect, the present invention provides a massive data retrieval system based on HASH mapping, comprising:

[0021] A data decomposition unit, used to decompose cache layer data into data segments in advance in the data storage stage, calculate characteristic hash value combinations of the data segments using a hash algorithm group, and save the characteristic hash value combinations of all data segments to the filtering layer;

[0022] A request decomposition unit, used to extract keywords from user request data and calculate a hash value combination of each keyword using the hash algorithm group;

[0023] A hash matching unit, used to search for matching data segments of keywords from the filtering layer by matching hash value combinations;

[0024] The data query unit is used to query the requested data from the cache layer if there are matching data fragments for all keywords of the user's requested data, and query the requested data from the persistence layer otherwise.

[0025] Furthermore, the data decomposition unit is used to:

[0026] After decomposing each piece of data in the cache layer into data fragments, all data fragments are deduplicated to obtain a data fragment set;

[0027] The data fragments are calculated using each hash algorithm in the hash algorithm group to obtain multiple characteristic hash values, and the multiple characteristic hash values ​​are saved as a characteristic hash value combination of the data fragment.

[0028] Furthermore, the hash algorithm group includes a left shift operation, a right shift operation and an information digest operation.

[0029] Furthermore, the request decomposition unit is used to:

[0030] Extract keywords from the request data sent by the user using an unsupervised keyword extraction method;

[0031] Multiple hash value calculation threads are established, each thread uses each hash algorithm in the hash algorithm group to calculate the keyword to obtain multiple hash values, and saves the multiple feature hash values ​​as the hash value combination of the keyword.

[0032] Furthermore, the hash matching unit is used to:

[0033] Determine whether there is a feature hash value combination that is the same as the keyword hash value combination in the filter layer:

[0034] If yes, it is determined that the keyword has a matching data segment in the filtering layer;

[0035] If not, it is determined that there is no matching data segment for the keyword in the filtering layer.

[0036] In a third aspect, a terminal is provided, including:

[0037] processor, memory, wherein:

[0038] The memory is used to store computer programs.

[0039] The processor is used to call and run the computer program from the memory, so that the terminal executes the above-mentioned terminal method.

[0040] According to a fourth aspect, a computer storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the methods described in the above aspects.

[0041] The beneficial effects of the present invention are:

[0042] The massive data retrieval method, system, terminal and storage medium based on HASH mapping provided by the present invention realizes pre-storage of features of the cache layer by decomposing the cache layer data into data fragments in advance in the data storage stage, calculating the characteristic hash value combination of the data fragments using a hash algorithm group, and saving the characteristic hash value combination of all data fragments to the filtering layer; then extracting keywords from the user request data, and calculating the hash value combination of each keyword using a hash algorithm group; searching for matching data fragments of keywords from the filtering layer by matching hash value combinations to realize fragment-based retrieval of the user request data, which improves the operation efficiency; if all keywords of the user request data have matching data fragments, query the request data from the cache layer, otherwise query the request data from the persistence layer. The present invention filters a large number of invalid search fields, protects the stability and efficiency of the service cluster, and improves the retrieval efficiency of massive data. It effectively avoids server performance degradation and downtime caused by cache penetration and cache breakdown.

[0043] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0046] Figure 2 is a schematic block diagram of a system according to an embodiment of the present invention.

[0047] Figure 3 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0049] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject may be a massive data retrieval system based on HASH mapping.

[0050] like Figure 1 As shown, the method includes:

[0051] Step 110, decomposing the cache layer data into data segments in advance in the data storage stage, calculating characteristic hash value combinations of the data segments using a hash algorithm group, and saving the characteristic hash value combinations of all data segments to the filtering layer;

[0052] Step 120, extracting keywords from the user request data, and calculating the hash value combination of each keyword using the hash algorithm group;

[0053] Step 130, searching for matching data segments of the keywords from the filtering layer by matching the hash value combination;

[0054] Step 140: If all keywords of the user's requested data have matching data fragments, the requested data is queried from the cache layer; otherwise, the requested data is queried from the persistence layer.

[0055] To facilitate understanding of the present invention, the following is a further description of the massive data retrieval method based on HASH mapping provided by the present invention based on the principle of the massive data retrieval method based on HASH mapping of the present invention, combined with the process of retrieving massive data based on HASH mapping in the embodiment.

[0056] In the general data retrieval process, some data structures or in-memory databases are often used to store commonly used query fields to reduce the number of database IO. The typical solution is to use the REDIS database as a cache, or directly use a hash table (HASH TABLE), such as HashMap in JAVA. This can be done when the amount of data is small. If the amount of data is hundreds of millions or even billions, the memory occupied will often reach an alarming level. However, if the memory space allocated to REDIS or the cache structure is too small, it is easy to cause cache breakdown in high-concurrency retrieval scenarios, and a large number of requests will directly reach the database, thereby generating a large amount of IO and affecting database performance.

[0057] In order to solve this problem, the present invention proposes the following core solutions:

[0058] 1. Preprocessing of data before storage

[0059] 2. Retrieval field preprocessing

[0060] 3. Filter invalid searches

[0061] Specifically, the massive data retrieval method based on HASH mapping includes:

[0062] S1. Decompose the cache layer data into data segments in advance during the data storage phase, calculate the characteristic hash value combinations of the data segments using a hash algorithm group, and save the characteristic hash value combinations of all data segments to the filtering layer.

[0063] After decomposing each piece of data in the cache layer into data fragments, all data fragments are deduplicated to obtain a data fragment set; each hash algorithm in the hash algorithm group is used to calculate the data fragments to obtain multiple feature hash values, and the multiple feature hash values ​​are saved as a feature hash value combination of the data fragments.

[0064] When storing data, the original data is directly stored in the persistence layer database, and the data is synchronized to the cache layer. When the cache layer receives the newly stored data, it decomposes the new data into segments, pre-processes the original data to be stored, and performs word segmentation on the data to form multiple data segments. For example, if the original data is "I love China", after word segmentation, the data segments formed are ["I", "love", "China"]. Then the feature hash value combination of each data segment is calculated. The feature hash value combination of the newly added data is saved to the filter layer.

[0065] The hash algorithm group includes a left shift operation, a right shift operation, and a message digest operation.

[0066] Take the right shift operation as an example. Create a bit array of size 10 [0,0,0,0,0,0,0,0,0,0], and use a very simple HASH algorithm - bit operation to shift the keyword left or right. For example, right shift operation: 'a'>>2, the result is 24, and the modulus operation 24%10 is performed, and the position of the hash value in the bit array is 4. Set the fourth position to 1, that is, [0,0,0,0,0,1,0,0,0,0]. In this way, the character 'a' is stored in the filter.

[0067] The HASH filter implemented in this way has a certain false alarm rate. It is possible that the filter has stored quite a lot of elements, that is, many bits are already set to 1, and then an element that has never been stored in the filter, such as 'mysql', is queried. After k HASH calculations, it is found that the mapped bits are all set to 1, and a false alarm is generated. That is, a query for an element that does not exist returns 'exists'.

[0068] The larger the length m of the filter, the more hash functions k are selected, and the lower the false alarm rate will be, but the false alarm rate will still increase with the number of elements n stored in the filter. But please note that if the filter returns non-existence, then the element must not exist.

[0069] The relationship between the false alarm rate p, the number of selected HASH functions k, the filter size m, and the number of elements n in the filter is as follows:

[0070]

[0071] According to the above formula, the selection formula of the number of HASH functions k can be derived:

[0072]

[0073] And the formula for determining the filter size m:

[0074]

[0075] Assume that you want to store 100 million 8-byte HASH strings. When the error rate p = 0.01 and the number of storage elements n = 10 8 , from the above formula, we can calculate m≈2.21×10 8 At this time, the filter occupies a space of 2.21×10 8 ÷1024÷1024÷8≈26MB, and if HashMap in JAVA is used for storage, it will take up 8kb×10 8 ÷1024×2≈1.6GB of storage space, and as the number of storage elements in the hash table increases, the query efficiency will gradually decrease, while the query efficiency of the filter is always O(1).

[0076] Therefore, considering the false alarm rate of the hash algorithm, this embodiment sets multiple hash algorithms to effectively avoid the false alarm problem.

[0077] S2. Extract keywords from the user request data, and calculate the hash value combination of each keyword using the hash algorithm group.

[0078] An unsupervised keyword extraction method is used to extract keywords from request data sent by the user; multiple hash value calculation threads are established, each thread uses each hash algorithm in the hash algorithm group to calculate the keyword to obtain multiple hash values, and the multiple feature hash values ​​are saved as a hash value combination of the keyword.

[0079] The method for calculating the hash value combination of keywords refers to step S1. In this embodiment, a thread is created for each keyword through multi-threaded synchronous calculation, thereby improving the request data processing efficiency.

[0080] S3. Search for matching data segments of keywords from the filtering layer by matching hash value combinations.

[0081] Determine whether there is a characteristic hash value combination that is the same as the keyword hash value combination in the filtering layer: if so, determine that the keyword has a matching data segment in the filtering layer; if not, determine that the keyword does not have a matching data segment in the filtering layer.

[0082] S4. If there are matching data fragments for all keywords in the user's requested data, the requested data is queried from the cache layer, otherwise the requested data is queried from the persistence layer.

[0083] The specific user retrieval request processing flow is as follows:

[0084] 1) When the search request is sent to the server, the server first performs word segmentation on the search field, splits the complete search statement into keyword groups, and then loops the keyword groups, using the filter's k HASH algorithm to calculate and map each keyword. If any of the corresponding bits in the bit array is 0, then we determine that the keyword does not exist in the database. When all keywords are cycled through and do not exist, the search ends and returns a null value. Otherwise, it is determined that the keyword exists and the search request will enter the REDIS cache layer.

[0085] 2) When any keyword in the previous step is hashed k times and the mapping value is present in the bit array, that is, when all the k bits are 1, the filter will allow the search to request the REDIS cache. If the result is found in the REDIS cache, the result will be directly returned and displayed. Otherwise, the search request will enter the persistence layer and query the database directly.

[0086] 3) When the REDIS cache fails to find the result in the previous step, the persistence layer database is requested to perform the actual query and return the query result. At this point, the entire query ends.

[0087] like Figure 2 As shown, the system 200 includes:

[0088] A data decomposition unit 210 is used to decompose the cache layer data into data segments in advance in the data storage stage, calculate the characteristic hash value combination of the data segments by using a hash algorithm group, and save the characteristic hash value combination of all data segments to the filtering layer;

[0089] The request decomposition unit 220 is used to extract keywords from the user request data and calculate the hash value combination of each keyword using the hash algorithm group;

[0090] A hash matching unit 230, configured to search for matching data segments of keywords from the filtering layer by matching hash value combinations;

[0091] The data query unit 240 is used to query the requested data from the cache layer if all keywords of the user's requested data have matching data fragments, otherwise query the requested data from the persistence layer.

[0092] Optionally, as an embodiment of the present invention, the data decomposition unit is used to:

[0093] After decomposing each piece of data in the cache layer into data fragments, all data fragments are deduplicated to obtain a data fragment set;

[0094] The data fragments are calculated using each hash algorithm in the hash algorithm group to obtain multiple characteristic hash values, and the multiple characteristic hash values ​​are saved as a characteristic hash value combination of the data fragment.

[0095] Optionally, as an embodiment of the present invention, the hash algorithm group includes a left shift operation, a right shift operation and an information digest operation.

[0096] Optionally, as an embodiment of the present invention, the request decomposition unit is used to:

[0097] Extract keywords from the request data sent by the user using an unsupervised keyword extraction method;

[0098] Multiple hash value calculation threads are established, each thread uses each hash algorithm in the hash algorithm group to calculate the keyword to obtain multiple hash values, and saves the multiple feature hash values ​​as the hash value combination of the keyword.

[0099] Optionally, as an embodiment of the present invention, the hash matching unit is used to:

[0100] Determine whether there is a feature hash value combination that is the same as the keyword hash value combination in the filter layer:

[0101] If yes, it is determined that the keyword has a matching data segment in the filtering layer;

[0102] If not, it is determined that there is no matching data segment for the keyword in the filtering layer.

[0103] Figure 3 A schematic diagram of the structure of a terminal 300 provided in an embodiment of the present invention. The terminal 300 can be used to execute the massive data retrieval method based on HASH mapping provided in an embodiment of the present invention.

[0104] The terminal 300 may include: a processor 310, a memory 320 and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention, and it may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0105] The memory 320 can be used to store the execution instructions of the processor 310, and the memory 320 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 can perform some or all of the steps in the following method embodiments.

[0106] The processor 310 is the control center of the storage terminal, and uses various interfaces and lines to connect various parts of the entire electronic terminal. It runs or executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of a plurality of packaged ICs with the same or different functions. For example, the processor 310 can include only a central processing unit (CPU). In the embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0107] The communication unit 330 is used to establish a communication channel so that the storage terminal can communicate with other terminals, receive user data sent by other terminals or send user data to other terminals.

[0108] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment provided by the present invention. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0109] Therefore, the present invention pre-decomposes the cache layer data into data fragments in the data storage stage, calculates the characteristic hash value combination of the data fragments using a hash algorithm group, and saves the characteristic hash value combination of all data fragments to the filter layer to achieve pre-storage of the cache layer features; then extracts keywords from the user request data, and calculates the hash value combination of each keyword using a hash algorithm group; searches for matching data fragments of keywords from the filter layer by matching hash value combinations to achieve fragmentary retrieval of user request data, which improves the computing efficiency; if all keywords of the user request data have matching data fragments, query the request data from the cache layer, otherwise query the request data from the persistence layer. The present invention filters a large number of invalid search fields, protects the stability and efficiency of the service cluster, and improves the retrieval efficiency of massive data. It effectively avoids server performance degradation and downtime caused by cache penetration and cache breakdown. The technical effects that can be achieved by this embodiment can be found in the description above and will not be repeated here.

[0110] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, including several instructions for enabling a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention.

[0111] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0112] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0115] Although the present invention has been described in detail by referring to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, and all of these shall be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A massive data retrieval method based on HASH mapping, It is characterized in that include: In advance, in the data storage stage, the cache layer data is decomposed into data segments, the characteristic hash value combinations of the data segments are calculated using the hash algorithm group, and the characteristic hash value combinations of all data segments are saved to the filtering layer; Extracting keywords from the user request data, and calculating the hash value combination of each keyword using the hash algorithm group; Find the matching data fragment of the keyword from the filtering layer by matching the hash value combination; If all keywords in the user's requested data have matching data fragments, the requested data is queried from the cache layer, otherwise the requested data is queried from the persistence layer; In the data storage stage, the cache layer data is decomposed into data segments in advance, the characteristic hash value combination of the data segments is calculated using the hash algorithm group, and the characteristic hash value combination of all data segments is saved to the filtering layer, including: After decomposing each piece of data in the cache layer into data fragments, all data fragments are deduplicated to obtain a data fragment set; Using each hash algorithm in the hash algorithm group to calculate the data fragments respectively to obtain multiple characteristic hash values, and saving the multiple characteristic hash values ​​as a characteristic hash value combination of the data fragments; Extracting keywords from the user request data and calculating the hash value combination of each keyword using the hash algorithm group includes: Extract keywords from the request data sent by the user using an unsupervised keyword extraction method; Establish multiple hash value calculation threads, each thread uses each hash algorithm in the hash algorithm group to calculate the keyword to obtain multiple hash values, and save the multiple feature hash values ​​as a hash value combination of the keyword; The matching data segments of keywords are found from the filtering layer by matching hash value combinations, including: Determine whether there is a feature hash value combination that is the same as the keyword hash value combination in the filter layer: If yes, it is determined that the keyword has a matching data segment in the filtering layer; If not, it is determined that the keyword does not have a matching data segment in the filtering layer; The specific user retrieval request processing flow is as follows: When the search request is sent to the server, the server first performs word segmentation on the search field, splits the complete search statement into keyword groups, and then loops the keyword groups, using the filter's k HASH algorithm to calculate and map each keyword. If any of the corresponding bits in the bit array is 0, then we determine that the keyword does not exist in the database. When all keywords are looped through and none of them exist, the search ends and returns a null value; otherwise, it is determined that the keyword exists and the search request will enter the REDIS cache layer; When any keyword in the previous step is hashed k times and the mapping value is present in the bit array, that is, when all the k bits are 1, the filter will allow the search to request the REDIS cache. If the result is found in the REDIS cache, the result will be directly returned and displayed; otherwise, the search request will enter the persistence layer and query the database directly. When the REDIS cache fails to find the result in the previous step, the persistence layer database is requested to perform the actual query and return the query result. At this point, the entire query ends. The hash algorithm group includes a left shift operation, a right shift operation and an information digest operation.

2. A system based on the massive data retrieval method based on HASH mapping as shown in claim 1, It is characterized in that include: A data decomposition unit, used to decompose cache layer data into data segments in advance in the data storage stage, calculate characteristic hash value combinations of the data segments using a hash algorithm group, and save the characteristic hash value combinations of all data segments to the filtering layer; A request decomposition unit, used to extract keywords from user request data and calculate a hash value combination of each keyword using the hash algorithm group; A hash matching unit, used to search for matching data segments of keywords from the filtering layer by matching hash value combinations; A data query unit, configured to query the requested data from the cache layer if all keywords of the user's requested data have matching data fragments, and query the requested data from the persistence layer otherwise; The data decomposition unit is used for: After decomposing each piece of data in the cache layer into data fragments, all data fragments are deduplicated to obtain a data fragment set; Using each hash algorithm in the hash algorithm group to calculate the data fragments respectively to obtain multiple characteristic hash values, and saving the multiple characteristic hash values ​​as a characteristic hash value combination of the data fragments; The hash algorithm group includes a left shift operation, a right shift operation and an information digest operation.

3. A terminal, It is characterized in that include: processor; A memory for storing execution instructions of the processor; Wherein, the processor is configured to execute the method of claim 1.

4. A computer-readable storage medium storing a computer program, It is characterized in that When the program is executed by a processor, the method according to claim 1 is implemented.

Images