Chinese Bloom filter implementation method and device, equipment and readable storage medium
By setting bits for commonly used Chinese characters in Chinese text, generating a Bloom index, and recording only the index of commonly used Chinese characters, the problems of excessive storage length of the Bloom filter and poor filtering efficiency in the Chinese retrieval system are solved, and the storage space is reduced and the retrieval efficiency is improved.
Patent Information
- Application Number
- CN202411784318.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-04-11
AI Technical Summary
In Chinese search systems, the storage length of existing Bloom filters is too large, occupying huge space and poor filtering efficiency, especially due to the large number of Chinese characters, the index length is too long.
By setting bits for commonly used Chinese characters in Chinese text, a Bronze index is generated, and only the index of commonly used Chinese characters is recorded, ignoring rare characters. The Bronze index length is set to 4096 bits, each bit corresponds to a Chinese character in the code table. For Chinese characters not in the code table, they are not recorded in the Bronze index, and special processing is performed during query.
It effectively reduces the storage length of the Bronx index, improves retrieval efficiency, reduces storage space requirements, and optimizes the utilization of computing resources.
Smart Images
Figure CN119807326A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of online analytical database processing, and in particular to a method, device, and readable storage medium for implementing a Chinese Bloom filter. Background Art
[0002] Since Chinese text contains a large number of characters, using Bloom filters requires a long flag. For example, GB18030-2022 "Chinese Coded Character Set for Information Technology" contains 88,115 Chinese characters. To determine whether a corresponding character appears in a text, 88,115 / 8 to 10KB of space is required. Since the encoding length is too long, the storage space occupied is too large; and most of them are uncommon characters, the actual Bloom effect is not good.
[0003] Therefore, it is a technical problem to be solved urgently by those skilled in the art to propose a technical solution that can effectively reduce the storage length of the Bloom index in a Chinese retrieval system while improving the efficiency of filtering. Summary of the invention
[0004] The purpose of this specification is to solve at least one of the above technical defects. The technical solution adopted in this application is as follows:
[0005] In a first aspect, an embodiment of the present specification provides a method for implementing a Chinese Bloom filter, the method comprising the following steps:
[0006] Step S1: setting a bit for each Chinese character in the Chinese text to mark whether the corresponding Chinese character appears in the corresponding search text;
[0007] Step S2: sort the Chinese characters by their frequency of occurrence, and take the first certain number of Chinese characters;
[0008] Step S3: The Bloom index length is set to 4096 bits, and each bit corresponds to a Chinese character in the code table;
[0009] Step S4: When Chinese text is initially input, all 4096 bits of the Bloom index are set to 0;
[0010] Step S5: Search each Chinese character in the text line. If the corresponding Chinese character in the code table appears, the bit corresponding to the Chinese character in the Bloom index is marked as 1; for Chinese characters not in the code table, the Bloom filter will not record them in the Bloom index, and special processing will be performed during query;
[0011] Step S6: When searching for text, the input keyword is matched with the code table. If the keyword is a Chinese character in the code table range and the corresponding bit in the Bloom index is 0, it is determined to be "unmatched"; if the corresponding bit is 1, it is judged to be "uncertain", and the text content is further searched to determine whether it contains the keyword; if the keyword is a Chinese character outside the code table, it is directly judged as "uncertain" and the text content is further searched.
[0012] Optionally, when searching for text, if the Chinese character of the keyword appears in the code table, but the corresponding bit of the Chinese filter is 0, it is determined that the Chinese character does not exist in the record line text and is judged as "no match".
[0013] Optionally, when searching for keywords, if the Chinese characters of the keywords appear in the code table, but the corresponding bit in the Chinese filter is 1, it cannot be determined that the Chinese text contains the Chinese characters, and the search result is "uncertain". The text of the search record line is used to further determine whether the keywords are included in the search record line.
[0014] Optionally, the method further includes: classifying and merging Chinese characters, every 8 Chinese characters in the code table share a flag bit, and using 64 bytes to establish a corresponding Bloom index, which has a total of 512 bits, and each bit indicates whether a group of Chinese characters appears in the text.
[0015] In a second aspect, an embodiment of the present specification provides a Chinese Bloom filter implementation device, the device comprising: a data access module, a Chinese Bloom filter module, and a Chinese data storage module, wherein:
[0016] The data access module is used for inserting, updating and retrieving Chinese data;
[0017] The Chinese Bloom filter module includes a Chinese filter executor and a Chinese filter coding table;
[0018] The Chinese data storage module is used to store Chinese data.
[0019] Optionally, in the Bloom filter module, by querying the Chinese filter code table, a filtering code segment of the Chinese original text is generated, the Chinese original text and the filtering code segment are inserted into the Chinese data storage module, and Chinese retrieval is performed. By querying the Chinese filter code table, a filtering code of the search keyword is generated and then the Chinese data storage area is queried, and the filtering code segment of the data storage area is first filtered to determine whether it does not contain the search keyword.
[0020] Optionally, in the determination of whether the search keyword is not contained, if the search keyword is not contained, the Chinese original text is directly skipped; if it is not certain whether it is contained, the Chinese original text in the data storage area is further searched to further determine whether it is contained, and finally the search result is returned to the data access module.
[0021] In a third aspect, an embodiment of this specification provides an electronic device, including a processor and a memory;
[0022] The memory is used to store operation instructions;
[0023] The processor is used to execute the above-mentioned Chinese Bloom filter implementation method by calling the operation instruction.
[0024] In a fourth aspect, an embodiment of the present specification provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for implementing the Chinese Bloom filter is implemented.
[0025] In a fifth aspect, a computer program product is provided, which includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the method for implementing the Chinese Bloom filter described above is fully or partially executed.
[0026] The beneficial effects brought by the Chinese Bloom filter implementation scheme disclosed in the embodiments of this specification are:
[0027] Quickly search Chinese texts and exclude rows that do not contain specific keywords, thereby reducing the amount of text that needs to be further searched and improving search efficiency;
[0028] By using bits to mark the occurrence of Chinese characters, the storage space required is reduced compared to storing a complete copy of the text;
[0029] By establishing a code table and marking bits, it is possible to accurately determine whether a Chinese character appears in the text, reducing the possibility of misjudgment;
[0030] For Chinese characters that are not in the code table, the Bloom filter will not be recorded in the Bloom index, effectively avoiding unnecessary calculations and storage, and optimizing the use of computing resources;
[0031] The technical solution provided in this specification can adapt to different application scenarios, and can provide effective support for both large-scale text databases and real-time retrieval systems, with a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.
[0033] Figure 1 A flowchart of a Chinese Bloom filter implementation method provided in an embodiment of the present application;
[0034] Figure 2 A Bloom index diagram of a Chinese Bloom filter implementation method provided in an embodiment of the present application;
[0035] Figure 3 A Bloom index diagram of a Chinese Bloom filter implementation method provided in an embodiment of the present application;
[0036] Figure 4 A schematic diagram of the structure of a Chinese Bloom filter implementation device provided in an embodiment of the present application;
[0037] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0038] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.
[0039] It will be understood by those skilled in the art that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0040] In order to make the objectives, technical solutions and advantages of the present application more clear, the specific implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0041] The core of the present invention is to provide a method for implementing a Chinese Bloom filter. Figure 1 The flowchart of the Chinese Bloom filter implementation method provided by this embodiment is shown. In order to better understand the method, Figure 1 The technical solution of this embodiment is described in detail:
[0042] Step S1: setting a bit for each Chinese character in the Chinese text to mark whether the corresponding Chinese character appears in the corresponding search text;
[0043] Step S2: sort the Chinese characters by their frequency of occurrence, and take the first certain number of Chinese characters;
[0044] Step S3: The Bloom index length is set to 4096 bits, and each bit corresponds to a Chinese character in the code table;
[0045] Step S4: When Chinese text is initially input, all 4096 bits of the Bloom index are set to 0;
[0046] Step S5: Search each Chinese character in the text line. If the corresponding Chinese character in the code table appears, the bit corresponding to the Chinese character in the Bloom index is marked as 1; for Chinese characters not in the code table, the Bloom filter will not record them in the Bloom index, and special processing will be performed during query;
[0047] Step S6: When searching for text, the input keyword is matched with the code table. If the keyword is a Chinese character in the code table range and the corresponding bit in the Bloom index is 0, it is determined to be "unmatched"; if the corresponding bit is 1, it is judged to be "uncertain", and the text content is further searched to determine whether it contains the keyword; if the keyword is a Chinese character outside the code table, it is directly judged as "uncertain" and the text content is further searched.
[0048] In a preferred embodiment of the present specification, when searching for text, if the Chinese characters of the keyword appear in the code table but the corresponding bit in the Chinese filter is 0, it is determined that the Chinese characters do not exist in the record line text and is judged as "no match".
[0049] A preferred method of the embodiments of the present specification is that, when searching for keywords, if the Chinese characters of the keywords appear in the code table, but the corresponding bit in the Chinese filter is 1, it cannot be determined that the Chinese text contains the Chinese characters, and the search result is "uncertain". Whether the keywords are included in the search record line is further determined by searching the text of the record line.
[0050] A preferred method of the embodiments of this specification is to classify and merge Chinese characters, and every 8 Chinese characters in the code table share a flag bit, and use 64 bytes to establish a corresponding Bloom index. The Bloom index has a total of 512 bits, and each bit indicates whether a group of Chinese characters appears in the text.
[0051] The Chinese Bloom filter implementation provided in the embodiments of this specification is used to quickly retrieve keywords in Chinese text. By setting bits for commonly used Chinese characters, a Bloom index is generated, and record rows that do not contain specific keywords are quickly excluded, thereby improving retrieval efficiency and reducing storage space. Further optimization includes classifying and merging Chinese characters, with every 8 Chinese characters sharing a flag bit to improve processing speed and accuracy. It achieves rapid retrieval of Chinese text, reduces the amount of text that needs further retrieval, and improves retrieval efficiency; and by using bits to mark the occurrence of Chinese characters, the storage space requirement is reduced compared to storing a complete copy of the text; the technical solution provided in this specification can adapt to different application scenarios, and can provide effective support for both large-scale text databases and real-time retrieval systems, with a wide range of applications.
[0052] The following will take the Chinese search system as an example to introduce the process of reducing the storage length of the Bloom index and improving the filtering efficiency by using the Chinese Bloom filter implementation method described in the embodiment of the present application. The specific implementation process is as follows:
[0053] The Bloom filter includes a specific code table and a corresponding algorithm, which generates a Bloom index for the input searched text and determines whether the searched text contains the searched keyword (or determines whether the searched text does not contain the searched keyword).
[0054] Taking the simplest Chinese Bloom filter as an example, a bit is set for each Chinese character to mark whether the corresponding Chinese character appears in the corresponding search text. There are more than 80,000 Chinese characters, and the required Bloom index length is: 80,000 / 8 = 10,000 bytes.
[0055] However, among these 80,000 Chinese characters, common characters still account for a minority, and rare characters rarely appear in common Chinese texts. Therefore, this filter only takes common Chinese characters and does not count rare Chinese characters into the Bloom filter. This can effectively save the index space of the Bloom filter.
[0056] Commonly used Chinese characters are obtained according to Chinese statistics. You can refer to Chinese character corpus and other websites, such as www.cncorpus.org, or collect some Chinese corpus for statistics. Finally, a statistical table similar to the following Table 1 is generated:
[0057]
[0058]
[0059] Table 1
[0060] Then sort by the Chinese characters with the highest frequency and take the first certain number of Chinese characters.
[0061] Only common Chinese characters are selected here. Taking 4096 characters as an example, only the 4096 characters from "的" to "邸" are included as the code table of the Chinese filter. Other Chinese characters are not in the code table and are not counted in the Bloom index either.
[0062] The length of the Bloom index is 4096 bits, and each bit corresponds to a Chinese character in the code table. The bits of the Bloom index correspond one by one to the sequence numbers in the code table.
[0063] When initially inputting Chinese text, all 4096 bits of the Bloom index are set to 0. Then start to retrieve each Chinese character in the text line. If the corresponding Chinese character in the code table appears, mark the bit corresponding to this Chinese character in the Bloom index as 1.
[0064] For Chinese characters not in the code table, the Bloom filter will not record them in the Bloom index and will handle them specially during query.
[0065] When retrieving text, compare the input keyword with the Chinese index. First, correspond each Chinese character to the code table. If the Chinese character is within the range of the code table and the corresponding bit in the Bloom index is 0, it indicates that the Chinese character does not appear in the retrieved text, and the corresponding record line can be determined as "not matching" without further retrieving the record line text. Otherwise, it is judged as "uncertain" and further retrieval of the text content is required to determine whether the keyword is included.
[0066] For Chinese characters outside this code table, it is also directly judged as "uncertain" and further retrieval of the text content is required.
[0067] In this way, without retrieving the text content, by querying the Bloom index, the non-matching Chinese record lines can be quickly excluded.
[0068] For each line of text, the length of the Bloom index used is: 4096 / 8 = 512 bytes. Compared with 80000 / 8 = 10000 bytes, the occupied space is reduced to 5.12% of the original space.
[0069] Furthermore, these 4096 Chinese characters are further classified and merged so that every 8 Chinese characters in the code table share one flag bit. Although there are 4096 characters in the code table, every adjacent 8 Chinese characters in the code table share one bit. In this way, only 64 bytes (4096 / 8 / 8) are required to effectively store the index length of Chinese character filtering and retrieval, and the occupied space is reduced to 0.64% of the original space.
[0070] When loading data, for each text field, a corresponding Bloom index is established using 64 bytes. This Bloom index has a total of 64 * 8 = 512 bits, and each bit indicates whether a certain group of Chinese characters (a total of 8) appears in this piece of text. When initially inputting Chinese text, all 512 bits of the Bloom index are set to 0, and then each Chinese character in the text line is retrieved. If the Chinese character is in the code table, the corresponding shared flag bit is set to 1, and Chinese characters not in the code table are not processed.
[0071] When retrieving keywords, the retrieval is still carried out according to the above steps. If the Chinese characters of the keyword appear in the code table but the corresponding bit in the Chinese filter is 0, it can be determined that the Chinese character does not appear in the record line text, and it can be judged as "not matching".
[0072] However, if the corresponding bit is 1, it cannot be determined that the Chinese text contains this Chinese character because this bit is shared by 8 Chinese characters in the code table. At this time, the retrieval result is "uncertain", and it is necessary to retrieve the text of the record line to further determine whether the keyword is included in the retrieval record line. But it can still quickly filter out record lines that do not contain the retrieval keyword, achieving the effect of accelerating the retrieval.
[0073] See Figure 2 As shown, when loading "Jin Zhuan...." into the database or text storage system, a Bloom index is generated according to the prefabricated filter code table. The character "Jin" shares a bit with a total of 8 Chinese characters, namely "Wo", "San", "Lu", "Qiang", "De", "Ke", "Shan". The last bit in the second row of the Bloom filter is marked as "1", indicating that at least one of the above 8 Chinese characters appears.
[0074] The character "Zhuan" shares a bit with a total of 8 Chinese characters, namely "Kai", "Nian", "Que", "Le", "Ke", "Mei", "Mei". The last bit in the fourth row of the index is marked as "1", indicating that at least one of the above 8 Chinese characters appears.
[0075] When retrieving data, for the input retrieval keyword, the corresponding binary bits are generated in the same way as above. If the corresponding number of this Chinese character in the Chinese index is 0, it means that this Chinese character does not appear in this piece of text, that is, "not matching", and the retrieval of the text content can be skipped. If the result of the operation is not 1, it means that the retrieval keyword may exist in this piece of text, that is, "uncertain". For record lines that are uncertain, a deep retrieval is performed through exact matching.
[0076] See Figure 3As shown, when the input "gold seal script" is used as the search keyword, the search server generates corresponding binary marker bits according to the code table (the character "gold" is in the last bit of the second row, and the character "seal script" is in the last bit of the fourth row), and uses these binary marker bits to match each index corresponding to the text one by one. In the matching result, if the corresponding bit value is "1", it indicates that this text may contain the two characters "gold seal script". However, it cannot be determined for sure that it is exactly the two characters "gold seal script", and further precise matching needs to be carried out through means such as forward indexing.
[0077] Based on Figure 1 the Chinese Bloom filter implementation method provided by the embodiment shown, Figure 4 Figure 1 shows a Chinese Bloom filter implementation device provided by an embodiment of the present application, as Figure 4 shown, the device mainly includes: a 201 data access module, a 202 Chinese Bloom filter module, and a 203 Chinese data storage module. Among them,
[0078] The data access module 201: is used for inserting, updating, and retrieving Chinese data;
[0079] The Chinese Bloom filter module 202 includes a Chinese filter executor and a Chinese filter coding table;
[0080] The Chinese data storage module 203 is used for storing Chinese data.
[0081] In a preferred embodiment of this specification, in the Chinese Bloom filter module 202, by querying the Chinese filter coding table, a filtered coding segment of the Chinese original text is generated, and the Chinese original text and the filtered coding segment are inserted into the Chinese data storage module for Chinese retrieval. By querying the Chinese filter coding table, a filtered code of the search keyword is generated, and then the Chinese data storage area is queried. First, the filtered coding segments in the data storage area are filtered to determine whether they do not contain the search keyword.
[0082] In a preferred embodiment of this specification, in the determination of whether the search keyword is not contained, if the search keyword is not contained, the Chinese original text is directly skipped; if it is not certain whether it is contained, the Chinese original text in the data storage area is further retrieved to further determine whether it is contained, and finally the search result is returned to the data access module.
[0083] It can be understood that the above-mentioned modules of the Chinese Bloom filter implementation device in this embodiment have the function of implementing Figure 1The functions of the corresponding steps of the method in the embodiment shown in . The function can be implemented by hardware, or by hardware executing the corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The above modules can be software and / or hardware, and the above modules can be implemented separately or integrated with multiple modules. For the functional description of the above modules, please refer to Figure 1 The corresponding description of the method in the embodiment shown in will not be repeated here.
[0084] The Chinese Bloom filter implementation provided in the embodiments of this specification is used to quickly retrieve keywords in Chinese text. By setting bits for commonly used Chinese characters, a Bloom index is generated, and record rows that do not contain specific keywords are quickly excluded, thereby improving retrieval efficiency and reducing storage space. Further optimization includes classifying and merging Chinese characters, with every 8 Chinese characters sharing a flag bit to improve processing speed and accuracy. It achieves rapid retrieval of Chinese text, reduces the amount of text that needs further retrieval, and improves retrieval efficiency; and by using bits to mark the occurrence of Chinese characters, the storage space requirement is reduced compared to storing a complete copy of the text; the technical solution provided in this specification can adapt to different application scenarios, and can provide effective support for both large-scale text databases and real-time retrieval systems, with a wide range of applications.
[0085] The embodiment of the present application also provides an electronic device, including a processor and a memory;
[0086] A memory, used for storing operation instructions;
[0087] The processor is used to execute the Chinese Bloom filter implementation method provided in any embodiment of the present application by calling an operation instruction.
[0088] As an example, Figure 5 A schematic diagram of the structure of an electronic device applicable to the embodiment of the present application is shown. Figure 5 As shown, the electronic device 2000 includes: a processor 2001 and a memory 2003. The processor 2001 and the memory 2003 are connected, such as through a bus 2002. Optionally, the electronic device 2000 may also include a transceiver 2004. It should be noted that in actual applications, the transceiver 2004 is not limited to one, and the structure of the electronic device 2000 does not constitute a limitation on the embodiments of the present application.
[0089] The processor 2001 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 2004 may include a receiver and a transmitter. The transceiver 2004 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.
[0090] Processor 2001 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 2001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0091] The bus 2002 may include a path to transmit information between the above components. The bus 2002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 2002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0092] The memory 2003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.
[0093] Optionally, the memory 2003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 2001. The processor 2001 is used to execute the application code stored in the memory 2003 to implement the Chinese Bloom filter implementation method provided in any embodiment of the present application.
[0094] The electronic device provided in the embodiment of the present application is applicable to any embodiment of the above method and will not be described in detail here.
[0095] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the Chinese Bloom filter implementation method shown in the above method embodiment is implemented.
[0096] The computer-readable storage medium provided in the embodiments of the present application is applicable to any embodiment of the above method and will not be described in detail here.
[0097] The Chinese Bloom filter implementation provided in the embodiments of this specification is used to quickly retrieve keywords in Chinese text. By setting bits for commonly used Chinese characters, a Bloom index is generated, and record rows that do not contain specific keywords are quickly excluded, thereby improving retrieval efficiency and reducing storage space. Further optimization includes classifying and merging Chinese characters, with every 8 Chinese characters sharing a flag bit to improve processing speed and accuracy. It achieves rapid retrieval of Chinese text, reduces the amount of text that needs further retrieval, and improves retrieval efficiency; and by using bits to mark the occurrence of Chinese characters, the storage space requirement is reduced compared to storing a complete copy of the text; the technical solution provided in this specification can adapt to different application scenarios, and can provide effective support for both large-scale text databases and real-time retrieval systems, with a wide range of applications.
[0098] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the specific device can be divided into different functional modules to complete all or part of the functions described above.
[0099] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0100] In several embodiments covered by the present application, it should be understood that the disclosed devices, equipment and methods can be implemented in other ways. For example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.
[0101] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.
[0102] In addition, each functional module in each embodiment of the present application can be implemented in the form of hardware or in the form of software functional modules. If these functional modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or part of the prior art that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a computer program product, which includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program product is stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, a computer instruction can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).
[0103] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.
Claims
1. A method for implementing a Chinese Bloom filter, characterized in that: The method comprises the following steps: Step S1: setting a bit for each Chinese character in the Chinese text to mark whether the corresponding Chinese character appears in the corresponding search text; Step S2: sort the Chinese characters by their frequency of occurrence, and take the first certain number of Chinese characters; Step S3: The Bloom index length is set to 4096 bits, and each bit corresponds to a Chinese character in the code table; Step S4: When Chinese text is initially input, all 4096 bits of the Bloom index are set to 0; Step S5: Search each Chinese character in the text line. If the corresponding Chinese character in the code table appears, the bit corresponding to the Chinese character in the Bloom index is marked as 1; for Chinese characters not in the code table, the Bloom filter will not record them in the Bloom index, and special processing will be performed during query; Step S6: When searching for text, the input keyword is matched with the code table. If the keyword is a Chinese character in the code table range and the corresponding bit in the Bloom index is 0, it is determined to be "unmatched"; if the corresponding bit is 1, it is judged to be "uncertain", and the text content is further searched to determine whether it contains the keyword; if the keyword is a Chinese character outside the code table, it is directly judged as "uncertain" and the text content is further searched.
2. The Chinese Bloom filter implementation method according to claim 1, characterized in that: When searching for text, if the Chinese character of the keyword appears in the code table, but the corresponding bit in the Chinese filter is 0, it is determined that there is no Chinese character in the record line text and it is judged as "no match".
3. The Chinese Bloom filter implementation method according to claim 1, characterized in that: When searching for keywords, if the Chinese characters of the keywords appear in the code table, but the corresponding bit in the Chinese filter is 1, it cannot be determined that the Chinese text contains the Chinese characters, and the search result is "uncertain". The text of the search record line is used to further determine whether the keywords are included in the search record line.
4. The Chinese Bloom filter implementation method according to claim 1, characterized in that: The method further includes: classifying and merging Chinese characters, wherein every 8 Chinese characters in the code table share a flag bit, and a corresponding Bloom index is established using 64 bytes. The Bloom index has a total of 512 bits, and each bit indicates whether a certain group of Chinese characters appears in the text.
5. A Chinese Bloom filter implementation device, characterized in that: The device comprises: a data access module, a Chinese Bloom filter module, and a Chinese data storage module, wherein: The data access module is used for inserting, updating and retrieving Chinese data; The Chinese Bloom filter module includes a Chinese filter executor and a Chinese filter coding table; The Chinese data storage module is used to store Chinese data.
6. The Chinese Bloom filter implementation device according to claim 5, characterized in that: In the Bloom filter module, by querying the Chinese filter code table, the filtering code segment of the Chinese original text is generated, and the Chinese original text and the filtering code segment are inserted into the Chinese data storage module to perform Chinese retrieval. By querying the Chinese filter code table, the filtering code of the search keyword is generated and then the Chinese data storage area is queried. The filtering code segment of the data storage area is first filtered to determine whether it does not contain the search keyword.
7. The Chinese Bloom filter implementation device according to claim 6, characterized in that: In the determination of whether the search keyword is not included, if the search keyword is not included, the Chinese original text is directly skipped; if it is not sure whether it is included, the Chinese original text in the data storage area is further searched to further determine whether it is included, and finally the search result is returned to the data access module.
8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store operation instructions; The processor is used to execute the method according to any one of claims 1 to 4 by calling the operation instruction.
9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
10. A computer program product, characterized in that The computer program product comprises one or more computer instructions, and when the computer program instructions are loaded and executed on a computer, the method according to any one of claims 1 to 4 is fully or partially executed.