Data management method and device, equipment, medium and product

Abstract

The invention discloses a data management method and device, equipment, a medium and a product. The method comprises the following steps: determining keywords of full-text retrieval; inputting the keyword into a preset filter, and filtering the keyword; and when the keyword is filtered by the filter, querying the keyword from the granule corresponding to the filter. According to the scheme, the problem of word segmentation flexibility of Clickhome hop index can be solved, and the support degree of a full-text retrieval scene is enhanced.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and more specifically to a data management method, apparatus, device, medium, and product. Background Technology

[0002] Current mainstream data management solutions mainly include ELK (Elasticsearch, Logstash, Kibana, a data management solution based on Elasticsearch) and EFK (Elasticsearch, Fluentd, Kibana, a data management solution based on Elasticsearch). Both ELK and EFK solutions primarily rely on the Elasticsearch search engine. Elasticsearch is based on the open-source search engine Lucene, and its core is a fast indexing mechanism based on inverted indexes. Inverted indexes segment articles through text analysis and then record the relationships between these segments and the articles for subsequent search. Inverted indexes need to record a large number of segmented word relationships, thus consuming significant disk space; furthermore, because searching using segmented words requires loading the complete segments into memory, it also consumes considerable memory.

[0003] The ClickHouse solution is based on the ClickHouse columnar data storage component. ClickHouse divides data into granules, the smallest management units, with a fixed number of rows. Multiple granules are then encapsulated into a single block, which is compressed and persisted, ultimately achieving compressed storage of large amounts of data. Granules are divided according to the primary key column (which may have more than one column) of the data table, sorted by priority. Rows with adjacent rows are then encapsulated into a single granule. The first row of each granule forms an index entry, which constitutes ClickHouse's sparse index. ClickHouse uses a multi-core streaming concurrency strategy for data reading and writing, and compresses and decompresses data in block-level uppercase, resulting in lower disk space usage and higher read / write efficiency. However, ClickHouse is limited by its built-in word segmentation not supporting Chinese characters and special characters, and lacks effective means to combine its built-in skip count index with third-party word segmentation. The skip count index has word segmentation limitations, which poses significant constraints in full-text search scenarios. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a data management method, apparatus, device, medium, and product that can resolve the word segmentation flexibility problem of Clickhouse jump count index and enhance support for full-text search scenarios.

[0005] This invention provides a data management method, including: Determine the keywords for full-text search; The keywords are input into a preset filter for filtering. When the filter passes through the filter, the keyword is queried from the granule corresponding to the filter.

[0006] Preferably, determining the keywords for full-text retrieval includes: Based on the query conditions of full-text search, the granule containing the target data is located through a sparse index; Determine whether the query conditions meet the preset keyword conditions; If not, perform a binary search using a sparse index based on the full-text search query conditions to retrieve the corresponding target data; If so, obtain the keywords corresponding to the stated keyword conditions.

[0007] Preferably, the method further includes: If the filter fails to pass the filter, skip the query operation for the granule corresponding to the filter; Input the keyword into the filter corresponding to other granules and determine whether it passes the filter.

[0008] Preferably, the step of inputting the keywords into a preset filter for filtering includes: The keyword is calculated using K preset hash functions to obtain K hash values; An encoder consisting of K neurons receives K hash values ​​and calculates the encoder's output. The encoder stores the index value information corresponding to all keywords. The encoding output and the identifier information of the granule corresponding to the filter are input into a preset decoder to determine whether the granule corresponding to the filter contains the target data corresponding to the index value information; When the target data is present, it is determined that the filter passes. If the target data does not exist, it is determined that the filter has failed.

[0009] Preferably, before inputting the keyword into the filter corresponding to other granules, the method further includes: Determine if there are filters corresponding to granules for which no filtering operation has been performed; If not, perform a binary search using a sparse index based on the full-text search query conditions to retrieve the corresponding target data; If so, input the keyword into the filter corresponding to the granule that has not undergone filtering.

[0010] Preferably, querying the keyword from the granule corresponding to the filter includes: Based on the query conditions of the full-text search, a binary search is performed using a sparse index to determine the locating file of the query. Obtain the persistent association information of the granule from the mrk file; The corresponding block is retrieved and decompressed from the pre-stored persistent file based on the persistent association information; The decompressed block is loaded into memory, and the target granule is determined based on the positioning information obtained from the mrk file; Based on the query conditions of the keywords, retrieve the corresponding target data from the target granule.

[0011] This invention also provides a data management device, the device comprising: The determination module is used to determine the keywords for full-text search; The filtering module is used to input the keywords into a preset filter for filtering. The query module is used to query the keyword from the granule corresponding to the filter when the filter is passed through the filter.

[0012] This invention also provides a communication device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the data management method as described in any of the above embodiments.

[0013] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to perform the data management method as described in any of the above embodiments.

[0014] This invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0015] This invention provides a data management method, apparatus, device, medium, and product. The method involves determining keywords for full-text search; inputting the keywords into a preset filter for filtering; and retrieving the keywords from the corresponding granule when they pass the filter. This solution addresses the flexibility issue of word segmentation in ClickHouse's skip-count index, enhancing support for full-text search scenarios. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the data management method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the Clickhouse query process provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the working process of the Bloom filter provided in an embodiment of the present invention; Figure 4 This is another flowchart illustrating the data management method provided in this embodiment of the invention; Figure 5 This is a schematic diagram of the structure of the recurrent neural network provided in an embodiment of the present invention; Figure 6 This is another flowchart illustrating the data management method provided in this embodiment of the invention; Figure 7 This is a schematic diagram of the structure of a data management device provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a communication device provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To address the aforementioned technical issues, this case provides a data management method, see [link to relevant documentation]. Figure 1 This is a flowchart illustrating a data management method provided in an embodiment of the present invention, the method comprising: Step S1: Determine the keywords for the full-text search; Step S2: Input the keywords into a preset filter for filtering; Step S3: When the filter passes through the filter, the keyword is queried from the granule corresponding to the filter.

[0019] In this specific implementation, the Clickhouse solution is based on the Clickhouse columnar data storage component. Clickhouse divides data into granules according to a fixed number of rows, then encapsulates multiple granules into a block and compresses and persists them, ultimately achieving compressed storage of large amounts of data. The granules are divided according to the primary key column (which may be more than one column) of the data table, sorted in order of priority, and finally, the data rows that are adjacent in the sort order are encapsulated into a granule. At the same time, the first row of each granule together constitutes an index entry, which forms the sparse index of Clickhouse.

[0020] Clickhouse employs a multi-core streaming concurrency strategy for data reading and writing, and compresses and decompresses data in block uppercase, resulting in lower system disk space usage and higher data read / write efficiency. However, it has significant limitations in full-text search scenarios. Full-text search refers to a search technique that first extracts key information from unstructured text to build an index, and then quickly retrieves the target text through the index. Clickhouse achieves full-text search through a skip count index, but its limitation lies in the fact that its built-in word segmentation does not support Chinese characters and special characters, and it lacks effective means to combine its built-in skip count index with third-party word segmentation.

[0021] See Figure 2 This is a schematic diagram of the Clickhouse query process provided in an embodiment of the present invention. The Clickhouse query process is as follows: Figure 2 As shown, firstly, a binary search is performed based on the sparse index according to the query conditions to locate the granule containing the query target; then, the persistent association information of the granule is obtained from the xxx.mrk file, and the block containing the granule is obtained and decompressed from the persistent file xxx.bin according to the association information; finally, the decompressed block is streamed into memory, and the granule in the block is located according to the information obtained from the xxx.mrk file.

[0022] The Clickhouse solution achieves full utilization of disk space by batch compressing granules. Simultaneously, it employs a strategy of large-volume, multi-core streaming concurrency for read and write operations, providing extremely high read and write performance while fully leveraging the multi-core performance of the CPU.

[0023] However, Clickhouse has significant shortcomings in full-text search scenarios. Clickhouse achieves full-text search through the hop count index, and its shortcomings stem from the word segmentation limitations of the hop count index: (1) it only supports the MergeTree table engine; (2) the word segmentation capabilities of the three advanced hop count indexes are severely limited, ngrambf_v1 only supports word segmentation according to a fixed length, and tokenbf_v1 and inverted do not support word segmentation containing Chinese characters and special characters.

[0024] This application is based on the Clickhouse scheme and introduces a solution based on the Bloom filter concept to address the word segmentation flexibility issue of Clickhouse skip count index, providing a high-performance, feature-rich data management method.

[0025] When implementing this solution, the retrieval mechanism for precise keyword filtering and targeted granule queries first performs an access check on the input keywords through a filter. After the check passes, the keyword query is then executed from the exclusive granule bound to the filter. This achieves pre-filtering and targeted retrieval, avoids invalid full-domain queries, and improves retrieval efficiency and accuracy.

[0026] Keywords, or the core terms / phrases entered by the user, are the core carriers of the search request. This solution only targets keyword queries. First, determine whether keywords are involved based on the full-text search criteria, and then identify the keywords for the target data. If the target data does not involve keyword queries, there is no need to filter it.

[0027] A pre-configured set of validation rules, with each filter corresponding to a specific validation condition and a different granule; A Granule, or retrieval data unit, is the smallest independent set for storing retrieval resources. Each filter is bound to a unique / specified granule, and the filter is the retrieval entry point for the granule.

[0028] If the filter passes: the keyword meets all the preset validation conditions of the corresponding filter, triggering subsequent targeted query actions; if the filter fails, the search is terminated, and the granule query operation is not executed.

[0029] It receives the target keywords input by the user, determines the search keywords, and uses them as the basis for subsequent filtering and querying.

[0030] Input standardized keywords into the preset filter, activate the filter's preset validation logic, and perform filtering and validation on the keywords: If the keyword meets all the validation conditions of the current filter, the exclusive granule bound to the filter will be automatically identified, and a targeted query command will be triggered. If the keyword does not meet the filter validation conditions, the search process will be terminated, and the following feedback result may be returned: "The keyword did not pass the filter of the corresponding granule, and there is no need to query this granule."

[0031] Provided the filter passes, keyword retrieval is performed only within the granule corresponding to that filter, retrieving resource data stored in the granule that matches the keyword, without involving a full-domain scan of other granules.

[0032] The target data is obtained by retrieving the matching results from the targeted granule.

[0033] It's important to note that a Bloom filter consists of a bit array of length m and k hash functions. When adding an element to the Bloom filter, the element is first hashed using the hash functions to obtain k hash indices. Then, the elements corresponding to these k indices in the array are set to 1. When checking if an element exists in the Bloom filter, the element is hashed using the hash functions to obtain k hash indices. Then, it is checked whether all the elements corresponding to these k indices in the array are 1. If all are 1, the element exists in the Bloom filter; otherwise, the element does not exist.

[0034] Clickhouse reduces disk usage through block-compressed storage and achieves high efficiency through sparse indexes and streaming multi-core concurrent operations, but its support for full-text search is limited. This proposal introduces a Bloom filter into Clickhouse's search process, using the Bloom filter as a keyword index to replace the skip count index, thereby improving Clickhouse's flexible search capabilities. It supports mixed word segmentation of Chinese, English, numbers, and special characters, with a segmentation accuracy of no less than 98%, meeting the diverse needs of full-text search scenarios and overcoming the segmentation limitations of Clickhouse's native skip count index.

[0035] In yet another embodiment provided by the present invention, see Figure 3 This is a schematic diagram of the working process of the Bloom filter provided in an embodiment of the present invention. Specifically, when step S1 is implemented: The process of generating a Bloom filter index is as follows: Figure 3 As shown, when the Clickhouse engine persists the data transmitted in the data channel, it uses an additional third-party word segmentation service to segment the data to obtain keywords, retrieves the granule information corresponding to the row of data from Clickhouse, and finally adds the keywords to the Bloom filter of the corresponding granule to form the keyword index of the granule.

[0036] See Figure 4This is another flowchart of the data management method provided in this embodiment of the invention. When performing full-text search, a filter is used to determine whether a keyword exists in a certain granule. If it does not exist, the operation of decompressing the granule from the disk space and loading it into memory is skipped, thereby achieving the purpose of accelerating Clickhouse full-text search.

[0037] After introducing the Bloom filter, ClickHouse's query process is as follows: First, locate the granule containing the target data using a sparse index based on the query conditions. Then, analyze whether the query conditions involve keyword queries. If not, perform the query according to the original logic. Otherwise, check if the keyword exists in the Bloom filter corresponding to the granule. If it does, perform the query according to the original logic. Otherwise, skip the granule and query other granules where the target data may exist.

[0038] In practice, when a user initiates a full-text search request, such as "query log data containing 'payment failed'", the Clickhouse engine parses the query conditions and distinguishes whether keyword queries are involved. Non-keyword queries (such as range queries and aggregation queries): directly follow the original process.

[0039] Keyword search: Extract search keywords (such as "payment failed") and proceed with the keyword search process.

[0040] Based on non-keyword filtering conditions in the query (such as time range and business type), a binary search is performed using the Clickhouse sparse index to locate the set of granules (candidate granule set) that may contain the target data, thus narrowing the search scope.

[0041] Bloom filters introduce only a very small amount of memory relative to the total amount of data. At the same time, Bloom filters can effectively search for the existence of query keywords in granules, thereby effectively avoiding disk I / O operations during keyword retrieval and improving the efficiency of full-text search.

[0042] In another embodiment provided by the present invention, when the keyword is not filtered by a filter, see [link to relevant documentation]. Figure 4 When the filter fails the query, there is still corresponding data for other granules. Therefore, a sparse index (in memory) is created based on the query conditions to relocate other granules. Specifically, the keyword is input into the filter corresponding to other granules to determine whether it passes the filter.

[0043] Input the query keywords into a third-party word segmentation service (if it is a multi-word query, it will be split and processed separately) to generate k corresponding hash values; The hash value is input into the recurrent neural network encoder, and combined with the identifier information of each granule in the candidate granule set, it is input into the decoder to obtain the model output: Output 0: This indicates that the granule does not contain the target keyword, so it is skipped directly without needing to be decompressed and loaded from the disk (to avoid invalid disk I / O).

[0044] Output 1: It is determined that the granule may contain the target keyword, and the subsequent data loading process will proceed.

[0045] Full-text search response time is improved by more than 60% compared to native ClickHouse, with a search latency of no more than 500ms for millions of data points, reaching the same level as ELK / EFK solutions. Memory usage during the search process is lower than ELK / EFK solutions, eliminating the need to store the complete inverted index and skipping disk I / O for decompressing and loading invalid granules.

[0046] In another embodiment of the present invention, since there is a probability of hash collisions between hash functions, Bloom filters are prone to misclassification, and the more elements inserted into the Bloom filter, the higher the probability of misclassification. The misclassification rate can be reduced by increasing the number of bit groups m. If the number of elements inserted into the Bloom filter is n, the formula for calculating its misclassification probability is as follows: ; A Bloom filter is created for each Clickhouse basic data management unit, granule. Keywords obtained by word segmentation of the data rows within the granule are inserted into the Bloom filter as elements, thus creating a keyword index for each granule.

[0047] Configuring a Bloom filter for each granule in ClickHouse can achieve flexible word segmentation and improve full-text search performance, but this method suffers from excessive space consumption. Firstly, to ensure comprehensive storage of word segmentation information, a single Bloom filter may be quite long, potentially occupying several megabytes (MB). While the space occupied by a single Bloom filter is relatively small, the number of granules in a ClickHouse is enormous, reaching tens of thousands or even millions. If a single Bloom filter is m (bits) long and the number of granules is n, then the total space consumption is O(m×n), exceeding ten thousand MB.

[0048] A recurrent neural network is constructed using the idea of ​​a Bloom filter to address the problem of excessive space consumption. See also Figure 5 This is a schematic diagram of the structure of a recurrent neural network provided in an embodiment of the present invention.

[0049] The encoder consists of k neurons that receive k hash values ​​generated from keywords using a hash function, encoding and storing information about all possible keywords that may appear in Clickhouse. The hash function chosen is MurmurHash3 to ensure a uniform distribution of hash values ​​and reduce the probability of collisions.

[0050] Decoder: Initially, it consists of one neuron that receives granule identification information (such as granule ID) and encoder output, and stores the association between keywords and granules. By interspersing additional neurons (with granule identification information as input), the number of parameters is increased to ensure that a massive amount of association data between granules and keywords can be stored.

[0051] Training dataset: Collect text data from typical Clickhouse application scenarios (such as log analysis and business queries), process it through a third-party word segmentation service, and generate a training sample set of "keyword-hash value-granule ID" with a sample size of no less than 1 million.

[0052] Training objective: Minimize the binary classification error of the model (output 1 indicates that the granule contains the target keyword, output 0 indicates that it does not), and iterate the training until the model accuracy is not less than 95% and the misclassification rate is less than 3%.

[0053] When data is added, modified, or deleted in Clickhouse, the word segmentation service is triggered to reprocess the corresponding data rows and update the keyword set; the encoder (supplementing new keyword hash values) and decoder (adjusting the association between granules and keywords) of the recurrent neural network are updated simultaneously to ensure that the index is effective in real time.

[0054] The principle of third-party word segmentation: By combining dictionary matching and semantic analysis, it breaks through the limitation of Clickhouse's native word segmentation, which only supports fixed length or English, and achieves accurate word segmentation of Chinese and special characters, thus improving the flexibility of keyword extraction.

[0055] Drawing on the core idea of ​​Bloom filters, which uses multiple hash placeholders to determine existence, an encoder is used to encode and store the hash values ​​of keywords, replacing the bit array of the Bloom filter. The decoder determines the existence of "keyword-granule" by associating granule identifiers with encoding results. At the same time, it addresses the problem of excessive space usage of Bloom filters in scenarios with a large number of granules by adding neurons (the total space usage is reduced from O(m×n) to O(k+n), where k is the number of encoder neurons and n is the number of decoder neurons).

[0056] The step of inputting the keywords into a preset filter for filtering includes: The Bloom filter receives k+1 elements as input, including the identifier of one granule and k hash values ​​generated from the target keyword. The Bloom filter outputs whether the target keyword exists in the granule. Based on this idea, a recurrent neural network (RNN) is constructed: k neurons receive the k hash values ​​generated from the target keyword, forming the encoder of the RNN, which encodes and stores all possible keyword information that may appear in Clickhouse; one neuron receives the granule identifier and the encoder output, forming the decoder of the RNN, which stores the association between keyword information and granules; the decoder output is a binary classification, used to identify whether the target keyword exists in a given granule.

[0057] Keywords are sparsely distributed in the Bloom filter corresponding to each granule. Therefore, configuring a Bloom filter for each granule results in significant wasted internal space for each filter. By centrally storing the keyword information of all granules in the encoder, space can be effectively saved. Correspondingly, to achieve an effect similar to a Bloom filter, the number of parameters in the recurrent neural network encoder must at least exceed the length of a single Bloom filter. Meanwhile, the decoder contains only one neuron, limiting its storage capacity and potentially preventing it from storing all the association information between granules and keywords, necessitating an increase in the number of decoder parameters. This can be achieved by interleaving additional neurons into the recurrent neural network (when adding neurons to the decoder, the granule identification information needs to be used as the neuron's input).

[0058] By using recurrent neural networks, the flexibility of word segmentation is improved, supporting mixed word segmentation of Chinese, English, numbers and special characters, improving word segmentation accuracy, meeting the diverse needs of full-text search scenarios, and breaking through the word segmentation limitations of Clickhouse's native jump count index.

[0059] To optimize space usage, compared to configuring a Bloom filter for each granule, space usage is reduced by more than 80%, and the total space usage in a scenario with millions of granules is controlled within 100MB.

[0060] Improve index update efficiency: The index update delay triggered by a single data change is no more than 10ms, which does not affect the native data read and write performance of Clickhouse.

[0061] In another embodiment of this invention, after filtering granules using a recurrent neural network, the overall query process is consistent with that when using a Bloom filter. The difference lies in the scenarios of misjudgment. When a Bloom filter determines that a keyword exists in a granule, the keyword may not actually exist in the granule (due to hash collisions). If a Bloom filter determines that a keyword does not exist in a granule, then the keyword must not exist in the granule. Therefore, a Bloom filter can directly filter granules. However, when a recurrent neural network determines that a keyword exists in a granule, the keyword may or may not exist (with a higher probability of existence). When it determines that a keyword does not exist in a granule, the keyword may or may not exist (with a higher probability of non-existence). A mechanism is needed to accommodate scenarios where the recurrent neural network misjudges keywords.

[0062] See Figure 6 This is another flowchart illustrating the data management method provided in this embodiment of the invention. Before inputting keywords into the filters corresponding to other granules, the method further includes: The recurrent neural network needs to be compatible with the following misjudgment scenario: the network judges that the target keyword does not exist in the granule, but in fact the target keyword exists in the granule.

[0063] One approach to accommodate this scenario is to record the granules that are filtered out by the recurrent neural network during each query involving keywords.

[0064] Determine if there are filters corresponding to granules for which no filtering operation has been performed; If so, a binary search is performed using a sparse index based on the full-text search query conditions to retrieve the corresponding target data; If no matching record is found after querying all possible granules containing the target, the recorded granules are traversed, attempting to retrieve the target record from these granules. At this point, the speed of the entire full-text search degrades to that before the addition of the recurrent neural network, but the single-search accuracy remains high. Based on the query conditions of the full-text search, a sparse index is used for binary search to retrieve the corresponding target data. As another preferred option, no processing is performed at this time, tolerating the misjudgments of the recurrent neural network. This approach has extremely high full-text search efficiency, but relatively low search accuracy, making it suitable for big data scenarios.

[0065] If no matching record is found after traversing all candidate granules, the compatibility mechanism is activated: traversing the granules filtered out by the model, repeating the loading and matching process to ensure retrieval accuracy. At this time, the retrieval speed degrades to the level of the original Clickhouse, but the accuracy reaches 100%. If users can tolerate a certain degree of error, such as in scenarios involving rapid filtering of large datasets, the compatibility mechanism can be disabled, and the results can be returned directly, significantly improving retrieval efficiency.

[0066] By employing the hierarchical retrieval principle of filters, the range of candidate granules is first narrowed down through a sparse index, then invalid granules are filtered out through a recurrent neural network, and finally precise matching is performed, achieving coarse, fine, and precise hierarchical retrieval, thereby reducing unnecessary disk I / O and memory usage.

[0067] By leveraging the characteristic of recurrent neural networks that "the probability of not containing keywords is higher when the output is 0", a fallback traversal mechanism is used to balance retrieval efficiency and accuracy, meeting the needs of different scenarios.

[0068] The full-text search response time is significantly improved compared to the native Clickhouse, with a search latency of no more than 500ms for millions of data points, reaching the same level as ELK / EFK solutions.

[0069] The memory usage during the retrieval process is lower than that of the ELK / EFK scheme. It does not require storing the complete inverted index, skips the decompression and loading of invalid granules, and reduces disk I / O operations.

[0070] It supports high-precision retrieval (100% accuracy with compatibility mechanism enabled) and high-efficiency retrieval (50% efficiency improvement with compatibility mechanism disabled); both modes are suitable for different scenarios such as log analysis, business monitoring, and big data filtering.

[0071] In another embodiment of the present invention, during data access and persistence, service production data (such as logs, business data, and other unstructured / semi-structured data) is received through a data pipeline and transmitted to the Clickhouse engine.

[0072] ClickHouse follows the original mechanism to sort data rows according to the priority of the primary key column of the data table. It then encapsulates adjacent data rows after sorting into the smallest management unit, granule, and compresses multiple granules into the smallest compressed storage unit, block. Finally, it persists the data in the form of an xxx.bin file and generates an xxx.mrk file to record the persistent association information of the granules, such as the block offset and the position of the granule in the block.

[0073] When maintaining a sparse index, a sparse index is built based on the first row of data in each granule, forming a set of index entries. This is used to quickly locate the range of the granule where the target data is located, maintaining Clickhouse's original high-efficiency positioning capabilities.

[0074] Based on Clickhouse's columnar storage characteristics, granule batch compression reduces disk usage; the sparse index is built based on the first row of sorted granules, supporting binary search for fast location and providing basic location capabilities for subsequent retrieval.

[0075] The step of querying the keyword from the granule corresponding to the filter includes: Clickhouse employs a multi-core streaming concurrency strategy for data reading and writing, and compresses and decompresses data in block uppercase, resulting in lower system disk space usage and higher data read / write efficiency. However, it has significant limitations in full-text search scenarios. Full-text search refers to a search technique that first extracts key information from unstructured text to build an index, and then quickly retrieves the target text through the index. Clickhouse achieves full-text search through a skip count index, but its limitation lies in the fact that its built-in word segmentation does not support Chinese characters and special characters, and it lacks effective means to combine its built-in skip count index with third-party word segmentation.

[0076] The query process of Clickhouse is shown in the figure: First, a binary search is performed based on the query conditions and the sparse index to locate the granule containing the query target; then, the persistent association information of the granule is obtained from the xxx.mrk file, and the block containing the granule is obtained and decompressed from the persistent file xxx.bin based on the association information; finally, the decompressed block is streamed into memory, and the granule in the block is located based on the information obtained from the xxx.mrk file.

[0077] Based on the principles of Clickhouse hop count indexing and Bloom filters, a recurrent neural network is proposed as the keyword index for Clickhouse. This network combines the idea of ​​using multiple hash operations on input elements to place entries, and then determining the existence of an element by judging the placement of these hash results, thus enabling the function of determining whether a keyword exists in a granule. Furthermore, it utilizes the principle of hop count indexing to improve full-text search performance by skipping I / O operations on granules where the keyword absolutely does not exist. Ultimately, it achieves flexible and efficient full-text search functionality, reaching a level of full-text search performance comparable to the ELK / EFK scheme (where full-text search is achieved by building an index similar to using keywords as keys and text indexes containing keywords as values, then quickly locating text using keywords during the search). The recurrent neural network achieves high-speed full-text search by quickly skipping the smallest data storage unit where the keyword does not exist and by using Clickhouse's multi-core streaming concurrency strategy.

[0078] See Figure 7 This is a schematic diagram of a data management device provided in an embodiment of the present invention. The device includes: The determination module is used to determine the keywords for full-text search; The filtering module is used to input the keywords into a preset filter for filtering. The query module is used to query the keyword from the granule corresponding to the filter when the filter is passed through the filter.

[0079] The data management device provided in this embodiment can execute all the steps and functions of the data management method provided in any of the above embodiments. The specific functions of the device will not be described in detail here.

[0080] See Figure 8 This is a schematic diagram of a communication device provided in an embodiment of the present invention. The communication device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a data management program. When the processor executes the computer program, it implements the steps in each of the above-described data management method embodiments, for example... Figure 1 The steps shown. Alternatively, when the processor executes the computer program, it implements the functions of each module in the above-described device embodiments.

[0081] For example, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the data management device. For example, the computer program can be divided into several modules, the specific functions of which have been described in detail in the data management method provided in any of the above embodiments; therefore, the specific functions of the device will not be repeated here.

[0082] The communication device described herein can be a desktop computer, laptop, handheld computer, or cloud server, etc. The communication device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the schematic diagram is merely an example of a communication device and does not constitute a limitation on a data management device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the communication device may also include input / output devices, network access devices, buses, etc.

[0083] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the data management device, connecting various parts of the data management device via various interfaces and lines.

[0084] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the data management device by running or executing the computer programs and / or modules stored in the memory and by calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0085] If the modules integrated into the data management device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0086] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the functional network element implementing the method described in the above embodiments.

[0087] The computer program product provided in this embodiment can execute all the steps and functions of the data management method provided in any of the above embodiments. The specific functions of the product will not be described in detail here.

[0088] It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered to be within the scope of protection of this invention.

Claims

1. A data management method, characterized in that, The method includes: Determine the keywords for full-text search; The keywords are input into a preset filter for filtering. When the filter passes through the filter, the keyword is queried from the granule corresponding to the filter.

2. The data management method according to claim 1, characterized in that, The determination of keywords for full-text search includes: Based on the query conditions of full-text search, the granule containing the target data is located through a sparse index; Determine whether the query conditions meet the preset keyword conditions; If not, perform a binary search using a sparse index based on the full-text search query conditions to retrieve the corresponding target data; If so, obtain the keywords corresponding to the stated keyword conditions.

3. The data management method according to claim 1, characterized in that, The method further includes: If the filter fails to pass the filter, skip the query operation for the granule corresponding to the filter; Input the keyword into the filter corresponding to other granules and determine whether it passes the filter.

4. The data management method according to claim 1, characterized in that, The step of inputting the keywords into a preset filter for filtering includes: The keyword is calculated using K preset hash functions to obtain K hash values; An encoder consisting of K neurons receives K hash values ​​and calculates the encoder's output. The encoder stores the index value information corresponding to all keywords. The encoding output and the identifier information of the granule corresponding to the filter are input into a preset decoder to determine whether the granule corresponding to the filter contains the target data corresponding to the index value information; When the target data is present, it is determined that the filter passes. If the target data does not exist, it is determined that the filter has failed.

5. The data management method according to claim 3, characterized in that, Before inputting the keyword into the filters corresponding to other granules, the method further includes: Determine if there are filters corresponding to granules for which no filtering operation has been performed; If not, perform a binary search using a sparse index based on the full-text search query conditions to retrieve the corresponding target data; If so, input the keyword into the filter corresponding to the granule that has not undergone filtering.

6. The data management method according to claim 1, characterized in that, The step of querying the keyword from the granule corresponding to the filter includes: Based on the query conditions of the full-text search, a binary search is performed using a sparse index to determine the location of the MRK file to be searched. Obtain the persistent association information of the granule from the mrk file; The corresponding block is retrieved and decompressed from the pre-stored persistent file based on the persistent association information; The decompressed block is loaded into memory, and the target granule is determined based on the positioning information obtained from the mrk file; Based on the query conditions of the keywords, retrieve the corresponding target data from the target granule.

7. A data management device, characterized in that, The device includes: The determination module is used to determine the keywords for full-text search; The filtering module is used to input the keywords into a preset filter for filtering. The query module is used to query the keyword from the granule corresponding to the filter when the filter passes through the filter.

8. A communication device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the data management method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the data management method as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 6.

Images