A pulsar-based log indexing method
Patent Information
- Application Number
- CN202211515993.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-30
- Publication Date
- 2026-10-30
- Estimated Expiration
- 2042-11-30
AI Technical Summary
[0039]进一步地，上述一种基于Pulsar的日志索引方法方法还包括：给索引内存的缓存配置最大的使用量。
Smart Images
Figure CN115729890B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of middleware technology, and in particular to a log indexing method based on Pulsar. Background Technology
[0002] Apache Pulsar is a top-level project of the Apache Software Foundation. It is a next-generation cloud-native distributed messaging platform that integrates messaging, storage, and lightweight functional computing. It adopts a computing and storage separation architecture design, supports multi-tenancy, persistent storage, and cross-regional data replication across multiple data centers, and has streaming data storage characteristics such as strong consistency, high throughput, low latency, and high scalability.
[0003] Apache Bookkeeper is a distributed, inherently scalable logging system where all data is written append-only, and data already written cannot be modified. Based on Bookkeeper, Pulsar appends messages to distributed nodes as a log, continuously reading data from the starting offset.
[0004] Apache Bookkeeper Ledge is the log abstraction of Apache Bookkeeper. Each Ledge can be regarded as an independent log file, providing only sequential query and append write interfaces. Based on the storage interface of Bookkeeper distributed log, which only allows append write and sequential read data, this application introduces an indexing method based on hash value partitioning. Summary of the Invention
[0005] To address the aforementioned issues, this application provides a log indexing method based on Pulsar.
[0006] This application provides a log indexing method based on Pulsar, including:
[0007] When writing to an index, the hash value of the index key is calculated;
[0008] The index data is partitioned and stored according to the HASH value to obtain index data for multiple partitions;
[0009] For the same partition in multiple partitions, the index data is sharded and stored in the corresponding cloud-native distributed log Ledge, and the corresponding index memory is updated.
[0010] When querying the index, load the data from the target log Ledge into the index memory.
[0011] Specifically, in this embodiment, the index data is stored in a hash partition, so that the index data of a single partition is automatically split into sufficiently small pieces. At the same time, a simplified storage format is adopted to store the index data in the index memory, so that the index data can be cached in memory and used directly, realizing fast retrieval based on KEY value, message attribute or message content.
[0012] Furthermore, the above-mentioned log indexing method based on Pulsar, which involves sharding and storing index data for the same partition across multiple partitions into corresponding cloud-native distributed log Ledges, also includes:
[0013] If the difference between the creation time of the current Ledge and the current time is greater than a preset threshold;
[0014] Close the current Ledge shard and create a new Ledge shard to store index data.
[0015] Specifically, in this embodiment, the index data of the same partition is sharded and stored based on time through the above steps, thereby ensuring that the data volume of a single shard ledge is small enough to be fully loaded into memory, thus achieving the effect of sharding.
[0016] Furthermore, the above-mentioned log indexing method based on Pulsar, which involves sharding and storing index data for the same partition across multiple partitions into corresponding cloud-native distributed log Ledges, also includes:
[0017] If the data volume of the current Ledge segment reaches the preset threshold;
[0018] Close the current Ledge shard and create a new Ledge shard to store index data.
[0019] Specifically, in this embodiment, the index data of the same partition is sharded and stored based on the data volume through the above two steps, thereby ensuring that the data volume of a single shard ledge is small enough to be fully loaded into memory, thus achieving the effect of sharding.
[0020] Furthermore, the aforementioned Pulsar-based log indexing method also includes:
[0021] Obtain the index data volume and hash distribution for each partition;
[0022] Determine whether the index data volume and / or hash distribution of each partition meet the preset conditions;
[0023] If the amount of index data and / or the hash distribution of each partition reach the preset conditions, the corresponding partition will be automatically split.
[0024] Specifically, in this embodiment of the application, when the large amount of data and the hash key are concentrated in the same partition, in order to improve query efficiency and reduce the hash key and number of individual partitions, the index data volume and hash key distribution of the index partition will be statistically analyzed to determine whether the preset conditions are met, and automatic splitting will be performed on these partitions.
[0025] Furthermore, the aforementioned Pulsar-based log indexing method also includes:
[0026] Periodically check if there are any timed-out index data fragments within the index data partition;
[0027] If there are expired index data fragments within the index data partition, delete the expired index data fragments.
[0028] Specifically, in this embodiment of the application, the periodic check for timed-out index data shards within the index data partition is achieved by starting a worker thread. The worker thread periodically checks each index partition for timed-out index data shards, and deletes these index data shards if they time out.
[0029] Furthermore, the aforementioned Pulsar-based log indexing method also includes the following when querying the index:
[0030] Calculate the hash value of the query conditions;
[0031] The index data partition is determined based on the HASH value of the query conditions;
[0032] The location of the index data shards is determined based on the time and ID.
[0033] Specifically, in this embodiment of the application, the above steps are used to calculate the HASH value of the query conditions, find the index partition based on the HASH value, and then find the location of the index data fragments based on the ID range and time.
[0034] Furthermore, the aforementioned Pulsar-based log indexing method also includes the following when querying the index:
[0035] Determine whether the query conditions include an index;
[0036] If the query conditions include an index, the query will be performed using the index; if the query conditions do not include an index, a full scan query will be performed using the query conditions.
[0037] Specifically, in this embodiment, the above method is implemented by extending the external API interface, such as adding a query interface based on content or KEY conditions. If the query conditions include an index, the index interface is called to perform the query; otherwise, the query interface based on content or KEY conditions is called to perform a full scan query, thereby achieving fast retrieval based on KEY value, message attribute, or message content.
[0038] Furthermore, in the above-mentioned log indexing method based on Pulsar, the KEY value of the index memory is a combination of {INDEX}_{NAME}_{BUCKET}_{LEDGE_ID}.
[0039] Furthermore, the aforementioned Pulsar-based log indexing method also includes configuring a maximum usage for the index memory cache.
[0040] Furthermore, in the aforementioned Pulsar-based log indexing method, the index memory is updated based on a least recently used strategy.
[0041] The advantages of this application's embodiments are as follows: This application uses HASH value index data for fine partitioning, and stores the index data of each partition in pages according to the data volume and / or time, so that the data of a single partition is automatically split into sufficiently small parts; by adopting a simplified storage format to update the index data to the corresponding index memory, each index data requires only very low storage space; by using memory caching, fast and efficient data retrieval is achieved; in addition, cloud-native distributed logs support automatic scaling, simplifying the cost of operation and maintenance deployment; newly added index messages will automatically add index data and can be queried immediately. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 1 ;
[0044] Figure 2 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 2 ;
[0045] Figure 3 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 3 ;
[0046] Figure 4 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 4 ;
[0047] Figure 5 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 5 ;
[0048] Figure 6 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 6 ;
[0049] Figure 7 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 7 . Detailed Implementation
[0050] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.
[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0052] Figure 1 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 1 .
[0053] This application provides a log indexing method based on Pulsar, combined with... Figure 1 It includes four steps, S101 to S104:
[0054] S101: When writing to the index, calculate the HASH value of the index KEY;
[0055] S102: The index data is partitioned and stored according to the HASH value to obtain index data of multiple partitions;
[0056] S103: Divide the index data of the same partition in multiple partitions into shards and store them in the corresponding cloud-native distributed log Ledge, and update the corresponding index memory;
[0057] S104: When querying the index, load the data of the target log Ledge into the index memory.
[0058] Specifically, in this embodiment, the index data is stored in a HASH partition through the four steps S101 to S104 described above. This allows the index data of a single partition to be automatically divided into sufficiently small segments, and a simplified storage format is adopted to store it in the index memory. As a result, the index data can be cached in memory and used directly to achieve fast retrieval.
[0059] In one possible implementation, when performing an index-based query, if the index is not present in memory, it is loaded from the cloud-native distributed log Ledge.
[0060] Figure 2 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 2 .
[0061] Furthermore, in the aforementioned Pulsar-based log indexing method, index data for the same partition across multiple partitions is sharded and stored in the corresponding cloud-native distributed log Ledge, combined with... Figure 2 It also includes two steps, S201 and S202:
[0062] S201: If the difference between the creation time of the current Ledge and the current time is greater than a preset threshold;
[0063] S202: Close the current Ledge shard and create a new Ledge shard to store index data.
[0064] Specifically, in this embodiment of the application, the index data of the same partition is sharded and saved based on time through the above two steps S201 to S202, thereby ensuring that the data volume of a single shard ledge is small enough to be fully loaded into memory, thus achieving the effect of sharding.
[0065] Figure 3 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 3 .
[0066] Furthermore, in the aforementioned Pulsar-based log indexing method, index data for the same partition across multiple partitions is sharded and stored in the corresponding cloud-native distributed log Ledge, combined with... Figure 3 It also includes two steps, S301 and S302:
[0067] S301: If the data volume of the current Ledge segment reaches the preset threshold;
[0068] S302: Close the current Ledge shard and create a new Ledge shard to store index data.
[0069] Specifically, in this embodiment of the application, the index data of the same partition is sharded and stored based on the data volume through the above two steps S301 to S302, thereby ensuring that the data volume of a single shard ledge is small enough to be fully loaded into memory, thus achieving the effect of sharding.
[0070] Figure 4 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 4 .
[0071] Furthermore, the above-mentioned log indexing method based on Pulsar, combined with... Figure 4 It also includes three steps: S401 to S403.
[0072] S401: Get the index data volume and hash distribution for each partition;
[0073] S402: Determine whether the index data volume and / or HASH distribution of each partition meet the preset conditions;
[0074] S403: If the index data volume and / or HASH distribution of each partition reach the preset conditions, the corresponding partition will be automatically split.
[0075] Specifically, in this embodiment of the application, when large amounts of data and hash keys are concentrated in the same partition through the three steps S401 to S403, in order to improve query efficiency and reduce the hash keys and number of individual partitions, the index data volume and hash key distribution of the index partition will be statistically analyzed to determine whether the preset conditions are met, and automatic splitting will be performed on these partitions.
[0076] Figure 5 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 5 .
[0077] Furthermore, the above-mentioned log indexing method based on Pulsar, combined with... Figure 5 It also includes two steps, S501 and S502:
[0078] S501: Periodically check whether there are any timed-out index data fragments within the index data partition;
[0079] S502: If there are timed-out index data fragments in the index data partition, delete the timed-out index data fragments.
[0080] Specifically, in this embodiment of the application, the periodic check for timed-out index data fragments in the above two steps S501 to S502 is achieved by starting a worker thread. The worker thread periodically checks each index partition for timed-out index data fragments, and deletes these index data fragments if they time out.
[0081] Figure 6 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 6 .
[0082] Furthermore, the aforementioned Pulsar-based log indexing method, when querying the index, combines... Figure 6 It also includes three steps, S601 to S603:
[0083] S601: Calculate the HASH value of the query conditions;
[0084] S602: Determine the index data partition based on the HASH value of the query conditions;
[0085] S603: Determine the location of the index data shard based on the time and ID.
[0086] Specifically, in this embodiment of the application, the HASH value of the query conditions is calculated through the above three steps S601 to S603, the index partition is found based on the HASH value, and the location of the index data fragment is further found according to the ID range and time.
[0087] Figure 7 A schematic diagram of a Pulsar-based log indexing method provided in this application embodiment. Figure 7 .
[0088] Furthermore, the aforementioned Pulsar-based log indexing method, when querying the index, combines... Figure 7 It also includes two steps, S701 and S702:
[0089] S701: Determine whether the query conditions include an index;
[0090] S702: If the query conditions include an index, the query will be performed using the index; if the query conditions do not include an index, a full scan query will be performed using the query conditions.
[0091] Specifically, in this embodiment of the application, the method for implementing the above two steps S701 to S702 is achieved by extending the external API interface, such as adding a query interface based on content or KEY conditions. If the query conditions include an index, the index interface is called to perform the query; otherwise, the query interface based on content or KEY conditions is called to perform a full scan query, thereby achieving fast retrieval based on KEY value, message attribute, or message content.
[0092] Furthermore, in the above-mentioned log indexing method based on Pulsar, the KEY value of the index memory is a combination of {INDEX}_{NAME}_{BUCKET}_{LEDGE_ID}.
[0093] Specifically, in this embodiment, a fixed-size MAP is used as the index memory, and the KEY value of the index memory is a combination of {INDEX}_{NAME}_{BUCKET}_{LEDGE_ID}.
[0094] Furthermore, the aforementioned Pulsar-based log indexing method also includes configuring a maximum usage for the index memory cache.
[0095] Specifically, in this embodiment of the application, the amount of index data stored in the index memory is made more flexible by configuring the maximum usage of the index memory cache.
[0096] Furthermore, in the aforementioned Pulsar-based log indexing method, the index memory is updated based on a least recently used strategy.
[0097] Specifically, in this embodiment, the index memory is updated using a least recently used strategy, which makes retrieval faster.
[0098] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this application and form different embodiments.
[0099] Those skilled in the art will understand that the descriptions of the various embodiments have different focuses, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0100] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A log indexing method based on Pulsar, characterized in that, include: When writing to an index, calculate the hash value of the index key; The index data is partitioned and stored according to the HASH value to obtain index data for multiple partitions; For the same partition in multiple partitions, the index data is sharded and stored in the corresponding cloud-native distributed log Ledge, and the corresponding index memory is updated. When querying the index, load the data from the target log Ledge into the index memory; The step of sharding and storing index data for the same partition across multiple partitions into corresponding cloud-native distributed log Ledges also includes: If the difference between the creation time of the current Ledge and the current time is greater than a preset threshold; Close the current Ledge shard and create a new Ledge shard to store index data; The step of sharding and storing index data for the same partition across multiple partitions into corresponding cloud-native distributed log Ledges also includes: If the data volume of the current Ledge segment reaches the preset threshold; Close the current Ledge shard and create a new Ledge shard to store index data.
2. The log indexing method based on Pulsar according to claim 1, characterized in that, The method further includes: Obtain the index data volume and hash distribution for each partition; Determine whether the index data volume and / or HASH distribution of each partition meet the preset conditions; If the index data volume and / or HASH distribution of each partition reach a preset condition, the corresponding partition will be automatically split.
3. The log indexing method based on Pulsar according to claim 1, characterized in that, The method further includes: Periodically check if there are any timed-out index data fragments within the index data partition; If there are expired index data fragments within the index data partition, delete the expired index data fragments.
4. The log indexing method based on Pulsar according to claim 1, characterized in that, When querying the index, the method further includes: Calculate the hash value of the query conditions; The index data partition is determined based on the HASH value of the query conditions; The location of the index data shards is determined based on the time and ID.
5. The log indexing method based on Pulsar according to claim 1, characterized in that, When querying the index, the method further includes: Determine whether the query conditions include an index; If the query conditions include an index, the query is performed using the index; if the query conditions do not include an index, a full scan query is performed using the query conditions.
6. The log indexing method based on Pulsar according to claim 1, characterized in that, The KEY value of the index memory is a combination of {INDEX}_{NAME}_{BUCKET}_{LEDGE_ID}.
7. The log indexing method based on Pulsar according to claim 1, characterized in that, The method also includes configuring a maximum usage for the cache of the index memory.
8. A log indexing method based on Pulsar according to claim 1, characterized in that, The index memory is updated based on the least recently used strategy.