Method for quickly retrieving messages in message queue

Abstract

The invention relates to the technical field of distribution and storage, and particularly provides a method for quickly retrieving messages in a message queue, which comprises the following steps: an index construction step: constructing at least one corresponding index file for each partition of a theme in the message queue; a write-in step: when a producer produces and additionally writes a message into the partition of the theme, writing an index entry of the message into an index file corresponding to the partition in an additional manner at the same time: an index loading step: when a message query operation is executed on a certain partition of the theme for the first time, writing the index entry of the message into the index file corresponding to the partition; lazy loading the index file corresponding to the partition into a memory; an index updating step: after the index file is loaded to the memory, performing subsequent updating operation on the index file on the memory index file and the disk index file at the same time; and a message query step. The method has the advantage that the corresponding message can be found with fewer disk IO times.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a method for quickly retrieving messages in a message queue. Background Technology

[0002] With the continuous development of the internet industry and the ever-growing scale of data, higher demands are being placed on enterprise applications. To prevent massive traffic surges from overwhelming applications and causing service overload and crashes, the importance of message queues is becoming increasingly apparent. As a crucial component of distributed system architecture, message queues primarily provide functions such as traffic shaping, application decoupling, asynchronous communication, and data synchronization.

[0003] Because message queues are typically designed for high throughput and feature sequential read / write operations, they do not directly provide APIs for querying historical messages, making it difficult to directly determine whether a message with a specific key exists. Therefore, most existing message queue products implement their message query functionality by having one or more consumer instances continuously pull and filter messages.

[0004] The disadvantages of existing technology are: 1. It is impossible to directly determine whether a message exists for a specific key. In production environments, users often need to determine whether messages have been lost due to various issues. However, neither the application layer nor the underlying layer of the message queue provides Put / Get interfaces, making it difficult to directly determine whether a message exists.

[0005] 2. Messages cannot be located with a single disk I / O operation. Due to the sequential read / write nature of message queues, to read a message, its preceding messages must be read sequentially. When the message volume is large, this significantly increases the reading time and the number of disk I / O operations.

[0006] 3. Waste of resources. Because a message queue topic may be shared by multiple users, if they all need to query messages but can only deploy a separate consumer instance, it can easily lead to a waste of resources.

[0007] 4. This may severely impact service performance. If the user deploys a query instance using the same group as the production environment, the query message thread will compete with normal business threads for the partition lock. If the query message instance encounters a problem that prevents the partition lock from being released, it will block the business threads, leading to performance issues.

[0008] Therefore, this application proposes a method for quickly retrieving messages in a message queue. Summary of the Invention

[0009] To achieve the above objectives, the present invention adopts the following technical solution: In one aspect, the present invention provides a method for quickly querying messages in a message queue, comprising the following steps: The index building step involves building at least one corresponding index file for each partition of a topic in the message queue. In the index writing step, when a producer produces and appends a message to a partition of the topic, it simultaneously appends an index entry for that message to the corresponding index file of that partition. Each index entry contains at least: The hash value of the message's key occupies 4 bytes; The message's unique identifier occupies 8 bytes; The starting offset of the message in its corresponding data file occupies 4 bytes; The index loading step involves lazy loading the index file corresponding to a partition of the topic into memory when a message query operation is performed on a partition for the first time. The index update step involves performing subsequent update operations on both the memory index and the disk index file simultaneously after the index file is loaded into memory. The message query step involves searching for messages related to a specific key in a partition of the topic based on the hash value of the key in the memory index, obtaining one or more matching index entries, and quickly locating the message in the corresponding data file based on the starting offset in these index entries, thereby retrieving the message.

[0010] In one optional implementation, the index building step creates an index file for each partition with a unique identifier, which includes at least the topic name and partition number information, to facilitate the management and location of the index file; Each index entry has a fixed total length of 16 bytes, with Keyhash occupying 4 bytes, MID occupying 8 bytes, and Offset occupying 4 bytes, making the index file structure regular and facilitating fast parsing and location. The unique identifier of the message is monotonically increasing and is used to indicate the production order of the message within the partition.

[0011] In one optional implementation, when multiple matching index items are obtained based on Keyhash, messages can be further filtered or sorted according to the range or sorting of MID to obtain messages in a specific time window or order. The lazy loading specifically includes: reading the index file corresponding to the partition from the disk into memory only when the first query request for the partition arrives, and building a data structure in memory that facilitates fast lookup.

[0012] In one optional implementation, the data structure built in memory for fast lookup is a hash table or skip list, wherein the key is Keyhash and the value is a list or set containing one or more MIDs and their corresponding Offsets. Update operations on memory indexes and disk index files adopt a method of first updating the memory index and then updating the disk index file asynchronously or synchronously, in order to balance operation performance and data consistency.

[0013] In one optional implementation, obtaining one or more matching index entries specifically includes: Calculate the hash value of the key to be queried; Search the memory index for all index entries corresponding to the hash value; Returns the MID and Offset information contained in these entries; It also includes deduplication of the returned offset information to ensure that each message is located only once.

[0014] In one optional implementation, the index file also supports queries based on MID ranges, that is, it can find the offset of all messages within a given MID start and end value, and combined with Keyhash queries, it can achieve more flexible message retrieval. When a message is written to an index entry, its timestamp is recorded in the index entry, and the message is filtered based on the timestamp.

[0015] In one alternative implementation, when the service restarts or the memory index is lost, the memory index is reconstructed by reloading the index file on the disk, eliminating the need to traverse all message data files in the partition to rebuild the index, thereby reducing service recovery time. When the memory index is updated, copy-on-write or other concurrency control mechanisms are used to ensure the correctness and performance of query operations during the index update process.

[0016] In one optional implementation, for different messages with the same Keyhash, their index entries are arranged in ascending or descending order of MID in the index file, so that the batch of message offsets obtained in step e are also ordered, which facilitates reading messages in sequence.

[0017] In another aspect, the present invention provides an electronic device comprising: At least one memory stores computer-executable instructions non-transiently; At least one processor, configured to run the computer-executable instructions, The computer-executable instructions are executed by the processor to implement the method for quickly retrieving messages in a message queue as described above.

[0018] A computer-readable storage medium storing computer-executable instructions that, when executed by at least one processor, implement the method for quickly retrieving messages in a message queue as described above.

[0019] Effects of the invention: This invention stores the key and offset mapping relationship of messages in an index file, thus enabling the retrieval of corresponding messages with fewer disk I / O operations. Traditional message query methods, on the other hand, require traversing the message file from the beginning and parsing each message sequentially.

[0020] This invention loads an index file into memory, enabling rapid determination of message existence based on the message's keyhash. Traditional methods for determining message existence in a message queue rely on consuming messages, which is extremely time-consuming.

[0021] The functionality of this invention is embedded in the message queue kernel, requiring no additional work from the user. The message query function proposed in this invention can be integrated into the message queue management platform for user use. In contrast, traditional message query functions require users to resubscribe to a new consumer group and redeploy the consumer instance, a cumbersome process.

[0022] This invention will not affect the running producer-consumer service. Traditional message querying requires deploying consumer instances to consume and then filter messages. This is not only time-consuming, but also affects disk I / O and machine load bandwidth when the message volume is large, thus impacting online services. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a method for quickly retrieving messages in a message queue, as provided in Embodiment 1 of the present invention; Figure 2 This is a block diagram of the electronic device provided in Embodiment 3 of the present invention; Figure 3 This is a block diagram of a computer-readable storage medium provided in Embodiment 3 of the present invention. Detailed Implementation

[0024] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0025] Hereinafter, the terms "first," "second," etc., are used for descriptive convenience only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0026] In this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed mechanical connection, a detachable mechanical connection, or an integral part; or, "connection" can be a direct connection or an indirect connection through an intermediate medium. Furthermore, unless otherwise explicitly specified and limited, the term "coupling" should be interpreted broadly. For example, "coupling" can be a direct electrical connection, such as physical contact and electrical conduction between two components; it can also be understood as an electrical connection between different components in a circuit structure through physical lines capable of transmitting electrical signals, such as copper foil or wires on a printed circuit board (PCB), to transmit electrical signals; or, "coupling" can be an indirect electrical connection between two components through an intermediate medium; or, "coupling" can be an electrical connection between two components in a non-contact manner, such as an electrical connection between two components using capacitive coupling to transmit electrical signals.

[0027] In this embodiment of the invention, directional terms such as "up," "down," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and can change accordingly depending on the orientation of the components in the accompanying drawings.

[0028] Example 1: like Figure 1 As shown, this embodiment of the invention provides a method for quickly querying messages in a message queue, comprising the following steps: Step S100: Index building step, build at least one corresponding index file for each partition of the topic in the message queue; Step S200: Index writing step. When the producer produces and appends a message to the partition of the topic, it simultaneously appends the index entry of the message to the index file corresponding to that partition. Each index entry contains at least: The hash value of the message's key occupies 4 bytes; The message's unique identifier occupies 8 bytes; The starting offset of the message in its corresponding data file occupies 4 bytes; Step S300: Index loading step. When a message query operation is performed on a partition of the topic for the first time, the index file corresponding to that partition is lazy-loaded into memory. Step S400: Index update step. After the index file is loaded into memory, all subsequent update operations on the index file are performed simultaneously on the memory index and the disk index file. Step S500: Message query step. When it is necessary to query messages related to a specific key in a certain partition of the topic, the hash value of the key is used to search in the memory index to obtain one or more matching index entries. The starting offset in these index entries is used to quickly locate the position of the message in the corresponding data file, thereby retrieving the message.

[0029] In the above embodiments, by entering a message key in a module of the message queue management platform's UI, message metadata or message content can be quickly queried. The message queue kernel code includes code for building index files for partitions and storing the mapping relationship between message keys and offsets. The message queue product documentation or user manual contains descriptions related to this application, such as storing the mapping relationship between message keys and offsets in the index file; and finding the message's location in the message queue requiring only one disk I / O. This is designed for message query scenarios in message queues, enabling rapid message location and preventing impact on machine performance and load during message querying, thus avoiding disruption to online business.

[0030] Example 2: like Figure 1 As shown, based on Embodiment 1, the steps provided in this embodiment of the invention include the index building step, which builds an index file for each partition with a unique identifier. This identifier includes at least the topic name and partition number information to facilitate the management and location of the index file. Each index entry has a fixed total length of 16 bytes, with Keyhash occupying 4 bytes, MID occupying 8 bytes, and Offset occupying 4 bytes, making the index file structure regular and facilitating fast parsing and location. The unique identifier of the message is monotonically increasing and is used to indicate the production order of the message within the partition.

[0031] In one optional implementation, when multiple matching index items are obtained based on Keyhash, messages can be further filtered or sorted according to the range or sorting of MID to obtain messages in a specific time window or order. The lazy loading specifically includes: reading the index file corresponding to the partition from the disk into memory only when the first query request for the partition arrives, and building a data structure in memory that facilitates fast lookup.

[0032] In one optional implementation, the data structure built in memory for fast lookup is a hash table or skip list, wherein the key is Keyhash and the value is a list or set containing one or more MIDs and their corresponding Offsets. Update operations on memory indexes and disk index files adopt a method of first updating the memory index and then updating the disk index file asynchronously or synchronously, in order to balance operation performance and data consistency.

[0033] In one optional implementation, obtaining one or more matching index entries specifically includes: Calculate the hash value of the key to be queried; Search the memory index for all index entries corresponding to the hash value; Returns the MID and Offset information contained in these entries; It also includes deduplication of the returned offset information to ensure that each message is located only once.

[0034] In one optional implementation, the index file also supports queries based on MID ranges, that is, it can find the offset of all messages within a given MID start and end value, and combined with Keyhash queries, it can achieve more flexible message retrieval. When a message is written to an index entry, its timestamp is recorded in the index entry, and the message is filtered based on the timestamp.

[0035] In one alternative implementation, when the service restarts or the memory index is lost, the memory index is reconstructed by reloading the index file on the disk, eliminating the need to traverse all message data files in the partition to rebuild the index, thereby reducing service recovery time. When the memory index is updated, copy-on-write or other concurrency control mechanisms are used to ensure the correctness and performance of query operations during the index update process.

[0036] In one optional implementation, for different messages with the same Keyhash, their index entries are arranged in ascending or descending order of MID in the index file, so that the batch of message offsets obtained in step e are also ordered, which facilitates reading messages in sequence.

[0037] In the above embodiments, to quickly locate a specific message in the message queue, an index file needs to be built for each partition of the topic. When a user produces a message, they append the message's keyhash (4 bytes), message ID (MID, 8 bytes), and starting offset (Offset, 4 bytes) to the index file. When querying messages in a partition, they are lazy-loaded into memory (and the index is subsequently updated simultaneously in memory and on disk). By using the provided keyhash, the offset of a batch of messages can be obtained, thus quickly locating the position of that batch of messages on disk.

[0038] To properly cache message metadata, storing the metadata of multiple messages in a single file is a very simple and effective method that facilitates rapid implementation and deployment. (Theoretically, when the service restarts, a machine can reconstruct its memory mapping by reading all partitions, but this is time-consuming because it involves reading a large amount of data from the disk.)

[0039] Most message queues create index files for topic partitions, but these index files are mappings between message indices and the starting offsets of data within the file, not between message keys and the starting offsets of data. Furthermore, the former does not store all the mappings between message indices and starting offsets; instead, it uses a sparse indexing technique, employing binary search to find the specific location of a message within the file. These two types of index files have fundamentally different compositions.

[0040] The specific process of message writing: 1. Assembling the message. After creating a message, the producer needs to assign values ​​to the necessary attributes (such as message key, message properties, message body, etc.). We can define the message structure as follows: StructMessage{ Key: String, Attributes:Map<String,Bytes> , Body:Bytes } The assembled message body content is as follows: Message{ Key: "This is a business key" Attributes:EmptyMap, Body: "Thisismessagebody".getBytes() } 2. Selecting a Broker. Before sending a message, the producer needs to select a broker for writing the message. The specific broker chosen depends on the message queue's load balancing strategy. Common load balancing strategies include: a.RoundRobin: Static round-robin method, which can send messages evenly to each Broker, but does not consider server performance.

[0041] b. WeightedRoundRobin: This weighted round-robin method allows setting different weights for each server, with servers having higher weights receiving more messages.

[0042] c. Hash: Hash methods, specifically hashing message attributes (usually message keys), are a common uniform distribution strategy. Its key characteristic is that messages with the same key will be hashed to the same broker. Within the same broker, messages with the same key will be stored in the same partition after hashing, which is very useful in scenarios requiring message ordering. Common hash algorithms include: murmurhash, md5, sha-1, and sha-256.

[0043] The algorithm for calculating the hash based on the message key to allocate a Broker is as follows: inthash = hash(message.Key); hash=hash>int32::MAX?hash:int32::MIN+1; intbrokerIndex=Math.abs(hash)%brokerSize; The biggest drawback of the hash method is that when a topic scales up or down the number of brokers based on the workload, the change in the number of brokers can cause messages produced later to be unable to be assigned to the same broker even if they have the same key. The usual solution to this problem is to only expand the number of brokers when the topic has no data or all the data has been consumed.

[0044] 3. Partition Selection. When a message is sent to the Broker via a request, the Broker also has an internal load balancing algorithm to determine where the message will ultimately be stored. Common partition load balancing strategies include: a.RoundRobin: Static round-robin method, which can evenly store messages in each partition of the corresponding topic on the Broker, but does not consider partition skew (because in online scenarios, topics often need to expand partitions to support increasingly larger business data volumes, at which point the data volume of the new partition and the old partition will differ greatly).

[0045] b. Hash: When selecting a topic partition for a message, hashing the message key remains a common uniform strategy. Its main characteristic is that the same message key will be hashed to the same partition. This algorithm is typically used in scenarios requiring ordered messages (i.e., the order in which consumers consume messages must match the order in which producers produce them). Common hash algorithms include: murmurhash, md5, sha-1, and sha-256.

[0046] The algorithm for calculating the hash based on the message key to allocate partitions is as follows: inthash = hash(message.Key); hash=hash>int32::MAX?hash:int32::MIN+1; intpartition=Math.abs(hash)%partitionSize; Similar to the problems encountered when expanding the Broker, the biggest drawback of the Hash method is that when the topic is expanded or shrunk based on the workload, the change in the number of partitions will cause messages produced later to be unable to be assigned to the same partition even if they have the same key. The general solution to this problem is to expand the partition only when the topic has no data or all the data has been consumed.

[0047] 4. Message Serialization. Serialization refers to converting a readable message structure into a series of byte arrays (usually accompanied by encryption and compression processes; however, whether encryption and compression are included here is irrelevant to the explanation, so the description of encryption and compression is omitted). After serializing the message, the broker stores it in a partition. Serialization can generally be done using serialization tools (such as protobuf, json, etc.) or a custom serializer. Here, to accommodate the statement below that the index does not need to store the message disk size, the message serialization process is briefly described.

[0048] a. Request a cache, with the first 0 to 3 bytes (4 bytes in length) set as reserved bytes to indicate the size of the message to be written to disk.

[0049] b. Bytes 4 and 5 are used to represent the message key length n, and the following n bytes are requested based on this value n.

[0050] c. The (n+4)th to (n+5)th bytes are used to represent the size of the message attribute Map. To simplify the description, both bytes are set to 0, indicating that the message attribute is empty.

[0051] d. Bytes n+7 to n+10 are used to represent the message body size m, and the following m bytes are requested based on this value m.

[0052] e. Calculate the total number of bytes occupied by all message attributes and assign a value to the first four bytes of the cache. Here, the first four bytes should store the value n+m+8.

[0053] It can be noted here that when a message is written to disk, 4 bytes are appended to the beginning of the byte sequence to store the message size. When querying a message, the offset is used to locate the message position. The first 4 bytes read are the message size, and then size bytes are read to obtain the complete message.

[0054] 5. Update the index. The broker generates index entries based on the serialized message, that is, assigns values ​​to each field of the index entry, appends the serialized message to the index file, and updates the in-memory index. The process of serializing index entries is similar to the process of serializing messages, and will not be described in detail here.

[0055] 6. Responding to the producer: This refers to the Broker returning a response to the client after storing the message on disk and updating the index file, allowing the client to continue sending requests for producing messages.

[0056] Step 5 here does not need to be in the same thread as writing the message; it can be updated asynchronously to reduce the impact on the production process.

[0057] The specific process for generating index entries: 1. Generate a Keyhash value based on the message key provided by the user. Since the index file and the topic partitions are in one-to-one correspondence, the hash calculation algorithm does not need to include location logic; it can be calculated directly using a hash algorithm. Taking the MD5 algorithm as an example, the calculation is as follows: key_hash=md5_hash(message.Key); 2. Calculate the Offset value. Because message queues are characterized by sequential read and write operations, and messages are stored tightly and contiguously, the Offset value can be calculated directly based on the size of the message files in that partition.

[0058] offset = partitionFileSize; 3. Calculate the IndexOffset value. During the construction of the in-memory index, expired messages and index entries in the log files and index files are periodically deleted. Therefore, when scanning the index file, the index of each index entry is added to the index entries. Similar to the method of calculating the offset, the size of the IndexOffset is the size of the index file.

[0059] indexOffset = indexFileSize; 4. When constructing index entries, it is not necessary to calculate the size of the message written to disk. As mentioned above, in the message file, after locating the message based on the offset, the first 4 bytes of the message bytes are the size of the message.

[0060] 5. The message ID (MID) is automatically assigned by the system and is generally generated based on attributes such as partition number and number of messages.

[0061] The steps for generating index entries are as follows: Figure 3 As shown: When multiple messages share the same key, there is no difference between appending to the index file and using a regular key. When duplicate keyhashes are loaded into memory, they are aggregated. In this case, each keyhash corresponds to a list, and each item in the list contains MID, Offset, and Size.

[0062] The specific process for message retrieval: 1. Assembly Request. In the message query module of the message queue management platform, enter the topic and message key information.

[0063] 2. Based on the Broker list information of the topic in the metadata, hash the topic to find its corresponding Broker and send a query request to it. The hash calculation method is as follows: inthash = hash(message.Key); hash=hash>int32::MAX?hash:int32::MIN+1; intbrokerIndex=Math.abs(hash)%brokerSize; 3. After receiving a query request, the broker uses the message key to perform a hash to locate the partition where the message is located and find the corresponding index file. The hash calculation method is as follows: inthash = hash(message.Key); hash=hash>int32::MAX?hash:int32::MIN+1; intpartitionIndex=Math.abs(hash)%partitionSize; 4. After locating the index file containing the message, calculate its hash within the index file and check if the message exists in the corresponding memory index. If it does not exist, return directly. The Keyhash calculation method is as follows: key_hash=md5_hash(message.Key); 5. If the message exists, locate the message in the partition based on the offset of the index entry, then obtain the message size based on the first 4 bytes read, and then retrieve the complete message byte array based on the message size.

[0064] 6. Respond to query requests. Encapsulate the retrieved message byte array into a response and send it to the client instance used by the management terminal. The management terminal then deserializes the message and displays it on the management terminal.

[0065] Because messages in a message queue are not permanently stored, they are typically retained for 7 days. Once all data has been consumed by consumers, the system automatically cleans up the data, requiring an update to the index file. Since message queue cleanup often involves deleting data from a contiguous block of memory, after deleting a message, the message key and MID with the largest offset of the deleted message must be retained. Based on these, the system searches for the corresponding index entry's offset in memory, then truncates the index file. If the corresponding index entry is not found in memory, the memory index is scanned based on the largest offset. If the message index corresponding to the index entry is smaller than the offset value, the index entry is deleted.

[0066] Based on Example 1, this embodiment adds a timestamp field to the index item, which occupies 4 bytes (or 8 bytes, such as Unix timestamps accurate to milliseconds).

[0067] The index structure then becomes: Key hash (4B) + MID (8B) + Offset (4B) + Timestamp (4B) = 20 bytes.

[0068] In the message retrieval process, in addition to providing the key, users can also specify a time range. After the system finds an index item that matches the key hash, it can further filter the offset that meets the time condition based on the timestamp, thereby retrieving messages more accurately.

[0069] This embodiment, building upon Embodiment 1, uses a skip list instead of a hash table to store the mapping from key hashes to MID-Offset lists in the memory index. Skip lists may offer better performance or features in certain scenarios (such as when range queries of key hashes are required, or when there are specific requirements for memory layout). Furthermore, MID-Offset lists with the same key hash can be sorted using skip lists to support more complex range queries (such as querying messages whose MID falls within a certain range).

[0070] This embodiment focuses on concurrency control of the index. When multiple producers simultaneously write messages to the same partition, or when there are simultaneous read and write operations: For appending to disk index files, since messages are appended sequentially within a partition, the writing of index entries is also sequential. The writing order and integrity can be guaranteed through simple file locks or atomic operations.

[0071] For updates to in-memory indexes (such as hash tables), a copy-on-write technique is used. When an update to the in-memory index is needed, a new version is created for modification, while read operations still use the old version. After modification, the pointer is atomically switched. This ensures lock-free and efficient read operations while guaranteeing the safety of write operations.

[0072] Example 3: Figure 2 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present invention is shown.

[0073] The electronic device may include a central processing unit / microprocessor / main control chip, etc. 4; and a storage medium 5, coupled to the central processing unit / microprocessor / main control chip, etc. 4, and storing computer-executable instructions therein for performing the steps of various methods of embodiments of the present invention when executed by the processor.

[0074] The central processing unit / microprocessor / main control chip, etc., can include, but are not limited to, one or more processors or microprocessors.

[0075] Storage medium 5 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0076] In addition, the electronic device may also include (but is not limited to) a data bus 6, an input / output bus / external bus / device bus 7, a display 8, and input / output devices 9 (e.g., keyboard, mouse, speaker, etc.).

[0077] The central processing unit / microprocessor / main control chip, etc. 4 can communicate with external devices (8, 9, etc.) via I / O bus 7 through wired or wireless network (not shown).

[0078] The storage medium 5 may also store at least one computer-executable instruction for performing the steps of various functions and / or methods in the embodiments described herein when the central processing unit / microprocessor / main control chip, etc., 4 is running.

[0079] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0080] Figure 3 A schematic diagram of a computer-readable storage medium according to an embodiment of the present invention is shown.

[0081] like Figure 3 As shown, the non-transitory computer-readable storage medium 11 stores instructions, such as computer-readable instructions 10. When the computer-readable instructions 10 are executed by a processor, the various methods described above can be performed. The non-transitory computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-transitory non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, the non-transitory computer-readable storage medium 11 can be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions 10 stored on the computer-readable storage medium 11, the various methods described above can be performed.

[0082] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0085] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods of the various embodiments of this invention through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0086] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for quickly retrieving messages in a message queue, characterized in that, Includes the following steps: The index building step involves building at least one corresponding index file for each partition of a topic in the message queue. In the index writing step, when a producer produces and appends a message to a partition of the topic, it simultaneously appends an index entry for that message to the corresponding index file of that partition. Each index entry contains at least: The hash value of the message's key occupies 4 bytes; The message's unique identifier occupies 8 bytes; The starting offset of the message in its corresponding data file occupies 4 bytes; The index loading step involves lazy loading the index file corresponding to a partition of the topic into memory when a message query operation is performed on a partition for the first time. The index update step involves performing subsequent update operations on both the memory index and the disk index file simultaneously after the index file is loaded into memory. The message query step involves searching for messages related to a specific key in a partition of the topic based on the hash value of the key in the memory index, obtaining one or more matching index entries, and quickly locating the message in the corresponding data file based on the starting offset in these index entries, thereby retrieving the message.

2. The method for quickly retrieving messages in a message queue as described in claim 1, characterized in that, The index building step creates a unique identifier for each partition's index file, which includes at least the topic name and partition number information to facilitate the management and location of the index file; Each index entry has a fixed total length of 16 bytes, with Keyhash occupying 4 bytes, MID occupying 8 bytes, and Offset occupying 4 bytes, making the index file structure regular and facilitating fast parsing and location. The unique identifier of the message is monotonically increasing and is used to indicate the production order of the message within the partition.

3. The method for quickly retrieving messages in a message queue as described in claim 1, characterized in that, When multiple matching index items are obtained based on Keyhash, messages can be further filtered or sorted according to the range or sorting of MID to obtain messages in a specific time window or order. The lazy loading specifically includes: reading the index file corresponding to the partition from the disk into memory only when the first query request for the partition arrives, and building a data structure in memory that facilitates fast lookup.

4. The method for quickly retrieving messages in a message queue as described in claim 1, characterized in that, The data structure built in memory for fast lookup is a hash table or skip list, where Keyhash is the key and a list or set containing one or more MIDs and their corresponding Offsets is the value. Update operations on memory indexes and disk index files adopt a method of first updating the memory index and then updating the disk index file asynchronously or synchronously, in order to balance operation performance and data consistency.

5. The method for quickly retrieving messages in a message queue as described in claim 1, characterized in that, Retrieving one or more matching index items specifically includes: Calculate the hash value of the key to be queried; Search the memory index for all index entries corresponding to the hash value; Returns the MID and Offset information contained in these entries; It also includes deduplication of the returned offset information to ensure that each message is located only once.

6. The method for quickly retrieving messages in a message queue as described in claim 1, characterized in that, The index file also supports queries based on MID ranges, which means that the offset of all messages within a given MID start and end value can be found. Combined with Keyhash queries, more flexible message retrieval can be achieved. When a message is written to an index entry, its timestamp is recorded in the index entry, and the message is filtered based on the timestamp.

7. The method for quickly retrieving messages in a message queue as described in claim 1, characterized in that, When the service restarts or the memory index is lost, the memory index is reconstructed by reloading the index file on the disk. This eliminates the need to traverse all message data files in the partition to rebuild the index, thereby reducing service recovery time. When the memory index is updated, copy-on-write or other concurrency control mechanisms are used to ensure the correctness and performance of query operations during the index update process.

8. The method for quickly retrieving messages in a message queue as described in claim 1, characterized in that, For different messages with the same Keyhash, their index entries are arranged in ascending or descending order of MID in the index file, so that the batch of message offsets obtained in step e are also ordered, making it convenient to read messages in sequence.

9. An electronic device, comprising: At least one memory stores computer-executable instructions non-transiently; At least one processor, configured to run the computer-executable instructions, The computer-executable instructions are executed by the processor to provide a method for quickly retrieving messages in a message queue according to any one of claims 1-8.

10. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer-executable instructions that, when executed by at least one processor, implement a method for quickly retrieving messages in a message queue according to any one of claims 1-8.

Images