Message queue processing method, device and message queue management system

Abstract

Embodiments of the present disclosure disclose a message queue processing method, apparatus, and message queue management system. The message queue processing method includes: establishing a first buffer area for caching hot messages and a second buffer area for caching cold messages in the virtual memory of a message queue storage node; determining an index value of the message requested by the read request according to the read request for the message queue; determining whether to read the requested message from the first buffer area or from the second buffer area according to the index value; and returning the message requested by the read request from the first buffer area or the second buffer area, which can implement caching of hot and cold message data in different buffer areas, physically separating the read caches of hot and cold message data, reducing the occupation of hard disk I / O by a large number of cold reads in the system, and moreover, the cold read access to the hard disk can be reduced by controlling the second buffer area to prevent performance bottlenecks in hard disk reading and writing.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and particularly to a message queue processing method, apparatus, and message queue management system. Background Art

[0002] Current message-oriented systems (message middleware) are basic software for sending and receiving messages in distributed systems. Message middleware, also known as a message queue, refers to using an efficient and reliable message passing mechanism to perform platform-independent data communication and integrating distributed systems based on data communication. A message queue can extend the communication of processes in a distributed environment. However, the message queue products of the prior art often face cold and hot data reading and writing problems in scenarios such as stream computing and consumption of a large amount of historical data, ultimately resulting in performance bottlenecks. For example, the problems faced by a message queue of the prior art are as follows:

[0003] · When the message queue is consumed with a delay, it will cause a performance bottleneck in hard disk reading and writing. Especially in the scenario of multi-queue delayed consumption, serious performance problems and service quality problems will occur.

[0004] · In the streaming mode (or called the stream mode) of the message queue, there will be a large number of cold reads (reading cold messages), which will seriously affect the performance when writing messages. Cold reads will occupy a large amount of memory resources and seriously affect follow-up consumption (hot message consumption) and message writing (resource competition in the page cache area Page Cache and hard disk IO competition). Moreover, the cold read has a long latency and low throughput.

[0005] For another example, another message queue of the prior art does not consider the separation of cold and hot message data reading and writing in the stream computing scenario, and there are the following technical drawbacks when reading a large amount of historical data:

[0006] · When the starting position of the message pulled by the client from the server is very small due to various reasons, the data pulled by the client at this time will not hit the Page Cache, and the data is stored on the hard disk. A large amount of data will be cold-read from the hard disk, resulting in low performance of message data reading, further affecting the speed of message consumption, and the overall system performance is seriously affected.

[0007] · When a large amount of data is read from the hard disk, it will occupy the hard disk IO, causing an IO bottleneck, which in turn affects the reading and writing of hot data of the storage node itself, and ultimately leads to a decline in the overall reading and writing performance of the system.

[0008] · Since a large amount of the data read is cold data, after being read into the memory, it will occupy a large amount of the system's PageCache, competing with the writing of messages and the reading of some data in the hot read (hot message reading) for the cache, thus affecting the capacity of the Page Cache originally belonging to hot data and ultimately resulting in a performance bottleneck in the overall reading and writing.

[0009] Therefore, the message queues in the prior art do not consider the separation of reading and writing of cold and hot message data, resulting in a performance bottleneck in the message reading and writing of the system using the message queue. Summary of the Invention

[0010] To solve the problems in the related art, the embodiments of the present disclosure provide a message queue processing method, an apparatus, and a message queue management system, which use different buffer areas to cache cold and hot message data, physically separate the read caches of cold and hot message data, overcome the impact of a large amount of cold message data reading on the overall system performance, improve the overall reading and writing performance and throughput of the system, and provide stronger support for the reading of a large amount of cold message data in the flow mode in the message scenario.

[0011] In a first aspect, the embodiments of the present disclosure provide a message queue processing method, where the method includes:

[0012] Establish a first buffer area for caching hot messages and a second buffer area for caching cold messages in the virtual memory of the message queue storage node;

[0013] According to the read request for the message queue, determine the index value of the message requested by the read request;

[0014] According to the index value, determine whether to read the requested message from the first buffer area or the second buffer area;

[0015] Return the message requested by the read request from the first buffer area or the second buffer area.

[0016] In combination with the first aspect, in the first implementation manner of the first aspect of the present disclosure, the index value of the message is the offset of the message in the message queue.

[0017] Wherein, the step of determining the index value of the message requested by the read request according to the read request for the message queue includes:

[0018] According to the read request for the message queue, determine the offset of the message requested by the read request.

[0019] Wherein, the step of determining whether to read the requested message from the first buffer area or the second buffer area according to the index value includes:

[0020] Determine whether to read the requested message from the first buffer or the second buffer based on the offset of the message requested by the read request and the maximum offset of the message queue.

[0021] Combined with the first implementation manner of the first aspect, in the second implementation manner of the first aspect of the present disclosure, the determining whether to read the requested message from the first buffer or the second buffer based on the offset of the message requested by the read request and the maximum offset of the message queue includes:

[0022] Determine to read the requested message from the first buffer if the difference between the offset of the message requested by the read request and the maximum offset of the message queue is less than or equal to a preset threshold, or

[0023] Determine to read the requested message from the second buffer if the difference between the offset of the message requested by the read request and the maximum offset of the message queue is greater than the preset threshold.

[0024] Combined with the first aspect or the first implementation manner of the first aspect, in the third implementation manner of the first aspect of the present disclosure, the second buffer only accepts reads from the message consumers of the message queue and does not accept writes from the message producers of the message queue.

[0025] Combined with the first aspect or the first implementation manner of the first aspect, in the fourth implementation manner of the first aspect of the present disclosure, the first buffer is a page cache (Page Cache), and the second buffer is a least recently used cache (LRU Cache).

[0026] Combined with the fourth implementation manner of the first aspect, in the fifth implementation manner of the first aspect of the present disclosure, the returning of the message requested by the read request from the first buffer or the second buffer includes:

[0027] In the case where it is determined to read the requested message from the second buffer and the message does not exist in the second buffer, read the message requested by the read request from the non-volatile memory of the message queue storage node into the least recently used cache (LRU Cache) in a direct I / O manner to return the requested message, or

[0028] In the case where it is determined to read the requested message from the first buffer and the message does not exist in the first buffer, a page fault occurs, and the message requested by the read request is read from the non-volatile memory of the message queue storage node into the page cache (Page Cache) to return the requested message.

[0029] In combination with the first aspect or the first implementation manner of the first aspect, in the sixth implementation manner of the first aspect, the operation of returning a message from the first buffer is performed by a first thread, and the operation of returning a message from the second buffer is performed by a second thread that is in a different thread pool from the first thread, so that the reading threads for the first buffer and the second buffer are separated.

[0030] In a second aspect, an embodiment of the present disclosure provides a message queue processing device, where the device includes:

[0031] A buffer establishment module, configured to establish a first buffer for caching hot messages and a second buffer for caching cold messages in the virtual memory of a message queue storage node

[0032] An index value determination module, configured to determine an index value of a message requested by the read request according to the read request for the message queue;

[0033] A message cache location determination module, configured to determine whether to read the requested message from the first buffer or the second buffer according to the index value;

[0034] A message return module, configured to return the message requested by the read request from the first buffer or the second buffer.

[0035] In a third aspect, an embodiment of the present disclosure provides a message queue management system, where the system includes:

[0036] A message queue client, a message queue server, and a message queue storage node,

[0037] where the message queue server establishes a first buffer for caching hot messages and a second buffer for caching cold messages in the virtual memory of the message queue storage node,

[0038] According to the read request of the message queue client for the message queue, the message queue server determines an index value of the message requested by the read request,

[0039] The message queue server determines whether to read the requested message from the first buffer or the second buffer according to the index value,

[0040] The message queue server returns the message requested by the read request from the first buffer or the second buffer to the message queue client.

[0041] Fourthly, an embodiment of the present disclosure provides a computer program product, including computer instructions, which when executed by a processor, implement the methods described in the first aspect, the first implementation manner to the sixth implementation manner of the first aspect.

[0042] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects:

[0043] According to the technical solution provided by the embodiment of the present disclosure, a first buffer for caching hot messages and a second buffer for caching cold messages are established in the virtual memory of the message queue storage node; according to a read request for the message queue, an index value of the message requested by the read request is determined; according to the index value, it is determined whether to read the requested message from the first buffer or the second buffer; the message requested by the read request is returned from the first buffer or the second buffer, so that different buffers can be used to cache hot and cold message data, physically separating the read caches of hot and cold message data, reducing the occupation of hard disk IO by a large number of cold reads in the system, and the cold read access to the hard disk can be reduced by controlling the second buffer to prevent performance bottlenecks in hard disk reading and writing. Therefore, the solution of the embodiment of the present disclosure overcomes the impact of a large number of cold message data reads on the overall system performance, improves the overall read and write performance and throughput of the system, and provides stronger support for reading a large number of cold message data in the stream mode in the message scenario.

[0044] According to the technical solution provided by the embodiment of the present disclosure, the index value of the message is the offset of the message in the message queue, where determining the index value of the message requested by the read request according to the read request for the message queue includes: determining the offset of the message requested by the read request according to the read request for the message queue, and determining whether to read the requested message from the first buffer or the second buffer according to the index value includes: determining whether to read the requested message from the first buffer or the second buffer according to the offset of the message requested by the read request and the maximum offset of the message queue, which can easily determine whether the requested message is a hot message cached in the first buffer or a cold message cached in the second buffer, reducing the occupation of hard disk IO by a large number of cold reads in the system, and the cold read access to the hard disk can be reduced by controlling the second buffer to prevent performance bottlenecks in hard disk reading and writing.

[0045] According to the technical solution provided by the embodiments of the present disclosure, determining whether to read the requested message from the first buffer or the second buffer based on the offset of the message requested by the read request and the maximum offset of the message queue includes: determining to read the requested message from the first buffer if the difference between the offset of the message requested by the read request and the maximum offset of the message queue is less than or equal to a preset threshold, or determining to read the requested message from the second buffer if the difference between the offset of the message requested by the read request and the maximum offset of the message queue is greater than the preset threshold. In this way, it is easy to determine whether the requested message is a hot message cached in the first buffer or a cold message cached in the second buffer, reducing the occupation of hard disk I / O by a large number of cold reads in the system. Moreover, the cold read access to the hard disk can be reduced by controlling the second buffer, preventing performance bottlenecks in hard disk reading and writing.

[0046] According to the technical solution provided by the embodiments of the present disclosure, by allowing only the message consumers of the message queue to read from the second buffer and not allowing the message producers of the message queue to write to it, an efficient self-managing cache model can be achieved without occupying the first buffer, providing high-performance cache reading and writing capabilities for a large amount of cold message data.

[0047] According to the technical solution provided by the embodiments of the present disclosure, by setting the first buffer as the page cache area PageCache and the second buffer as the least recently used cache area LRU Cache, an efficient self-managing cache model can be achieved without occupying the page cache area Page Cache, reducing the occupation of Page Cache by a large number of cold reads in the system, improving the performance of system writing and hot reading, and reducing the occupation of hard disk I / O by a large number of cold reads in the system. The cold read access to the hard disk can be reduced by controlling the LRU Cache, providing high-performance cache reading and writing capabilities for a large amount of cold message data.

[0048] According to the technical solution provided by the embodiments of the present disclosure, returning the message requested by the read request from the first buffer or the second buffer includes: when it is determined to read the requested message from the second buffer and the message does not exist in the second buffer, reading the message requested by the read request from the non-volatile memory of the message queue storage node into the least recently used buffer (LRU Cache) in a direct I / O manner to return the requested message; or when it is determined to read the requested message from the first buffer and the message does not exist in the first buffer, a page fault occurs, and the message requested by the read request is read from the non-volatile memory of the message queue storage node into the page cache (Page Cache) to return the requested message. In this way, an efficient self-managed cache model can be implemented without occupying the page cache (Page Cache), reducing the occupation of the Page Cache by a large number of cold reads in the system, improving the system write and hot read performance, and reducing the occupation of the hard disk I / O by a large number of cold reads in the system. The cold read access to the hard disk can be reduced by controlling the LRU Cache, and a high-performance cache read / write capability for a large amount of cold message data can be provided.

[0049] According to the technical solution provided by the embodiments of the present disclosure, the operation of returning a message from the first buffer is executed by a first thread, and the operation of returning a message from the second buffer is executed by a second thread in a different thread pool from the first thread, so that the read threads for the first buffer and the second buffer are separated. A separate thread pool is used to separately accelerate cold reads, and at the same time, the number of cores occupied by the cold read thread pool is limited. Hot reads use another separate thread pool to work. By using the thread separation mode, while improving the cold read ability, the interference to the hot read performance is reduced.

[0050] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. Brief Description of the Drawings

[0051] In combination with the drawings, through the following detailed description of non-limiting embodiments, other features, objectives, and advantages of the present disclosure will become more apparent. In the drawings:

[0052] Figure 1 A flowchart showing a message queue processing method according to an embodiment of the present disclosure;

[0053] Figure 2 A schematic diagram showing an exemplary structure of a message queue management system according to an embodiment of the present disclosure;

[0054] Figure 3Schematic diagram showing an implementation scenario of a message queue processing method according to an embodiment of the present disclosure;

[0055] Figure 4 Block diagram showing the structure of a message queue processing apparatus according to an embodiment of the present disclosure. Detailed implementation

[0056] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for clarity, parts irrelevant to the description of the exemplary embodiments are omitted in the drawings.

[0057] In the present disclosure, it should be understood that terms such as "including" or "having" are intended to indicate the presence of the labels, numbers, steps, actions, components, parts, or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of the presence or addition of one or more other labels, numbers, steps, actions, components, parts, or combinations thereof.

[0058] In addition, it should be noted that, without conflict, the embodiments in the present disclosure and the labels in the embodiments can be combined with each other. The present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0059] In the embodiments of the present disclosure, the following concepts may be mentioned:

[0060] LRU cache (LRU Cache): Least Recently Used cache. Least Recently Used is a known cache eviction strategy that allows data that is frequently used recently to reside in the cache.

[0061] Producer: A client that produces (writes) messages to the message queue.

[0062] Consumer: A client that fetches (reads) messages from the message queue for consumption.

[0063] Message: An element in the message model that carries information for business communication.

[0064] Message queue: A distributed communication system that provides asynchronous decoupling and peak shaving and valley filling between subsystems.

[0065] Cold message (data): Messages (data) stored on low-speed devices such as disks, which are messages (data) with a low access frequency.

[0066] Hot message (data): Messages (data) residing in system memory or cache, which are messages (data) with a high access frequency.

[0067] Separation of read and write for hot and cold message data: The read and write of cold message data are isolated from those of hot message data and do not affect each other.

[0068] Considering the disadvantages of the existing technical solutions, the inventor of the present disclosure proposes a new solution: by establishing a second buffer area dedicated to storing cold message data in the system buffer area and using the first buffer area to cache hot message data, the read and write of hot and cold message data in the message queue are separated. The embodiments of the present disclosure provide a high-performance message queue, ensuring that message data is hot and cold separated, with a large throughput and high real-time performance. The solutions of the embodiments of the present disclosure have been tested and applied in message queue products to ensure the high-performance implementation of pulling cold message data from the message queue.

[0069] According to the technical solution provided by the embodiments of the present disclosure, a first buffer area for caching hot messages and a second buffer area for caching cold messages are established in the virtual memory of the message queue storage node; according to the read request for the message queue, the index value of the message requested by the read request is determined; according to the index value, it is determined whether to read the requested message from the first buffer area or the second buffer area; and the message requested by the read request is returned from the first buffer area or the second buffer area. By using different buffer areas to cache hot and cold message data, the read caches of hot and cold message data are physically separated, reducing the occupation of hard disk IO by a large number of cold reads in the system. Moreover, the cold read access to the hard disk can be reduced by controlling the second buffer area, preventing performance bottlenecks in hard disk read and write. Therefore, the solutions of the embodiments of the present disclosure overcome the impact of a large number of cold message data reads on the overall system performance, improve the overall read and write performance and throughput of the system, and provide stronger support for the read of a large number of cold message data in the flow mode in the message scenario.

[0070] To solve the above problems, the present disclosure proposes a message queue processing method, device, and message queue management system.

[0071] Figure 1 The flowchart showing a message queue processing method according to an embodiment of the present disclosure is as follows. Figure 1 As shown, the message queue processing method includes steps S101, S102, S103, and S104.

[0072] In step S101, a first buffer area for caching hot messages and a second buffer area for caching cold messages are established in the virtual memory of the message queue storage node.

[0073] In step S102, according to the read request for the message queue, the index value of the message requested by the read request is determined.

[0074] In step S103, determine whether to read the requested message from the first buffer or the second buffer according to the index value.

[0075] In step S104, return the message requested by the read request from the first buffer or the second buffer.

[0076] In an embodiment of the present disclosure, the architecture of the message queue may include a client and a server of the message queue. The client of the message queue may include a producer client and a consumer client. Moreover, a client may be both a producer and a consumer. The server is used to write message data to the storage node or return message data from the storage node in response to a production request or a consumption request of the client. The storage node may include a volatile memory such as DRAM (Dynamic Random Access Memory) as the memory, and may also include a non-volatile memory such as SSD (Solid State Disk) as the hard disk.

[0077] In the related art, the storage node of the message queue includes a memory and a hard disk, and a page buffer may be set in the memory (virtual memory) to cache the messages produced by the producer and consumed by the consumer. However, the solution of caching message data through the page buffer will seriously affect the performance when writing messages in the face of a large number of cold reads. Cold reads will occupy a large amount of memory resources and seriously affect follow-up consumption (hot message consumption) and message writing (resource competition in the page buffer and hard disk IO competition), and the cold read has a long latency and low throughput.

[0078] To solve the problems of the related art, the solution of the embodiment of the present disclosure adopts a dual-buffer solution, that is, a first buffer for caching hot messages and a second buffer for caching cold messages are established in the virtual memory of the storage node.

[0079] In an embodiment of the present disclosure, the first buffer is a page cache Page Cache, and the second buffer is a least recently used cache LRU Cache.

[0080] According to the technical solution provided by the embodiments of the present disclosure, by making the first buffer be the page cache PageCache and the second buffer be the least recently used cache LRU Cache, an efficient self-managed cache model can be implemented without occupying the page cache Page Cache, reducing the occupation of the Page Cache by a large number of cold reads in the system, improving the system writing and hot read performance, and reducing the occupation of the hard disk IO by a large number of cold reads in the system. The cold read access to the hard disk can be reduced by controlling the LRU Cache, and a high-performance cache read and write capability for a large amount of cold message data can be provided.

[0081] In an embodiment of the present disclosure, hot message data is cached in the page cache, and cold message data is cached in the LRU cache. In an embodiment of the present disclosure, the page cache can be a cache area established by the system of the storage node in virtual memory, or can be a cache area established by the message queue server adopting the solution according to the embodiments of the present disclosure in virtual memory. The page cache can receive the hot message data written by the producer through the message queue server, and can also receive the hot message data read by the consumer through the message queue server. In the related art, a page cache is established in the virtual memory of the storage node, but the page cache in the related art is not only used to cache hot message data but can also cache hot and cold message data, and its specific working mode is significantly different from the page cache in the embodiments of the present disclosure. The page cache dedicated to caching hot message data in the embodiments of the present disclosure can be implemented based on the page cache of the system in the related art, or rather, the message queue processing method of the embodiments of the present disclosure is based on the related art to establish a page cache for caching hot message data. In an embodiment of the present disclosure, the LRU cache is a cache area dedicated to storing cold message data established independently of the page cache in virtual memory by the message queue server adopting the solution according to the embodiments of the present disclosure. It only accepts the request of the consumer to read cold message data, does not accept the request of the consumer to read hot message data, and does not accept the producer to write message data. Therefore, the embodiments of the present disclosure achieve the separation of hot and cold data. The cold data will not affect the use of the page cache in the system, and at the same time, the access to the hard disk IO can be controlled and reduced by controlling the capacity of the self-built LRU cache, ultimately achieving the separation of hot and cold data.

[0082] In an embodiment of the present disclosure, the second buffer only accepts the reading of the message consumer of the message queue and does not accept the writing of the message producer of the message queue.

[0083] According to the technical solution provided by the embodiments of the present disclosure, by allowing only the message consumers of the message queue to read from the second buffer and not allowing the message producers of the message queue to write to it, an efficient self - managed cache model can be achieved without occupying the first buffer, and high - performance cache read - write capabilities for a large amount of cold message data can be provided.

[0084] The following refers to Figure 2 Describe the exemplary structure of a message queue management system according to an embodiment of the present disclosure. Figure 2 A schematic diagram showing the exemplary structure of a message queue management system according to an embodiment of the present disclosure.

[0085] As Figure 2 shown, the message queue management system may include a message queue client 210, a message queue server (not shown in the figure), and a message queue storage node 240. The message queue server is used to accept the write request from the message queue client 210 (producer) to write message data to the storage node 240, and is used to accept the read request from the message queue client 210 (consumer) to return message data from the storage node 240. In Figure 2 the shown message queue management system, the specific description of the message queue server is omitted.

[0086] As Figure 2 shown, the storage node 240 includes a virtual memory 220 and a hard disk 230. Among them, the virtual memory is a part of the memory of the storage node, which can be implemented by using physical memory and a hard disk, and its specific construction method can be known from the related art and will not be elaborated here. The virtual memory 220 includes a first buffer 221 for caching hot messages and a second buffer 222 for caching cold messages. Among them, the first buffer 221 can be a page cache (Page Cache), and the second buffer 222 is a least recently used cache (LRU Cache). The first buffer 221 and the second buffer 222 cache message data in the form of data blocks 223. In the first buffer 221, the message data can be cached as a mapped file through the MMAP memory mapping method.

[0087] As Figure 2As shown, the hard disk 230 caches the message data written from the first buffer 221 in the form of data blocks 231. In an embodiment of the present disclosure, according to a preset disk flushing rule, for example, when the first buffer 221 is full, or the message data cached in the first buffer 221 has not been read within a certain time range, or the call frequency within a certain time period is lower than the preset frequency, the corresponding message data in the first buffer 221 is flushed to the hard disk 230 and stored as a new data block 232. In an embodiment of the present disclosure, the message queue client 210 (producer) sequentially writes the hot message data into the first buffer 221 according to the method in the related art, for example, the page buffer, and can write the message data into the hard disk 230 from the first buffer 221 by means such as asynchronous disk flushing according to the method in the related art.

[0088] As Figure 2 shown, the first buffer 221 caches the hot message data. When the message queue server determines that the message data requested to be read by the message queue client 210 (consumer) is hot message data, the message queue server returns the hot message data from the first buffer 221 to the message queue client 210 (consumer). The message queue client 210 (consumer) can perform follow-up hot reading on the hot message data sequentially written into the first buffer 221, that is, the message queue client 210 (consumer) reads following the writing of the hot message data of the message queue client 210 (producer) in the first buffer 221. For example, the scenario of adopting the follow-up hot reading operation can be a stream computing (or called streaming) scenario.

[0089] As Figure 2 shown, the second buffer 222 caches the cold message data. When the message queue server determines that the message data requested to be read by the message queue client 210 (consumer) is cold message data, the message queue server returns the message data from the second buffer 222 to the message queue client 210 (consumer). The message queue client 210 (consumer) can perform cold reading on the cold message data in the second buffer 222, that is, the message data is fetched from the hard disk 230 to the second buffer 222 through direct IO, and the message queue server returns the cold message data from the second buffer 222 to the message queue client 210 (consumer). In an embodiment of the present disclosure, the IO access frequency to the hard disk 230 can be controlled by controlling the capacity of the second buffer 222. Therefore, the reading separation of hot and cold message data can be achieved, and the processing speed of cold reading of the second buffer 222 can be improved, thereby improving the efficiency of the message queue as a whole.

[0090] In one embodiment of the present disclosure, when the message queue client 210 (consumer) requests to read message data from the message queue server, it does not know whether the requested message data is hot message data or cold message data. Instead, it is the message queue server that determines whether the requested message data is hot message data or cold message data, and returns the hot message data from the first buffer 221 to the message queue client 210 (consumer), or returns the cold message data from the second buffer 222 to the message queue client 210 (consumer). In one embodiment of the present disclosure, through the separation of hot and cold data, the reading of cold data does not affect the use of the first buffer 221 (system page buffer), and at the same time, by controlling the capacity of the self-built second buffer 222 (LRU buffer), the access to the hard disk IO can be controlled and reduced, ultimately achieving the separation of hot and cold data.

[0091] The following refers to Figure 3 to further describe the implementation scenario of the message queue processing method according to an embodiment of the present disclosure. Figure 3 The schematic diagram showing the implementation scenario of the message queue processing method according to an embodiment of the present disclosure.

[0092] As Figure 3 shown, when the read request arrives at the storage node 310, the message queue server confirms whether the read request is a follow-up read (hot read) or a cold read. If it is confirmed that the read request is a follow-up read, the hot message data is returned from the page buffer 320 in the volatile memory (the memory of the storage node 310, such as DRAM). In the case where it is determined to read the hot message data from the page buffer 320 and the hot message data does not exist in the page buffer 320, a page fault occurs, and the message requested by the read request is read from the non-volatile memory, that is, the hard disk 340, into the page buffer 320 to return the requested message. If it is confirmed that the read request is a cold read (step S301), the cold message data is returned from the self-built LRU buffer 330. If the requested cold message data exists in the self-built LRU buffer 330, the corresponding cold message is returned, otherwise, proceed to step S303 (step S302). In step S303, the data block corresponding to the requested message is fetched from the hard disk 340 to the self-built LRU buffer 330 through direct IO to return the requested message.

[0093] In one example, based on Figure 3 the implementation scenario shown, the message reading can be performed according to the following process:

[0094] 1. The consumer sends a read message request to the message queue storage node 310.

[0095] 2. Use the message data offset of the read request and the maximum written offset to determine whether this read request is a cold read.

[0096] 3. If the read request is a cold read, the corresponding data will be read from the self-built LRU cache area 330. If the corresponding data exists, the corresponding data will be returned from the LRU cache area 330, and the implementation of the relevant LRU policy mechanism will be carried out.

[0097] 4. If the cold read data does not exist in the LRU cache area 330, the corresponding block data will be read from the hard disk 340 through direct IO into the LRU cache area 330, added to the head of the LRU cache linked list, and the corresponding data will be returned.

[0098] 5. If the read request is a hot read, that is, it is determined that the data requested to be read is hot data for follow-up reading (consumption) according to the offset position, the corresponding data will be directly returned from the page cache area 320.

[0099] 6. If the hot data does not exist in the page cache area 320, a page fault will occur, and the corresponding data will be loaded from the hard disk 340 into the page cache area 320, and finally the data will be returned.

[0100] During the message reading process, cold and hot data are separated. Cold data will not affect the use of the system page cache area 320. At the same time, by controlling the capacity of the self-built LRU cache area 330, the access to the hard disk IO can be controlled and reduced, and finally the separation of cold and hot data is realized.

[0101] In an embodiment of the present disclosure, referring to Figure 1 The index value of the described message is the offset of the message in the message queue. Among them, determining the index value of the message requested by the read request according to the read request for the message queue includes: determining the offset of the message requested by the read request according to the read request for the message queue. Among them, determining whether to read the requested message from the first cache area or the second cache area according to the index value includes: determining whether to read the requested message from the first cache area or the second cache area according to the offset of the message requested by the read request and the maximum offset of the message queue.

[0102] According to the technical solution provided by the embodiments of the present disclosure, the index value of the message is the offset of the message in the message queue. Among them, step S102 includes: determining the offset of the message requested by the read request according to the read request for the message queue. Among them, step S103 includes: determining whether to read the requested message from the first buffer or the second buffer according to the offset of the message requested by the read request and the maximum offset of the message queue, so that it is easy to determine whether the requested message is a hot message cached in the first buffer or a cold message cached in the second buffer, reducing the occupation of hard disk IO by a large number of cold reads in the system, and moreover, the cold read access to the hard disk can be reduced by controlling the second buffer to prevent performance bottlenecks in hard disk reading and writing.

[0103] In an embodiment of the present disclosure, the management of the offset mainly refers to managing the consumption progress of each message queue, and the consumption progress of the message queue will be saved on the message queue server during consumption. In an embodiment of the present disclosure, the offset of each message can represent the message length in hexadecimal. In an embodiment of the present disclosure, the queue offset is incremented by 1 when 1 more message is added to the current queue. In an embodiment of the present disclosure, the maximum offset of the message queue is not the offset of the latest message, but the offset of the latest message + 1. The greater the difference between the offset of the message requested by the read request and the maximum offset, the earlier the requested message indicates.

[0104] In an embodiment of the present disclosure, the determining whether to read the requested message from the first buffer or the second buffer according to the offset of the message requested by the read request and the maximum offset of the message queue includes: determining to read the requested message from the first buffer according to that the difference between the offset of the message requested by the read request and the maximum offset of the message queue is less than or equal to a preset threshold, or determining to read the requested message from the second buffer according to that the difference between the offset of the message requested by the read request and the maximum offset of the message queue is greater than the preset threshold.

[0105] According to the technical solution provided by the embodiments of the present disclosure, determining whether to read the requested message from the first buffer or the second buffer based on the offset of the message requested by the read request and the maximum offset of the message queue includes: determining to read the requested message from the first buffer when the difference between the offset of the message requested by the read request and the maximum offset of the message queue is less than or equal to a preset threshold, or determining to read the requested message from the second buffer when the difference between the offset of the message requested by the read request and the maximum offset of the message queue is greater than the preset threshold. In this way, it can be easily determined whether the requested message is a hot message cached in the first buffer or a cold message cached in the second buffer, reducing the occupancy of the hard disk IO caused by a large number of cold reads in the system. Moreover, by controlling the second buffer, the cold read access to the hard disk can be reduced, preventing performance bottlenecks in hard disk reading and writing.

[0106] In an embodiment of the present disclosure, step S104 includes: when it is determined to read the requested message from the second buffer and the message does not exist in the second buffer, reading the message requested by the read request from the non-volatile memory of the message queue storage node into the least recently used cache (LRU Cache) in a direct IO manner to return the requested message; or when it is determined to read the requested message from the first buffer and the message does not exist in the first buffer, a page fault occurs, and the message requested by the read request is read from the non-volatile memory of the message queue storage node into the page cache (Page Cache) to return the requested message.

[0107] According to the technical solution provided by the embodiments of the present disclosure, returning the message requested by the read request from the first buffer or the second buffer includes: when it is determined to read the requested message from the second buffer and the message does not exist in the second buffer, reading the message requested by the read request from the non-volatile memory of the message queue storage node into the least recently used buffer (LRU Cache) in a direct I / O manner to return the requested message; or when it is determined to read the requested message from the first buffer and the message does not exist in the first buffer, a page fault occurs, and the message requested by the read request is read from the non-volatile memory of the message queue storage node into the page cache (Page Cache) to return the requested message. In this way, an efficient self-managed cache model can be implemented without occupying the page cache (Page Cache), reducing the occupation of the Page Cache by a large number of cold reads in the system, improving the system write and hot read performance, and reducing the occupation of the hard disk I / O by a large number of cold reads in the system. The cold read access to the hard disk can be reduced by controlling the LRU Cache, and a high-performance cache read and write capability for a large amount of cold message data can be provided.

[0108] In an embodiment of the present disclosure, the operation of returning a message from the first buffer is performed by a first thread, and the operation of returning a message from the second buffer is performed by a second thread that is in a different thread pool from the first thread, so that the read threads for the first buffer and the second buffer are separated.

[0109] According to the technical solution provided by the embodiments of the present disclosure, the operation of returning a message from the first buffer is performed by a first thread, and the operation of returning a message from the second buffer is performed by a second thread that is in a different thread pool from the first thread, so that the read threads for the first buffer and the second buffer are separated. The cold read is separately accelerated using a separate thread pool, and at the same time, the number of cores occupied by the cold read thread pool is limited. The hot read uses another separate thread pool to work. By using the thread separation mode, while improving the cold read ability, the interference to the hot read performance is reduced.

[0110] In an embodiment of the present disclosure, by using thread separation, the cold read is separately accelerated using threads, and at the same time, the number of cores occupied by the cold read thread pool is limited. The hot read uses another separate thread pool to work. Therefore, by using the thread separation mode, while improving the cold read ability, the interference to the hot read performance can be reduced. Moreover, the embodiments of the present disclosure are more friendly to stream computing and provide a stronger data cold read ability.

[0111] The following refers to Figure 4Describe a message queue processing device according to an embodiment of the present disclosure. Figure 4 The structural block diagram of a message queue processing device according to an embodiment of the present disclosure is shown. As Figure 4 shown, the message queue processing device 400 includes a buffer area establishment module 401, an index value determination module 402, a message cache location determination module 403, and a message return module 404.

[0112] The buffer area establishment module 401 is configured to establish a first buffer area for caching hot messages and a second buffer area for caching cold messages in the virtual memory of the message queue storage node.

[0113] The index value determination module 402 is configured to determine the index value of the message requested by the read request according to the read request for the message queue.

[0114] The message cache location determination module 403 is configured to determine whether to read the requested message from the first buffer area or the second buffer area according to the index value.

[0115] The message return module 404 is configured to return the message requested by the read request from the first buffer area or the second buffer area.

[0116] The message queue processing device according to an embodiment of the present disclosure can implement caching of hot and cold message data in different buffer areas, enabling physical separation of the read caches for hot and cold message data, reducing the occupation of hard disk I / O by a large number of cold reads in the system, and reducing cold read access to the hard disk by controlling the second buffer area to prevent performance bottlenecks in hard disk read and write. Therefore, the solution of the embodiments of the present disclosure overcomes the impact of a large number of cold message data reads on the overall system performance, improves the overall read and write performance and throughput of the system, and provides stronger support for reading a large number of cold message data in the stream mode in the message scenario.

[0117] Those skilled in the art can understand that the technical solutions described with reference to Figure 4 can be combined with the embodiments described with reference to Figures 1 to 3 to achieve the technical effects achieved by the embodiments described with reference to Figures 1 to 3 The specific content can refer to the description made above according to Figures 1 to 3 and will not be elaborated herein.

[0118] In an embodiment of the present disclosure, a message queue management system is provided, where the system includes:

[0119] a message queue client, a message queue server, and a message queue storage node,

[0120] Among them, the message queue server establishes a first buffer for caching hot messages and a second buffer for caching cold messages in the virtual memory of the message queue storage node.

[0121] According to the read request of the message queue client for the message queue, the message queue server determines the index value of the message requested by the read request.

[0122] The message queue server determines whether to read the requested message from the first buffer or the second buffer according to the index value.

[0123] The message queue server returns the message requested by the read request from the first buffer or the second buffer to the message queue client.

[0124] The message queue management system according to an embodiment of the present disclosure can implement caching of hot and cold message data in different buffers, enabling physical separation of the read caches of hot and cold message data, reducing the occupation of hard disk I / O by a large number of cold reads in the system, and reducing cold read access to the hard disk by controlling the second buffer to prevent performance bottlenecks in hard disk read and write. Therefore, the solution of the embodiments of the present disclosure overcomes the impact of a large number of cold message data reads on the overall system performance, improves the overall read and write performance and throughput of the system, and provides stronger support for reading a large number of cold message data in the stream mode in the message scenario.

[0125] Those skilled in the art can understand that the message queue management system can be combined with the embodiments described with reference to Figures 1 to 3 to thus have the technical effects achieved by the embodiments described with reference to Figures 1 to 3 The specific content can be referred to the above description according to Figures 1 to 3 and will not be elaborated herein.

[0126] In particular, according to the embodiments of the present disclosure, the methods described above with reference to the drawings can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly embodied on a computer-readable medium, and the computer program includes program code for executing the methods in the drawings. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part and / or installed from a removable medium. For example, an embodiment of the present disclosure includes a readable storage medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, program code for executing the methods in the drawings is implemented.

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0128] The units or modules involved in the embodiments described in the present disclosure can be implemented in software or in hardware. The units or modules described can also be provided in a processor, and the names of these units or modules do not, in some cases, constitute a limitation on the units or modules themselves.

[0129] On the other hand, the present disclosure also provides a computer-readable storage medium, which can be the computer-readable storage medium included in the nodes described in the above embodiments; or it can be a computer-readable storage medium that exists separately and is not assembled into a device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the methods described in the present disclosure.

[0130] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles applied. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, technical solutions formed by mutually replacing the above features with technical features having similar functions (but not limited to) disclosed in the present disclosure.

Claims

1. A message queue processing method, wherein, The method includes: establishing a first buffer for caching hot messages and a second buffer for caching cold messages in the virtual memory of the message queue storage node; determining the offset of the message requested by the read request in the message queue according to the read request for the message queue; determining whether to read the requested message from the first buffer or the second buffer according to the offset of the message requested by the read request in the message queue and the maximum offset of the message queue; returning the message requested by the read request from the first buffer or the second buffer.

2. The method according to claim 1, wherein The determining whether to read the requested message from the first buffer or the second buffer according to the offset of the message requested by the read request and the maximum offset of the message queue includes: determining to read the requested message from the first buffer if the difference between the offset of the message requested by the read request and the maximum offset of the message queue is less than or equal to a preset threshold, or determining to read the requested message from the second buffer if the difference between the offset of the message requested by the read request and the maximum offset of the message queue is greater than the preset threshold.

3. The method according to claim 1, wherein The second buffer only accepts reads from the message consumers of the message queue and does not accept writes from the message producers of the message queue.

4. The method according to claim 1, wherein The first buffer is a page cache (Page Cache), and the second buffer is a least recently used cache (LRU Cache).

5. The method according to claim 4, wherein, The returning the message requested by the read request from the first buffer or the second buffer includes: in the case where it is determined to read the requested message from the second buffer and the message does not exist in the second buffer, reading the message requested by the read request from the non-volatile memory of the message queue storage node into the least recently used cache (LRU Cache) in a direct I / O manner to return the requested message, or in the case where it is determined to read the requested message from the first buffer and the message does not exist in the first buffer, a page fault occurs, and the message requested by the read request is read from the non-volatile memory of the message queue storage node into the page cache (Page Cache) to return the requested message.

6. The method according to claim 1, wherein, Performing the operation of returning messages from the first buffer through a first thread, and performing the operation of returning messages from the second buffer through a second thread in a different thread pool from the first thread, so that the read threads for the first buffer and the second buffer are separated.

7. A message queue processing device, wherein, The apparatus includes: a buffer establishment module configured to establish a first buffer for caching hot messages and a second buffer for caching cold messages in the virtual memory of the message queue storage node an index value determination module configured to determine the offset of the message requested by the read request in the message queue according to the read request for the message queue; A message cache location determination module, configured to determine whether to read the requested message from the first buffer or the second buffer according to the offset of the message requested by the read request in the message queue and the maximum offset of the message queue; A message return module, configured to return the message requested by the read request from the first buffer or the second buffer.

8. A message queue management system, wherein, The system includes: A message queue client, a message queue server, and a message queue storage node, wherein the message queue server establishes a first buffer for caching hot messages and a second buffer for caching cold messages in the virtual memory of the message queue storage node, According to the read request of the message queue client for the message queue, the message queue server determines the offset of the message requested by the read request in the message queue; The message queue server determines whether to read the requested message from the first buffer or the second buffer according to the offset of the message requested by the read request in the message queue and the maximum offset of the message queue, The message queue server returns the message requested by the read request from the first buffer or the second buffer to the message queue client.

9. A computer program product, including computer instructions, which implement the method according to any one of claims 1-6 when executed by a processor.

Images