A Cloud Storage Method and System Based on RocketMQ

Abstract

The present invention relates to a cloud storage method and system based on RocketMQ, belonging to the field of cloud storage. The method includes: expanding a message storage engine in Apache RocketMQ distributed messages; creating an access interface for the Apache Bookkeeper distributed logging system; when writing data, obtaining a QueueLedger object, constructing complete data of a CommitLOG object, and calling a write interface on the QueueLedger object to update data caching; an asynchronous thread checks the IndexFile writing progress of each QueueLedger object; when reading data, preferentially perform data caching reading, determine whether corresponding data exists in the data caching, if so, immediately return the corresponding data, otherwise, based on the IndexFile data retrieval technology, load data in the corresponding Ledger and return it.

Description

Technical Field

[0001] The present invention belongs to the field of cloud storage, and particularly relates to a cloud storage method and system based on RocketMQ. Background Art

[0002] Apache RocketMQ is a distributed message with low latency, high concurrency, high availability, and high reliability. Due to its simple architecture, rich business functions, and strong scalability, it is widely adopted by many enterprise developers and cloud providers.

[0003] However, the current local storage resources of RocketMQ are fixed and cannot change dynamically according to demand, which limits the flexibility and adaptability of the system in the face of different loads and data growth, easily leads to uneven data distribution, causes some Broker nodes to be overloaded, while other nodes are relatively idle, affecting the overall performance and stability of the system. Summary of the Invention

[0004] In view of the above deficiencies of the prior art, the purpose of the invention is to provide a cloud storage method and system based on RocketMQ. In the Apache RocketMQ distributed message, expand the message storage engine, so that the message data is stored in the distributed log system, and realize the separation of computing and storage without changing the cluster deployment architecture and fully compatible with the open-source ecosystem, which can change dynamically according to demand, improve the utilization rate of resources, improve the flexibility and adaptability of the system in the face of different loads and data growth, and improve the overall performance and stability of the system.

[0005] In the first aspect of the present invention, a cloud storage method based on RocketMQ is proposed, including:

[0006] S1, expand the message storage engine in the Apache RocketMQ distributed message;

[0007] S2, create an access interface for the Apache Bookkeeper distributed log system;

[0008] S3, expand the access interface and maintain the LedgerManager object and the QueueLedger object;

[0009] S4, when writing data, obtain the QueueLedger object, construct the complete data of the CommitLOG object, and call the write interface on the QueueLedger object to update the data cache;

[0010] S5, an asynchronous thread checks the writing progress of the IndexFile of each QueueLedger object;

[0011] S6. When reading data, give priority to reading from the data cache. Determine whether the corresponding data exists in the data cache. If so, immediately return the corresponding data. Otherwise, based on the IndexFile data retrieval technology, load the data from the corresponding Ledger and return it.

[0012] Furthermore, the access interface is used to open logs, append APPEND log content, and / or randomly access one or more log data.

[0013] Furthermore, the LedgerManager object contains all Ledger objects of Apache Bookkeeper; the QueueLedger object contains the metadata information of all Ledgers in the queue.

[0014] Furthermore, the cloud storage method based on RocketMQ further includes:

[0015] Maintain the QueueOffset value through the QueueLedger object;

[0016] Among them, the QueueOffset value is used to determine the physical location of the message data.

[0017] Furthermore, the specific steps of S5 include:

[0018] S501. Maintain the IndexLedger object and cache the IndexFile index data through the IndexLedger object;

[0019] S502. When data is written, read the complete data of the CommitLOG object;

[0020] S503. Generate IndexFile data based on the IndexFile index data and the complete data of the CommitLOG object;

[0021] S504. Record the IndexFile data into the Ledger object of the IndexFile.

[0022] Furthermore, the specific steps of the IndexFile data retrieval technology in S6 include:

[0023] S601. Calculate the Hash value of the query Key;

[0024] S602. Take the remainder of the Hash value and the number of buckets of the index data to obtain the offset value of the target bucket;

[0025] S603. Determine whether the offset value is less than 0. If so, it is determined that there is no corresponding data. Otherwise, read the IndexFile data corresponding to the offset of the IndexFileLedger. When the previous-index value of the IndexFile data is greater than or equal to 0, use the previous-index value as the offset to find the next IndexFile data until all are found;

[0026] S604. Compare whether the Hash value of the IndexFile data is the same as the Hash value of the query key. If so, read the corresponding CommitLOG data and return it.

[0027] Furthermore, the cloud storage method based on RocketMQ further includes:

[0028] Regularly check the message storage policy of each TOPIC, obtain the last data write time of each Ledger in each Queue through the QueueLedger interface, and delete the corresponding Ledger data according to the message storage policy.

[0029] In the second aspect of the present invention, a cloud storage system based on RocketMQ is proposed, including:

[0030] An expansion module for expanding the message storage engine in the Apache RocketMQ distributed message;

[0031] A creation module for creating an access interface to the Apache Bookkeeper distributed log system;

[0032] A first maintenance module for expanding the access interface and maintaining the LedgerManager object and the QueueLedger object;

[0033] A writing module for, when writing data, obtaining the QueueLedger object, constructing the complete data of the CommitLOG object, and calling the write interface on the QueueLedger object to update the data cache;

[0034] An inspection module for asynchronously checking the IndexFile write progress of each QueueLedger object;

[0035] A reading module for, when reading data, preferentially reading from the data cache, determining whether there is corresponding data in the data cache. If so, immediately return the corresponding data. Otherwise, load and return the data in the corresponding Ledger based on the IndexFile data retrieval technology.

[0036] Further, the access interface is used to open logs, append log content, and / or randomly access one or more log data.

[0037] Further, the LedgerManager object contains all Ledger objects of Apache Bookkeeper; the QueueLedger object contains metadata information of all Ledgers in the queue.

[0038] Further, the cloud storage method based on RocketMQ further includes:

[0039] A second maintenance module for maintaining the QueueOffset value through the QueueLedger object;

[0040] wherein the QueueOffset value is used to determine the physical location of the message data.

[0041] Further, the checking module is specifically used for:

[0042] Maintaining the IndexLedger object and caching the IndexFile index data through the IndexLedger object;

[0043] When there is data writing, reading the complete data of the CommitLOG object;

[0044] Generating IndexFile data according to the IndexFile index data and the complete data of the CommitLOG object;

[0045] Recording the IndexFile data into the Ledger object of the IndexFile.

[0046] Further, the reading module is specifically used for:

[0047] Calculating the Hash value of the query key;

[0048] Taking the remainder of the Hash value and the number of buckets of the index data to obtain the offset value of the target bucket;

[0049] Judging whether the offset value is less than 0. If so, it is determined that there is no corresponding data. Otherwise, reading the IndexFile data corresponding to the offset of the IndexFile Ledger. When the previous-index value of the IndexFile data is greater than or equal to 0, using the previous-index value as the offset to find the next IndexFile data until all are found;

[0050] Compare whether the Hash value of the IndexFile data is the same as the Hash value of the query key. If so, read the corresponding CommitLOG data and return it.

[0051] Furthermore, the cloud storage method based on RocketMQ further includes:

[0052] A deletion module, which is used to regularly check the message storage policy of each TOPIC, obtain the last data write time of each Ledger in each Queue through the QueueLedger interface, and delete the corresponding Ledger data according to the message storage policy.

[0053] The beneficial effects of the present invention are as follows:

[0054] The method and system of the present invention extend the message storage engine in Apache RocketMQ distributed messages, enabling message data to be stored in a distributed log system. Without changing the cluster deployment architecture and being fully compatible with the open-source ecosystem, it realizes the separation of computing and storage, which can be dynamically changed as needed, improves the utilization rate of resources, enhances the flexibility and adaptability of the system in the face of different loads and data growth, and improves the overall performance and stability of the system. Description of the Drawings

[0055] The drawings are only for the purpose of showing specific embodiments and are not considered to be a limitation of the present invention. Throughout the drawings, the same reference signs represent the same components. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present invention, and those skilled in the art can also obtain other drawings based on these drawings.

[0056] Figure 1 It is a flowchart of a cloud storage method based on RocketMQ provided by an embodiment of the present invention;

[0057] Figure 2 It is a structural diagram of a cloud storage system based on RocketMQ provided by an embodiment of the present invention. Detailed Embodiments

[0058] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings. Obviously, the described embodiments are some embodiments of the present invention, rather than all embodiments. It should be understood that these descriptions are only exemplary and are not used to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present invention.

[0059] In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0060] In the description of the present invention, it should be noted that unless otherwise clearly specified and limited, the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention. In addition, the terms "first", "second", "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. The terms "installed", "connected", "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0061] Here, exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. On the contrary, they are only examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0062] The present invention proposes a cloud storage method and system based on RocketMQ to solve the problems that the local storage resources of current RocketMQ are fixed and cannot change dynamically according to demand, which limits the flexibility and adaptability of the system in the face of different loads and data growth, easily leads to unbalanced data distribution, resulting in overloading of some Broker nodes and relative idleness of other nodes, affecting the overall performance and stability of the system.

[0063] Method Embodiment

[0064] Referring to the attached Figure 1 drawings, a flowchart of a cloud storage method based on RocketMQ provided by an embodiment of the present invention is shown.

[0065] A cloud storage method based on RocketMQ provided by an embodiment of the present invention includes:

[0066] Specifically, the method includes steps S1 to S6.

[0067] S1. In Apache RocketMQ distributed messaging, extend the message storage engine.

[0068] Among them, Apache RocketMQ is a distributed messaging with low latency, high concurrency, high availability, and high reliability. Due to its simple architecture, rich business functions, and strong scalability, it is widely adopted by many enterprise developers and cloud providers.

[0069] Furthermore, the storage files of RocketMQ include three parts, one is the CommitLOG for storing data, the ConsumeQueue for storing logical Queues, and the IndexFile for storing string indexes.

[0070] It should be noted that by extending the storage engine of Apache RocketMQ and introducing an external distributed logging system, separating data storage to the distributed logging system to achieve separation of computing and storage, the problem of single-point capacity bottleneck can be solved and resource utilization can be improved.

[0071] S2. Create an access interface for the Apache Bookkeeper distributed logging system.

[0072] Among them, Apache Bookkeeper is a distributed and naturally scalable logging system. All data is written in an append-only manner, and the data that has been written cannot be modified.

[0073] Optionally, the access interface is used to open logs, append APPEND log content, and / or randomly access one or more log data.

[0074] In the present invention, creating an access interface for Apache Bookkeeper can bring higher flexibility, performance optimization, and data security guarantee to the message storage method based on RocketMQ. Through the design of the interface, the two systems can be effectively integrated, making full use of their respective advantages to provide a reliable message processing and storage solution for applications.

[0075] S3. Extend the access interface and maintain the LedgerManager object and the QueueLedger object.

[0076] Optionally, the LedgerManager object contains all Ledger objects of Apache Bookkeeper.

[0077] Optionally, the QueueLedger object contains the metadata information of all Ledgers in the queue.

[0078] Among them, Ledger is a logical unit of Bookkeeeper. A Ledger can be understood as an independent log file or a message queue. There are only two operations for the core interface of Ledger. One is to APPEND data, and the other is to read data randomly.

[0079] S4. When writing data, obtain the QueueLedger object, construct the complete data of the CommitLOG object, and call the write interface on the QueueLedger object to update the data cache.

[0080] In the present invention, obtain the QueueLedger object, construct the complete data of the CommitLOG object on it, and then call the write interface to update the data cache. Doing so can effectively manage and update the data cache to ensure that the written message data can be quickly acquired and processed by the system.

[0081] S5. The asynchronous thread checks the writing progress of the IndexFile for each QueueLedger object.

[0082] In a possible implementation manner, S5 specifically includes sub-steps S501 to S504:

[0083] S501. Maintain the IndexLedger object and cache the IndexFile index data through the IndexLedger object.

[0084] S502. When there is data writing, read the complete data of the CommitLOG object.

[0085] S503. Generate the IndexFile data according to the IndexFile index data and the complete data of the CommitLOG object.

[0086] S504. Record the IndexFile data into the Ledger object of the IndexFile.

[0087] In the present invention, the asynchronous thread checks the writing progress of the IndexFile for each QueueLedger object and caches the IndexFile index data by maintaining the IndexLedger object. This method can achieve asynchronous monitoring of the data writing progress, avoid blocking the main processing flow, and at the same time optimize the overall performance and response speed of the system.

[0088] Further, each time the IndexFile is written, the index data in the memory needs to be updated. The index data is initially obtained by reading the last data from the Ledger of the IndexFile. If there is no last data, it is necessary to read the Ledger of the corresponding CommitLog and the Ledger of the IndexFile in full to reconstruct this data. Each time the program exits or the index cache is released, the full Index File index data needs to be written to the Ledger of the IndexFile.

[0089] S6. When reading data, first perform a data cache read, and determine whether the corresponding data exists in the data cache. If so, immediately return the corresponding data; otherwise, based on the IndexFile data retrieval technology, load and return the data in the corresponding Ledger.

[0090] In the present invention, first attempt to read data from the data cache. If the data already exists in the cache, the corresponding data can be immediately returned, greatly shortening the response time of data access and improving the real-time performance and response performance of the system.

[0091] In a possible implementation manner, the IndexFile data retrieval technology in S6 specifically includes sub-steps S601 to S604:

[0092] S601. Calculate the Hash value of the query Key.

[0093] S602. Take the remainder of the Hash value and the number of buckets of the index data to obtain the offset value of the target bucket.

[0094] S603. Determine whether the offset value is less than 0. If so, it is determined that there is no corresponding data; otherwise, read the IndexFile data corresponding to the offset of the IndexFile Ledger. When the previous-index value of the IndexFile data is greater than or equal to 0, use the previous-index value as the offset to find the next IndexFile data until all are found.

[0095] S604. Compare whether the Hash value of the IndexFile data is consistent with the Hash value of the query Key. If so, read the corresponding CommitLOG data and return it.

[0096] In the present invention, the IndexFile data retrieval technology ensures the consistency and integrity of data. By comparing the Hash value of the IndexFile data with the Hash value of the query key, after confirming the correctness of the data, the corresponding CommitLOG data is read and returned. This mechanism guarantees that the data read by the system is accurate and complete, avoiding problems such as data errors or losses.

[0097] In a possible implementation manner, the cloud storage method based on RocketMQ further includes:

[0098] Maintaining the QueueOffset value through the QueueLedger object.

[0099] Wherein, the QueueOffset value is used to determine the physical location of the message data.

[0100] In the present invention, the maintenance of QueueOffset allows the system to track the consumed message offsets in each message queue. Consumers can know from which position they need to start consuming messages through these offsets, ensuring the order and consistency of messages. This mechanism is essential for message consumers, especially when dealing with repeated consumption or message redelivery. By accurately managing the QueueOffset value, consumers can effectively avoid the situations of repeated consumption and message loss. This precise message position management can improve the efficiency and reliability of message consumption, ensuring that consumers can consume messages in the expected order and rate.

[0101] In a possible implementation manner, the cloud storage method based on RocketMQ further includes:

[0102] Regularly checking the message storage policy of each TOPIC, obtaining the last data write time of each Ledger in each Queue through the QueueLedger interface, and deleting the corresponding Ledger data according to the message storage policy.

[0103] In the present invention, regularly checking and cleaning the old Ledger data in the message storage policy under each TOPIC, managed through the QueueLedger interface, is one of the important measures to keep the RocketMQ system running efficiently and stably. This approach not only saves resources but also helps ensure that the system can effectively support business requirements in the long term.

[0104] Furthermore, the present invention uses a distributed logging system to store the information data of RocketMQ, realizing the separation of computing and storage, reducing the usage cost and improving the resource utilization rate, and solving the single-point storage capacity limitation. Compared with the open-source RocektMQ solution that stores data in an object storage, it has higher data reliability and better performance. At the same time, in combination with the distributed logging system, it provides support for higher-granularity and diverse data expiration policies based on topics, better adapts to the characteristics of cloud-native, is stateless, facilitates process transfer, and supports the ability to dynamically scale nodes.

[0105] The beneficial effects of the present invention are as follows:

[0106] The method and system of the present invention, in the Apache RocketMQ distributed message, expand the message storage engine, enabling the message data to be stored in a distributed logging system. Without changing the cluster deployment architecture and being fully compatible with the open-source ecosystem, it realizes the separation of computing and storage, which can change dynamically as needed, improves the resource utilization rate, enhances the flexibility and adaptability of the system in the face of different loads and data growth, and improves the overall performance and stability of the system.

[0107] System embodiments

[0108] Refer to the appended Figure 2 illustrates a schematic structural diagram of a cloud storage system based on RocketMQ provided by an embodiment of the present invention.

[0109] A cloud storage system 20 based on RocketMQ provided by an embodiment of the present invention includes:

[0110] An expansion module 201, used to expand the message storage engine in the Apache RocketMQ distributed message;

[0111] A creation module 202, used to create an access interface for the Apache Bookkeeper distributed logging system;

[0112] A first maintenance module 203, used to expand the access interface and maintain the LedgerManager object and the QueueLedger object;

[0113] A writing module 204, used to obtain the QueueLedger object when writing data, construct the complete data of the CommitLOG object, and call the writing interface on the QueueLedger object to update the data cache;

[0114] An inspection module 205, used for an asynchronous thread to check the writing progress of the IndexFile of each QueueLedger object;

[0115] A reading module 206, which is used to give priority to data cache reading when reading data, determine whether corresponding data exists in the data cache, and if so, immediately return the corresponding data; otherwise, load and return the data in the corresponding Ledger based on the IndexFile data retrieval technology.

[0116] Furthermore, the access interface is used to open logs, append APPEND log content, and / or randomly access one or more log data.

[0117] Furthermore, the LedgerManager object contains all Ledger objects of Apache Bookkeeper; the QueueLedger object contains the metadata information of all Ledgers in the queue.

[0118] Furthermore, the cloud storage method based on RocketMQ further includes:

[0119] A second maintenance module, which is used to maintain the QueueOffset value through the QueueLedger object;

[0120] wherein, the QueueOffset value is used to determine the physical location of the message data.

[0121] Furthermore, the checking module 205 is specifically used for:

[0122] Maintaining an IndexLedger object and caching IndexFile index data through the IndexLedger object;

[0123] When there is data writing, reading the complete data of the CommitLOG object;

[0124] Generating IndexFile data according to the IndexFile index data and the complete data of the CommitLOG object;

[0125] Recording the IndexFile data into the Ledger object of the IndexFile.

[0126] Furthermore, the reading module 206 is specifically used for:

[0127] Calculating the Hash value of the query key;

[0128] Taking the remainder of the Hash value and the number of buckets of the index data to obtain the offset value of the target bucket;

[0129] Determine whether the offset value is less than 0. If so, it is determined that there is no corresponding data. Otherwise, read the IndexFile data corresponding to the offset of the IndexFile Ledger. When the previous-index value of the IndexFile data is greater than or equal to 0, use the previous-index value as the offset to find the next IndexFile data until all are found;

[0130] Compare whether the Hash value of the IndexFile data is the same as the Hash value of the query key. If so, read the corresponding CommitLOG data and return it.

[0131] Furthermore, the cloud storage method based on RocketMQ further includes:

[0132] A deletion module, which is used to regularly check the message storage policy of each TOPIC, obtain the last data write time of each Ledger in each Queue through the QueueLedger interface, and delete the corresponding Ledger data according to the message storage policy.

[0133] The cloud storage system 20 based on RocketMQ provided by the present invention can implement each step and technical effect of the above-mentioned cloud storage method based on RocketMQ. To avoid repetition, the present invention will not elaborate further.

[0134] The beneficial effects of the present invention are as follows:

[0135] The method and system of the present invention extend the message storage engine in Apache RocketMQ distributed messaging, enabling message data to be stored in a distributed log system. Without changing the cluster deployment architecture and being fully compatible with the open source ecosystem, it realizes the separation of computing and storage, which can be dynamically changed as needed, improves resource utilization, enhances the flexibility and adaptability of the system in the face of different loads and data growth, and improves the overall performance and stability of the system.

[0136] The applicant of the present invention has made a detailed description and illustration of the embodiments of the present invention in combination with the accompanying drawings of the specification. However, those skilled in the art should understand that the above embodiments are only the preferred implementation embodiments of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, rather than a limitation on the protection scope of the present invention. On the contrary, any improvement or modification based on the spirit of the present invention should fall within the protection scope of the present invention.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A cloud storage method based on RocketMQ, characterized in that Including: S1, in Apache RocketMQ distributed messaging, extend the message storage engine; S2, create an access interface for the Apache Bookkeeper distributed logging system; S3, extend the access interface and maintain the LedgerManager object and the QueueLedger object; S4, when writing data, obtain the QueueLedger object, construct the complete data of the CommitLOG object on the QueueLedger object, and call the write interface on the QueueLedger object to update the data cache; S5, an asynchronous thread checks the IndexFile write progress of each QueueLedger object; S6, when reading data, give priority to reading from the data cache, determine whether the corresponding data exists in the data cache, if so, immediately return the corresponding data, otherwise, based on the IndexFile data retrieval technology, load and return the data in the corresponding Ledger; The specific content of S5 includes: S501, maintain the IndexLedger object and cache the IndexFile index data through the IndexLedger object; S502, when there is data writing, read the complete data of the CommitLOG object; S503, generate IndexFile data according to the IndexFile index data and the complete data of the CommitLOG object; S504, record the IndexFile data into the Ledger object of IndexFile.

2. The cloud storage method based on RocketMQ according to claim 1, wherein The access interface is used to open the log, append APPEND log content, and / or randomly access one or more log data.

3. The cloud storage method based on RocketMQ according to claim 1, wherein The LedgerManager object contains all Ledger objects of Apache Bookkeeper; the QueueLedger object contains the metadata information of all Ledgers in the queue.

4. The cloud storage method based on RocketMQ according to claim 1, wherein, Also included: Maintain the QueueOffset value through the QueueLedger object; Among them, the QueueOffset value is used to determine the physical location of the message data.

5. The cloud storage method based on RocketMQ according to claim 1, wherein The specific content of the IndexFile data retrieval technology in S6 includes: S601, calculate the Hash value of the query Key; S602, take the remainder of the Hash value and the number of buckets of the IndexFile index data to obtain the offset value of the target bucket; S603, determine whether the offset value is less than 0, if so, determine that there is no corresponding data, otherwise, read the IndexFile data corresponding to the offset of the IndexFileLedger; S604, when the previous-index value of the IndexFile data is greater than or equal to 0, use the previous-index value as the offset to find the next IndexFile data until all are found; S605. Compare whether the Hash value of the IndexFile data is the same as the Hash value of the query key. If so, read the corresponding CommitLOG data and return it.

6. The cloud storage method based on RocketMQ according to claim 1, wherein It also includes: Regularly check the message storage policy of each TOPIC. Obtain the last data write time of each Ledger in each Queue through the QueueLedger interface, and delete the corresponding Ledger data according to the message storage policy.

7. A cloud storage system based on RocketMQ, characterized in that, It includes: An extension module for extending the message storage engine in Apache RocketMQ distributed messages; A creation module for creating an access interface to the Apache Bookkeeper distributed log system; A first maintenance module for extending the access interface and maintaining the LedgerManager object and the QueueLedger object; A write module for, when writing data, obtaining the QueueLedger object, constructing the complete data of the CommitLOG object on the QueueLedger object, and calling the write interface on the QueueLedger object to update the data cache; An inspection module for asynchronously checking the IndexFile write progress of each QueueLedger object; A read module for, when reading data, preferentially reading from the data cache, determining whether the corresponding data exists in the data cache. If so, immediately return the corresponding data. Otherwise, load and return the data from the corresponding Ledger based on the IndexFile data retrieval technology; Specifically, the inspection module is used for: Maintaining an IndexLedger object and caching IndexFile index data through the IndexLedger object; When there is data writing, reading the complete data of the CommitLOG object; Generating IndexFile data based on the IndexFile index data and the complete data of the CommitLOG object; Recording the IndexFile data into the Ledger object of the IndexFile.

8. The cloud storage system based on RocketMQ according to claim 7, wherein The access interface is used to open the log, append APPEND log content, and / or randomly access one or more log data.

9. The cloud storage system based on RocketMQ according to claim 7, wherein The LedgerManager object contains all Ledger objects of Apache Bookkeeper; the QueueLedger object contains the metadata information of all Ledgers in the queue.

Images