Cloud storage method and system based on rocketmq

Abstract

The present invention belongs to the field of cloud storage, and relates to a cloud storage method and system based on RocketMQ. The method comprises: in Apache RocketMQ distributed messaging, extending a message storage engine; creating an access interface for an Apache Bookeeper distributed logging system; when writing data, acquiring QueueLedger objects, constructing complete data of a CommitLOG object, and calling write interfaces on the QueueLedger objects to perform data cache updating; checking the IndexFile write progress of each QueueLedger object by means of an asynchronous thread; and when reading data, preferentially performing data cache reading, and determining whether there is corresponding data in a data cache, and if so, immediately returning the corresponding data, otherwise, loading the data from a corresponding Ledger on the basis of an IndexFile data retrieval technique and returning the data.

Description

A cloud storage method and system based on RocketMQ TECHNICAL FIELD

[0001] The present application belongs to the field of cloud storage, and particularly relates to a cloud storage method and system based on RocketMQ. BACKGROUND

[0002] Apache RocketMQ is a low-latency, high-concurrency, high-availability and high-reliability distributed message. Because of its simple architecture, rich business functions and strong scalability, it is widely used by many enterprise developers and cloud vendors.

[0003] However, the local storage resources of the current RocketMQ are fixed and cannot be dynamically changed as needed, which limits the flexibility and adaptability of the system when facing different loads and data growth, and easily leads to uneven data distribution, causing some Broker nodes to be overloaded while other nodes are relatively idle, affecting the overall performance and stability of the system. SUMMARY

[0004] In view of the above deficiencies of the prior art, the purpose of the present application is to provide a cloud storage method and system based on RocketMQ, which extends the message storage engine in Apache RocketMQ distributed message, saves message data in a distributed log system, realizes the separation of computing and storage without changing the cluster deployment architecture and fully compatible with the open source ecology, can dynamically change as needed, improves the utilization of resources, improves the flexibility and adaptability of the system when facing different loads and data growth, and improves the overall performance and stability of the system.

[0005] In a first aspect of the present application, a cloud storage method based on RocketMQ is provided, comprising:

[0006] S1, extending the message storage engine in Apache RocketMQ distributed message;

[0007] S2, creating an access interface of Apache Bookkeeper distributed log system;

[0008] S3, extending the access interface to maintain LedgerManager object and QueueLedger object;

[0009] S4, when writing data, obtaining the QueueLedger object, constructing the complete data of CommitLOG object, and calling the write interface on the QueueLedger object to update the data cache;

[0010] S5, the asynchronous thread checks the IndexFile write progress of each QueueLedger object;

[0011] S6, when reading data, the data cache reading is preferentially performed, and it is judged whether the corresponding data exists in the data cache, if yes, the corresponding data is immediately returned, otherwise, based on the IndexFile data retrieval technology, the data is loaded in the corresponding Ledger and returned.

[0012] Further, the access interface is used for opening the log, appending the APPEND log content, and / or randomly accessing one or more pieces of log data.

[0013] Further, the LedgerManager object contains all Apache Bookkeeper Ledger objects; the QueueLedger object contains the metadata information of all Ledgers in the queue.

[0014] Further, the cloud storage method based on RocketMQ further comprises:

[0015] The QueueOffset value is maintained through the QueueLedger object;

[0016] The QueueOffset value is used to determine the physical location of the message data.

[0017] Further, the S5 specifically comprises:

[0018] S501, an IndexLedger object is maintained, and the IndexFile index data is cached through the IndexLedger object;

[0019] S502, when there is data writing, the complete data of the CommitLOG object is read;

[0020] S503, the IndexFile data is generated according to the IndexFile index data and the complete data of the CommitLOG object;

[0021] S504, the IndexFile data is recorded into the Ledger object of the IndexFile.

[0022] Further, the IndexFile data retrieval technology in the S6 specifically comprises:

[0023] S601, the Hash value of the query Key is calculated;

[0024] S602, the Hash value is taken modulo the bucket number of the index data to obtain the offset value of the target bucket;

[0025] S603, it is judged whether the offset value is less than 0, if yes, it is determined that there is no corresponding data, otherwise, the IndexFile data corresponding to the offset of the IndexFile Ledger is read, when the previous-index value of the IndexFile data is greater than or equal to 0, the next IndexFile data is found based on the previous-index value as the offset, until all are found.

[0026] S604, it is compared whether the Hash value of the IndexFile data is consistent with the Hash value of the query Key, if yes, the corresponding CommitLOG data is read and returned.

[0027] Further, the cloud storage method based on RocketMQ further comprises:

[0028] The message storage strategy of each TOPIC is checked in time, the last data writing time of each Ledger in each Queue is obtained through the QueueLedger interface, and the corresponding Ledger data is deleted according to the message storage strategy.

[0029] In a second aspect, a cloud storage system based on RocketMQ is provided, comprising:

[0030] The expansion module is used for expanding the message storage engine in the Apache RocketMQ distributed message.

[0031] The creation module is used for creating an access interface of the Apache Bookkeeper distributed log system.

[0032] The first maintenance module is used for expanding the access interface and maintaining the LedgerManager object and the QueueLedger object.

[0033] The writing module is used for acquiring the QueueLedger object when data is written, constructing complete data of the CommitLOG object, and calling a writing interface on the QueueLedger object to update data caching.

[0034] The checking module is used for checking the IndexFile writing progress of each QueueLedger object in an asynchronous thread.

[0035] The reading module is configured to, when reading data, preferentially perform a data cache reading, determine whether corresponding data exists in the data cache, if yes, immediately return the corresponding data, and if not, load data in the corresponding Ledger based on an IndexFile data searching technology and return the data.

[0036] Further, the access interface is configured to open a log, append APPEND log content, and / or randomly access one or more pieces of log data.

[0037] Further, the LedgerManager object contains all Apache Bookkeeper Ledger objects, and the QueueLedger object contains metadata information of all Ledgers in the queue.

[0038] Further, the cloud storage method based on RocketMQ further includes:

[0039] The second maintenance module is configured to maintain a QueueOffset value through the QueueLedger object.

[0040] The QueueOffset value is used to determine a physical location of message data.

[0041] Further, the checking module is specifically configured to:

[0042] An IndexLedger object is maintained, and IndexFile index data is cached through the IndexLedger object.

[0043] When data writing exists, complete data of a CommitLOG object is read.

[0044] IndexFile data is generated according to the IndexFile index data and the complete data of the CommitLOG object.

[0045] The IndexFile data is recorded into a Ledger object of the IndexFile.

[0046] Further, the reading module is specifically configured to:

[0047] A Hash value of the query Key is calculated.

[0048] The Hash value is subjected to a remainder operation with a bucket number of the index data to obtain an offset value of a target bucket.

[0049] Determine whether the offset value is less than 0, if yes, determine that there is no corresponding data, otherwise, read the IndexFile data corresponding to the offset of the IndexFile Ledger, when the previous-index value of the IndexFile data is greater than or equal to 0, find the next IndexFile data based on the previous-index value as the offset, until all are found;

[0050] Compare the Hash value of the IndexFile data with the Hash value of the query Key, if yes, read the corresponding CommitLOG data and return.

[0051] Further, the cloud storage method based on RocketMQ further comprises:

[0052] The deletion module is configured to check the message storage strategy of each TOPIC at regular time intervals, acquire the last data writing time of each Ledger in each Queue through the QueueLedger interface, and delete the corresponding Ledger data according to the message storage strategy.

[0053] The present application has the following advantages:

[0054] The method and system of the present application expand the message storage engine in Apache RocketMQ distributed messages, save the message data in the distributed log system, realize the separation of calculation and storage without changing the cluster deployment architecture and completely compatible with the open source ecology, can dynamically change as needed, improve the utilization rate of resources, improve the flexibility and adaptability of the system when facing different loads and data growth, and improve the overall performance and stability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0055] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0056] Fig. 1 is a flowchart of a cloud storage method based on RocketMQ according to an embodiment of the present application;

[0057] Fig. 2 is a structural diagram of a cloud storage system based on RocketMQ according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions of the present application will be clearly and completely described below in conjunction with the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. It should be understood that these descriptions are only exemplary, and are not used to limit the scope of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should be within the scope of protection of the present application.

[0059] In addition, in the following description, the description of well-known structures and techniques is omitted to avoid unnecessary confusion of the concepts disclosed in the present application.

[0060] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for description purposes and cannot be understood as indicating or implying relative importance. The terms "mounting", "connection", "connection" should be broadly understood, for example, it can be fixed connection, or detachable connection, or integrally connected; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or the communication between two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0061] The exemplary embodiments will be described in detail herein, with examples shown in the drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Rather, they are merely examples of methods and systems consistent with some aspects of the present application as detailed in the appended claims.

[0062] The present application proposes a RocketMQ-based cloud storage method and system, which solves the problem that the current RocketMQ local storage resource is fixed and cannot dynamically change as needed, which limits the flexibility and adaptability of the system when facing different loads and data growth, easily leading to uneven data distribution, causing some Broker nodes to be overloaded, while other nodes are relatively idle, affecting the overall performance and stability of the system, etc.

[0063] Method embodiments

[0064] Referring to Figure 1 of the accompanying drawings, a flowchart of a cloud storage method based on RocketMQ is shown.

[0065] The cloud storage method based on RocketMQ comprises the following steps.

[0066] Specifically, the method comprises steps S1 to S6.

[0067] S1, in Apache RocketMQ distributed messages, extend the message storage engine.

[0068] Apache RocketMQ is a low-latency, high-concurrency, high-availability, and high-reliability distributed message. Due to its simple architecture, rich business functions, and strong scalability, it is widely used by many enterprise developers and cloud vendors.

[0069] Further, the storage file of RocketMQ contains three parts: CommitLOG for saving data, ConsumeQueue for saving logical Queue, and IndexFile for saving string index.

[0070] It should be noted that extending the storage engine of Apache RocketMQ introduces an external distributed log system. By separating data storage to the distributed log system, the separation of computing and storage is achieved, which can solve the single-point capacity bottleneck problem and improve resource utilization.

[0071] S2, create an access interface of Apache Bookkeeper distributed log system.

[0072] Apache Bookkeeper is a distributed and naturally scalable log system. All data is written in an append manner and cannot be modified once written.

[0073] Optionally, the access interface is used to open the log, append APPEND log content, and / or randomly access one or more log data.

[0074] In the present application, the creation of the access interface of Apache Bookkeeper can bring higher flexibility, performance optimization and data security guarantee to the message storage method based on RocketMQ. Through the design of the interface, the two systems can be effectively integrated to make full use of their respective advantages and provide reliable message processing and storage solutions for applications.

[0075] S3, extend the access interface, maintain the LedgerManager object and the QueueLedger object.

[0076] Optionally, the LedgerManager object contains all the Ledger objects of Apache Bookkeeper.

[0077] Optionally, the QueueLedger object contains the metadata information of all Ledgers in the queue.

[0078] Wherein, the Ledger is a logical unit of Bookkeeeper. A Ledger can be understood as an independent log file or a message queue. The core interface of Ledger has only two operations, one is APPEND data, and the other is random read data.

[0079] S4, when writing data, obtain the QueueLedger object, construct the complete data of the CommitLOG object, and call the write interface on the QueueLedger object to update the data cache.

[0080] In the application, the QueueLedger object is obtained, the complete data of the CommitLOG object is constructed thereon, and then the write interface is called to update the data cache. In this way, the data cache can be effectively managed and updated, and the written message data can be quickly acquired and processed by the system.

[0081] S5, the asynchronous thread checks the IndexFile write progress of each QueueLedger object.

[0082] In a possible implementation, S5 specifically includes sub-steps S501 to S504:

[0083] S501, maintain the IndexLedger object, and cache the IndexFile index data through the IndexLedger object.

[0084] S502, when there is data writing, read the complete data of the CommitLOG object.

[0085] S503, generate the IndexFile data according to the IndexFile index data and the complete data of the CommitLOG object.

[0086] S504, record the IndexFile data to the Ledger object of the IndexFile.

[0087] In the present application, the asynchronous thread checks the IndexFile writing progress of each QueueLedger object by maintaining the IndexLedger object to cache the IndexFile index data. In this way, the asynchronous monitoring of data writing progress can be realized, avoiding blocking the main processing flow, while optimizing the overall performance and response speed of the system.

[0088] Further, each time the IndexFile is written, the index data in the memory needs to be updated. The index data is initially read from the last data of the Ledger of the IndexFile. If there is no last data, the Ledger of the corresponding CommitLog and the Ledger of the IndexFile need to be read in full to reconstruct the data. Each time the program is exited or the index cache is released, the full Index File index data needs to be written to the Ledger of the IndexFile.

[0089] S6, when reading data, the data cache reading is preferentially performed, and it is judged whether the corresponding data exists in the data cache. If yes, the corresponding data is immediately returned, otherwise, based on the IndexFile data retrieval technology, the data is loaded in the corresponding Ledger and returned.

[0090] In the present application, first, data is read from the data cache. If the data already exists in the cache, the corresponding data can be immediately returned, greatly shortening the response time of data access and improving the real-time performance and response performance of the system.

[0091] In one possible implementation, the IndexFile data retrieval technology in S6 specifically includes sub-steps S601 to S604:

[0092] S601, the Hash value of the query Key is calculated.

[0093] S602, the Hash value is taken modulo the number of buckets of the index data to obtain the offset value of the target bucket.

[0094] S603, it is judged whether the offset value is less than 0. If yes, it is determined that there is no corresponding data, otherwise, the IndexFile data corresponding to the offset of the IndexFile Ledger is read. When the previous-index value of the IndexFile data is greater than or equal to 0, the previous-index value is taken as the offset to find the next IndexFile data until all are found.

[0095] S604, compare the Hash value of the IndexFile data with the Hash value of the query Key, if yes, read the corresponding CommitLOG data and return.

[0096] In the present application, the IndexFile data retrieval technique ensures the consistency and integrity of the data. By comparing the Hash value of the IndexFile data with the Hash value of the query Key, the correctness of the data is confirmed before reading the corresponding CommitLOG data and returning. This mechanism ensures that the data read by the system is accurate and complete, avoiding the problem of data errors or loss.

[0097] In one possible implementation, the RocketMQ-based cloud storage method further includes:

[0098] The QueueOffset value is maintained through the QueueLedger object.

[0099] The QueueOffset value is used to determine the physical location of the message data.

[0100] In the present application, the maintenance of QueueOffset allows the system to track the consumed message offset in each message queue. Consumers can know where they need to start consuming messages through these offsets, ensuring the order and consistency of messages. This mechanism is essential for message consumers, especially when dealing with repeated consumption or message re-delivery. By accurately managing the QueueOffset value, consumers can effectively avoid the situation of repeated consumption and loss of messages. This precise message location management can improve the efficiency and reliability of message consumption, ensuring that consumers can consume messages in the expected order and rate.

[0101] In one possible implementation, the RocketMQ-based cloud storage method further includes:

[0102] Check the message storage strategy of each TOPIC regularly, get the last data write time of each Ledger in each Queue through the QueueLedger interface, and delete the corresponding Ledger data according to the message storage strategy.

[0103] In the present application, the old Ledger data in the message storage strategy under each TOPIC is regularly checked and cleaned up through the QueueLedger interface, which is one of the important measures to maintain the efficient and stable operation of RocketMQ system. This approach not only saves resources, but also helps to ensure that the system can effectively support business needs for a long time.

[0104] Further, the application uses a distributed log system to save information data of RocketMQ, realizes separation of calculation and storage, reduces use cost and improves resource utilization, and solves the limitation of single-point storage capacity. Compared with the scheme of saving in object storage of open source RocketMQ, the application has higher data reliability and better performance. Meanwhile, in cooperation with the distributed log system, the application provides support for higher granularity and various data expiration strategies based on topics, is more suitable for cloud native characteristics such as statelessness, facilitates process transfer, and supports dynamic expansion and contraction of nodes.

[0105] The application has the following beneficial effects:

[0106] The method and system of the application expand a message storage engine in Apache RocketMQ distributed messages, save message data in a distributed log system, realize separation of calculation and storage without changing a cluster deployment architecture and completely compatible with an open source ecology, can dynamically change on demand, improve resource utilization, improve flexibility and adaptability of the system when facing different loads and data growth, and improve overall performance and stability of the system.

[0107] System embodiment

[0108] Referring to FIG. 2 of the specification, a structural schematic diagram of a cloud storage system based on RocketMQ provided by an embodiment of the application is shown.

[0109] The cloud storage system 20 based on RocketMQ provided by the embodiment of the application comprises:

[0110] The extension module 201 is configured to expand a message storage engine in Apache RocketMQ distributed messages.

[0111] The creation module 202 is configured to create an access interface of an Apache Bookkeeper distributed log system.

[0112] The first maintenance module 203 is configured to expand the access interface and maintain a LedgerManager object and a QueueLedger object.

[0113] The writing module 204 is configured to acquire the QueueLedger object when writing data, construct complete data of a CommitLOG object, and call a writing interface on the QueueLedger object to update data caching.

[0114] The checking module 205 is configured to check IndexFile writing progress of each QueueLedger object by an asynchronous thread.

[0115] The reading module 206 is used for, when reading data, performing data cache reading preferentially, judging whether corresponding data exists in the data cache, if yes, returning the corresponding data immediately, otherwise, loading data in the corresponding Ledger based on the IndexFile data retrieval technology and returning.

[0116] Further, the access interface is used for opening the log, appending APPEND log content, and / or randomly accessing one or more pieces of log data.

[0117] Further, the LedgerManager object contains all Apache Bookkeeper Ledger objects; and the QueueLedger object contains metadata information of all Ledgers in the queue.

[0118] Further, the cloud storage method based on RocketMQ further comprises the following steps.

[0119] The second maintenance module is used for maintaining the QueueOffset value through the QueueLedger object.

[0120] The QueueOffset value is used for determining the physical location of the message data.

[0121] Further, the checking module 205 is specifically used for:

[0122] The IndexLedger object is maintained, and the IndexFile index data is cached through the IndexLedger object.

[0123] When there is data writing, the complete data of the CommitLOG object is read;

[0124] According to the IndexFile index data and the complete data of the CommitLOG object, the IndexFile data is generated.

[0125] The IndexFile data is recorded into the Ledger object of the IndexFile.

[0126] Further, the reading module 206 is specifically used for:

[0127] The Hash value of the query Key is calculated.

[0128] The Hash value is taken modulo the bucket number of the index data to obtain the offset value of the target bucket.

[0129] Determine whether the offset value is less than 0, if yes, determine that there is no corresponding data, otherwise, read the IndexFile data corresponding to the offset of the IndexFile Ledger, when the previous-index value of the IndexFile data is greater than or equal to 0, find the next IndexFile data based on the previous-index value as the offset, until all are found.

[0130] Compare the Hash value of the IndexFile data with the Hash value of the query Key, if yes, read the corresponding CommitLOG data and return.

[0131] Further, the cloud storage method based on RocketMQ further comprises:

[0132] The deleting module is used for checking the message storage strategy of each TOPIC at regular time intervals, acquiring the last data writing time of each Ledger in each Queue through the QueueLedger interface, and deleting the corresponding Ledger data according to the message storage strategy.

[0133] The cloud storage system 20 based on RocketMQ provided by the application can realize each step and technical effect of the cloud storage method based on RocketMQ, and the application will not be described again to avoid repetition.

[0134] The application has the following beneficial effects:

[0135] The method and system of the application expand the message storage engine in the Apache RocketMQ distributed message, save the message data in the distributed log system, realize the separation of calculation and storage without changing the cluster deployment architecture and completely compatible open source ecology, can dynamically change according to needs, improve the utilization rate of resources, improve the flexibility and adaptability of the system when facing different loads and data growth, and improve the overall performance and stability of the system.

[0136] The application applicant has made a detailed description and explanation of the embodiment examples of the application in combination with the drawings of the specification, but the person skilled in the art should understand that the above embodiment examples are only preferred embodiments of the application, and the detailed description is only to help the reader better understand the spirit of the application, and is not a limitation on the protection scope of the application, on the contrary, any improvement or modification based on the spirit of the application should fall within the protection scope of the application.

[0137] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, but not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered within the protection scope of the present application.

Claims

1. A cloud storage method based on RocketMQ, characterized in that, include: S1 extends the message storage engine in Apache RocketMQ distributed messaging; S2, Create an access interface for the Apache Bookkeeper distributed logging system; S3, extend the access interface and maintain the LedgerManager object and the QueueLedger object; S4. When writing data, obtain the QueueLedger object, construct the complete data of the CommitLOG object, and call the write interface on the QueueLedger object to update the data cache. S5, An asynchronous thread checks the IndexFile write progress of each QueueLedger object; S6: When reading data, the data cache is read first to determine if the corresponding data exists in the data cache. If it does, the corresponding data is returned immediately. Otherwise, the data is loaded from the corresponding Ledger based on IndexFile data retrieval technology and returned.

2. The cloud storage method based on RocketMQ according to claim 1, characterized in that, The access interface is used to open logs, append log content, and / or randomly access one or more log data.

3. The cloud storage method based on RocketMQ according to claim 1, characterized in that, The LedgerManager object contains all the Ledger objects in Apache Bookkeeper; the QueueLedger object contains metadata information for all Ledgers in the queue.

4. The cloud storage method based on RocketMQ according to claim 1, characterized in that, Also includes: The QueueOffset value is maintained through the QueueLedger object; The QueueOffset value is used to determine the physical location of the message data.

5. The cloud storage method based on RocketMQ according to claim 1, characterized in that, S5 specifically includes: S501, Maintain an IndexLedger object, and cache IndexFile index data through the IndexLedger object; S502, when data is being written, read the complete data from the CommitLOG object; S503, Generate IndexFile data based on the IndexFile index data and the complete data of the CommitLOG object; S504, the IndexFile data is recorded into the IndexFile's Ledger object.

6. The cloud storage method based on RocketMQ according to claim 1, characterized in that, The IndexFile data retrieval technology in S6 specifically includes: S601, Calculate the hash value of the query key; S602, take the remainder between the hash value and the number of buckets of the index data to obtain the offset value of the target bucket; S603, determine if the offset value is less than 0. If it is, determine that there is no corresponding data. Otherwise, read the IndexFile data corresponding to the offset of the IndexFile Ledger. When the previous-index value of the IndexFile data is greater than or equal to 0, use the previous-index value as the offset to find the next IndexFile data until all data have been found. S604: Compare the hash value of the IndexFile data with the hash value of the query key. If they match, read the corresponding CommitLOG data and return it.

7. The cloud storage method based on RocketMQ according to claim 1, characterized in that, Also includes: Periodically check the message storage policy of each TOPIC, obtain the last data write time of each Ledger in each Queue through the QueueLedger interface, and delete the corresponding Ledger data according to the message storage policy.

8. A cloud storage system based on RocketMQ, characterized in that, include: Extension modules are used to extend the message storage engine in Apache RocketMQ distributed messaging. Create a module to create the access interface for the Apache Bookkeeper distributed logging system; The first maintenance module is used to extend the access interface and maintain the LedgerManager object and the QueueLedger object; The writing module is used to obtain the QueueLedger object when writing data, construct the complete data of the CommitLOG object, and call the writing interface on the QueueLedger object to update the data cache. The inspection module is used for asynchronous thread inspection of the IndexFile write progress of each QueueLedger object; The read module is used to prioritize reading the data cache when reading data, and to determine whether the corresponding data exists in the data cache. If it does, the corresponding data is returned immediately; otherwise, the data is loaded from the corresponding Ledger based on IndexFile data retrieval technology and returned.

9. The cloud storage system based on RocketMQ according to claim 8, characterized in that, The access interface is used to open logs, append log content, and / or randomly access one or more log data.

10. The cloud storage system based on RocketMQ according to claim 8, characterized in that, The LedgerManager object contains all the Ledger objects in Apache Bookkeeper; the QueueLedger object contains metadata information for all Ledgers in the queue.

Images