Proxy bridging method, system, readable storage medium and device for time-sequential queue

Abstract

The embodiment of the application provides a kind of timing queue's agent bridging method, system, readable storage medium and equipment, it is related to timing queue storage technical field, this method includes: receiving timing queue by business agent end, for write message, write message is converted to form the new parameter of write message;Through sidecar, generate call instruction based on the new parameter of write message, according to call instruction, call the message write interface of message proxy bridge node;Write message is sent to target message proxy bridge node by message write interface, and write message is written into the rewrite log of target message proxy bridge node;Rewrite log is sent to the storage space of existing product asynchronously, for subsequent operation related based on rewrite log.Therefore, it can be achieved without upgrading Redis version, and it is not necessary to use Redis as storage, so as to reduce the time cost and economic cost of solving the problem of using existing Redis storage space.

Description

Technical Field

[0001] This invention relates to the field of time-series queue storage technology, and specifically to a proxy bridging method, system, readable storage medium, and device for time-series queues. Background Technology

[0002] As the development of information technology innovation deepens, higher versions of Redis (originally the storage medium used by time-series queues) increasingly reflect the commercialization of US-listed companies. This is because the changed commercial open-source license has many potential risks and is not suitable for upgrading.

[0003] Therefore, existing products have remained at the major Redis version 5.0. However, the official approach of forcing users to upgrade by exploiting high-risk vulnerabilities (in October 2024, the official team disclosed three high-risk vulnerabilities, requiring users to upgrade to Redis version 6.2.17 or higher, but actual testing showed that these vulnerabilities only existed in commercial features added in versions 6.0 and above). Domestic security companies' related vulnerability scanning rules and handling suggestions were overly aggressive, causing significant confusion for the security rating of projects using Redis (a message queue storage system).

[0004] For existing systems, i.e., systems currently using Redis storage versions lower than 5.0, replacing these existing Redis storage versions would require adaptive modifications and full regression testing of the entire business system, resulting in very high time and economic costs. Therefore, a method is needed that is application-compatible, utilizes existing Redis SDKs without replacement, and is low-cost for storing messages in time-series queues. Summary of the Invention

[0005] This invention provides a proxy bridging method, system, readable storage medium, and device for time-series queues to solve the problems in the prior art.

[0006] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a proxy bridging method for time-series queues, implemented based on an application server and a message broker bridge node. The method interfaces with the storage space of existing products through the application server and the message broker bridge node. The storage space of the existing products includes Redis storage space. The application server includes a business proxy and a sidecar. The proxy bridging method for time-series queues includes:

[0007] The service agent receives a time-series queue, and for each write message in the time-series queue, it transforms the write message to form new parameters for the write message.

[0008] The sidecar generates a call instruction based on the new parameters for writing messages, and calls the message writing interface of the message broker bridge node according to the call instruction, and uses the called message broker bridge node as the target message broker bridge node.

[0009] The write message is sent to the target message broker bridge node through the message write interface, and the write message is written to the rewrite log of the target message broker bridge node.

[0010] The rewrite log is asynchronously sent to the storage space of the existing product for subsequent operations based on the rewrite log.

[0011] Secondly, embodiments of the present invention provide a proxy bridging system for time-series queues, including an application server and a message proxy bridge node. The system interfaces with the storage space of existing products, including Redis storage space, through the application server and the message proxy bridge node. The application server includes a business proxy and a sidecar, wherein:

[0012] The business agent is used to receive the time-series queue, and for each write message in the time-series queue, to transform the write message into new parameters for the write message;

[0013] The sidecar is used to generate a call instruction based on the new parameters for writing messages, call the message writing interface of the message broker bridge node according to the call instruction, and take the called message broker bridge node as the target message broker bridge node.

[0014] The target message broker bridge node in the message broker bridge node is used to receive the write message through the message write interface, write the write message into the rewrite log of the target message broker bridge node; and asynchronously send the rewrite log to the storage space of the existing product for related subsequent operations based on the rewrite log.

[0015] Thirdly, embodiments of the present invention provide a computer-readable storage medium that stores one or more programs, which, when executed by a computer device, cause the computer device to perform the aforementioned proxy bridging method for a time-series queue.

[0016] Fourthly, embodiments of the present invention provide a computer device, comprising:

[0017] A processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the aforementioned proxy bridging method for a time-series queue.

[0018] The above technical solution has the following beneficial effects:

[0019] For existing systems, i.e., systems currently using Redis storage versions lower than 5.0, the existing Redis storage system is not used directly. Instead, the time-series queue is received directly through the business agent, and the messages in the time-series queue are written directly to the rewrite log of the target message broker bridge node via a sidecar call. This satisfies the storage requirements of messages in the time-series queue with strict time-series requirements. Furthermore, the rewrite log is asynchronously sent to the existing Redis storage space. This process uses the UDS (Unix Domain Socket) inter-process communication protocol, bypassing the network, thus enabling subsequent operations based on the rewrite log (i.e., based on the stored messages with strict time-series requirements).

[0020] The proxy bridging method of this invention stores messages with strict timing requirements. Redis storage assists in tasks other than storing these messages, thus avoiding direct network exposure of Redis. Instead, auditing instructions are sent to Redis only through the proxy. This eliminates the need to replace existing systems using Redis storage versions lower than 5.0, or to rely on Redis for storage altogether, thereby reducing the time and economic costs of resolving issues arising from using existing Redis storage. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a proxy bridging method for a time-series queue according to an embodiment of the present invention;

[0023] Figure 2 This is a logical structure diagram of a proxy bridging system for a time-series queue according to an embodiment of the present invention;

[0024] Figure 3 This is a logical structure diagram of a computer device according to an embodiment of the present invention;

[0025] Figure 4 This is an overall architecture diagram of an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, in conjunction with embodiments of the present invention, a proxy bridging method for time-series queues is provided. This method is implemented based on an application server and a message broker bridge node. The application server and the message broker bridge node interface with the storage space of existing products, including Redis storage space. The application server includes a business proxy and a sidecar. The proxy bridging method for time-series queues includes:

[0028] S101: Receive the time-series queue through the business agent, and for each write message in the time-series queue, transform the write message to form new parameters for the write message;

[0029] S102: The sidecar generates a call instruction based on the new parameters for writing messages, and calls the message writing interface of the message broker bridge node according to the call instruction, and uses the called message broker bridge node as the target message broker bridge node.

[0030] S103: Send the write message to the target message broker bridge node through the message write interface, and write the write message into the rewrite log of the target message broker bridge node;

[0031] S104: The rewrite log is asynchronously sent to the storage space of the existing product for subsequent operations based on the rewrite log.

[0032] For existing systems, specifically those already using Redis storage based on version 5.0 (currently used for storing message queues), the system doesn't directly utilize the existing Redis storage. Instead, it receives the time-series queue directly through the business agent and uses a sidecar to call the target message broker bridge node to write the messages in the time-series queue directly to the rewrite log of the target message broker bridge node, thus satisfying the storage requirements for messages in time-series queues with strict timing requirements. Furthermore, the rewrite log is asynchronously sent to the existing Redis storage. This process uses the UDS (Unix DomainSocket) inter-process communication protocol, bypassing the network, and allows for subsequent operations based on the rewrite log (i.e., based on the stored messages with strict timing requirements).

[0033] The proxy bridging method of this invention stores messages with strict timing requirements. Redis storage assists in tasks other than storing these messages, thus eliminating the need to upgrade the Redis version. Therefore, it is not mandatory to use Redis as the storage solution. This avoids replacing existing Redis storage systems with versions lower than 5.0, thereby reducing the time and economic costs associated with resolving issues related to using existing Redis storage and upgrading or replacing it.

[0034] Preferably, in S101, for each write message in the time-series queue, the write message is transformed to form new parameters for the write message, including:

[0035] S101-1: For each write message in the time-series queue, the producer in the business agent parses the write message to obtain the message content, and writes the message content to the sidecar.

[0036] S101-2: The message content is sent to the target message proxy bridge node by calling the message writing interface of the message proxy bridge node through the side vehicle, with the called message proxy bridge node as the target message proxy bridge node;

[0037] S101-3: Based on the message timestamp generated by the target message broker bridge node according to the message content, the storage offset of the written message is determined by the consumer in the business broker terminal, wherein the new parameters of the written message include the message content and the storage offset.

[0038] The sidecar is located on the application server, and inter-process communication within the node enables the sending of write information to subsequent processes, thus not exposing Redis to the network.

[0039] Preferably, in S102, the process of generating a call instruction based on new parameters for writing messages using the sidecar, and then calling the message writing interface of the message broker bridge node according to the call instruction, includes:

[0040] S102-1: Generate a message ID using the sidecar based on the storage offset and the message timestamp;

[0041] S102-2: Based on the message content and the message ID, a call instruction is generated by the sidecar, and the message writing interface of the target message broker bridge node is called according to the call instruction.

[0042] The purpose of this invention is to transform the write information in the time-series queue, which originally required Redis storage space, into a unified data format specified by the message writing interface of the target message broker bridge node that can be called by the sidecar.

[0043] Preferably, the proxy bridging method for the time-series queue further includes:

[0044] Before writing the message content, acquire a distributed lock and lock the sidecar using the distributed lock;

[0045] The acquisition of the distributed lock includes:

[0046] If the existing product's storage space only includes Redis storage space, a distributed lock is acquired after multi-layer verification before transmitting write messages to the Redis storage space;

[0047] The acquired distributed lock includes 4-5 layers of verification. The specific verification layers are: the first layer is a timestamp, the second layer is the lock queue, the third layer is the GETSET operation of the lock value, the fourth layer is the timeout setting of the lock value, and the fifth layer is a lease lock, which can be merged with the fourth layer depending on the situation. Through multi-layered verification, a distributed exclusive lock can be simultaneously effective in three dimensions in non-atomic operations: multiple clients, multiple partition concurrency, and concurrent read / write operations on the same partition. This allows for locking the sidecar using the distributed lock.

[0048] Preferably, if the existing product's storage space only includes Redis storage space, a distributed lock is acquired after multi-level verification before transmitting write messages to the Redis storage space, including:

[0049] If Redis storage space is used as the storage space, for the write message, the Redis STREAM data structure is used as the lock queue, the corresponding timestamp obtained from the lock queue is used to generate a distributed instruction, and the initial timestamp is returned;

[0050] Convert the initial timestamp into an initial intermediate timestamp suitable for Redis, represented using double-precision fractions;

[0051] The client ID that sends the write message and the initial intermediate timestamp to be locked are written into a lock queue in Redis that is a ZSET data structure named with a specific distributed lock name. The specific distributed lock queue includes the corresponding initial timestamps generated for the lock requirements of different data transmission tools, including sidecars.

[0052] Obtain the head of the queue in the lock queue, wherein the word of the head of the queue includes the client ID and a double-precision fraction of the initial intermediate timestamp;

[0053] The success of acquiring a specific distributed lock is determined based on the head of the queue.

[0054] Specifically, when using Redis storage, the message content received by the target message broker bridge node uses the Redis STREAM data structure as a lock queue. The lock queue is used to obtain timestamps and generate distributed instructions, returning the initial timestamp. The data written to the Redis STREAM (character sequence) data structure is generated according to time sequence, with the following rules: milliseconds (starting from 00:00 on January 1, 1970), hyphens "-", and a 64-bit binary sequence number (determining the order in which the same millisecond occurs, with a maximum of 2^64 bits). 63 Redis's actual performance is 100,000 to 200,000 requests per second, so retaining only 4 digits for the sequence number (corresponding to the generation of millions of data per second) is sufficient for its use.

[0055] The initial timestamp is converted into an initial intermediate timestamp represented as a double-precision fraction; specifically, the rules for converting a timestamp into a double-precision number (score) are as follows:

[0056] 1) For the integer part, subtract 20000000000000 from the milliseconds (because the precision of double-precision numbers is insufficient to support 13 integer digits plus 4 decimal digits, the first digit needs to be truncated, so the supported time range only extends to 2033. Consider using its negative range. Taking the first two digits of the current time point's timestamp as 17, the maximum supported timestamp is 2700000000000, which is 2055. Subsequently, as needed, after 2033, i.e., after reaching 2000000000000, the time can be changed to 3000000000000 through system switching, which can support up to 2065; after reaching 3000000000000 in 2065, it can be changed to 40000000000000, which can support up to 2096; and so on).

[0057] 2) For the decimal part, retain 4 digits of the serial number as 4 decimal places. Depending on the sign of the integer part: if the integer part is negative, subtract 1; if the integer part is positive, directly use it as the decimal part.

[0058] The initial timestamp is converted into an initial intermediate timestamp represented using double-precision fractions; the client ID of the sender of the write message to be locked and the initial intermediate timestamp are written into a lock queue in Redis named after the distributed lock. This distributed lock queue includes the corresponding initial timestamps generated for the lock requirements of different business agents; the lock queue adopts a ZSET data structure, which is an existing technology.

[0059] Obtain the head of the lock queue, where the head includes the client ID and a double-precision fraction; determine whether the acquisition of a specific distributed lock was successful based on the head.

[0060] When the client ID of the head of the queue is the same as the client ID of the sender of the write message to be locked, and the double-precision fraction of the head of the queue is equal to the initial intermediate timestamp, an atomic operation is performed to set the lock value of the distributed lock command. The lock value is the current intermediate timestamp represented by a string with 3 decimal places, and the old timestamp of the original lock value of the initial specific distributed lock command is returned.

[0061] When the old timestamp does not exist, all data in the lock queue that is less than the initial intermediate timestamp by a preset duration is cleared. Then, the head of the queue is retrieved again. If the value of the head of the queue is equal to the initial intermediate timestamp, it indicates that the distributed lock acquisition was successful; otherwise, the distributed lock acquisition failed. Because it is concurrent processing, extreme cases (such as Java garbage collection and severe network latency occurring between two instructions, during which another client performs cleanup of expired data) may cause the initial intermediate timestamp to not exist.

[0062] When the old timestamp is greater than the initial intermediate timestamp, it indicates that the distributed lock has been acquired by another client (the client with the later timestamp). The client then relinquishes the lock acquisition and clears its own data from the lock queue.

[0063] When the old timestamp is less than the initial intermediate timestamp or there is no initial intermediate timestamp, the head of the queue is retrieved again. If the value of the head of the queue is equal to the initial intermediate timestamp, it indicates that the distributed lock has been successfully acquired. Otherwise, the distributed lock acquisition fails, and the acquisition of the lock is abandoned and the data in the lock queue is cleared.

[0064] After successfully acquiring the distributed lock, use SETE to update the lock value based on the latest timestamp and simultaneously set the timeout for the distributed lock.

[0065] After successfully acquiring the distributed lock, use SETE to update the lock value based on the latest timestamp and simultaneously set the timeout for the distributed lock.

[0066] When the lock's effective scope is limited to multiple clients, the lock value is converted into a lease lock, and its key name follows the atomic instruction required by the API, meaning the lock value's key name has business meaning. When the lock's effective scope also needs to be effective across concurrent read / write operations or across multiple API instructions in terms of time dimension, the lock value does not contain business meaning. Instead, a separate lease lock is added to carry the business meaning, provided the lock value is effective.

[0067] When Redis storage is not used, an interface is extended on the coordination center to implement read and write operations on the lock queue through a linked list in the virtual machine memory (JVM memory) of the master node. Specifically, the timestamp, lock queue, and lock value are used from the data structure within the single JVM master node of the coordination center; the atomicity of the timestamp is ensured by the single nature of the JVM master node of the coordination center; the lock queue uses a lock table for data storage; and the lock value uses the JVM's locking mechanism (synchronized) to ensure the exclusivity of the read and write processes.

[0068] Preferably, the proxy bridging method for the time-series queue further includes:

[0069] After the consumer in the business agent determines the storage offset of the write message, before submitting the storage offset to the sidecar, after confirming that the lease of the distributed lock is valid, the lease of the distributed lock is executed, and the action of submitting the storage offset is locked by the distributed lock to ensure that the action of submitting the offset holds the lease.

[0070] The steps for determining the validity of a distributed lock lease include:

[0071] Based on the initial intermediate timestamp, check whether the initial intermediate timestamp exists in the lock queue and the lock value respectively;

[0072] If the initial intermediate timestamp is not present in both the lock queue and the lock value, the lease of the distributed lock expires, and the sidecar does not perform the action of submitting the storage offset.

[0073] If the initial intermediate timestamp exists in both the lock queue and the lock value, then the lease of the distributed lock is valid.

[0074] Only locks / leases with expiration times are valid. The expiration time is set to prevent deadlocks from being released. Whether a lock can be successfully acquired depends on the verification of the head of the queue in the ZSET.

[0075] Preferably, S103: Sending the write message to the target message broker bridge node through the message write interface, and writing the write message to the rewrite log of the target message broker bridge node, includes:

[0076] S103-1: Send the message content and the message ID to the target message broker bridge node through the message writing interface;

[0077] S103-2: The write message is written into the rewrite log of the target message broker bridge node. Within the message log of the rewrite log, the message ID and the message content are sequentially and intermittently written to the last line of the rewrite log based on the message ID, and the line number of the last line is obtained. Simultaneously, the message filename is determined. The message filename refers to the segment number under the partition directory where the write message is located. The segment number refers to the sequence number set for each segment when the message is divided into segments with a fixed number of message lines under the partition directory.

[0078] S103-3: Within the write index log of the rewrite log, construct the write index log file name of the write message, wherein the write index log file name includes the message file name and the storage offset.

[0079] By using the redo log of the message broker bridge node to store the write messages of the time-series queue, high availability, reliability and consistency are achieved, thus avoiding the use of Redis for storage. Redis is only used for related subsequent operations based on the redo log. In other words, high availability, reliability and consistency do not depend on the performance of Redis.

[0080] Preferably, the proxy bridging method for the time-series queue may further include:

[0081] S105: Write the rewrite log to the disk. On the disk, the queue name of the time-series queue is used as the first-level storage directory, and the partition number corresponding to the partition directory is used as the second-level storage directory.

[0082] Because the rewrite log of the message broker bridge node is statically stored, the rewrite log itself is only temporary storage and will not exist forever, so it cannot be used for persistent storage. Therefore, it needs to be written to disk for persistent storage.

[0083] Preferably, the proxy bridging method for the time-series queue may further include:

[0084] S106: In the sharding mechanism of the coordination center node, one node is periodically determined from multiple message broker bridge nodes as the sharding master node, and the other nodes are as slave nodes. The sharding master node and the slave nodes are broadcast to the sidecar and the configuration of each node for routing use; wherein, the target message broker bridge node refers to the sharding master node.

[0085] The proxy bridging method for the time-series queue may further include:

[0086] S107: When the write message is sent to the target message broker bridge node through the message write interface and the write message is written into the rewrite log of the target message broker bridge node, the target broker node simultaneously sends the write message write request to at least one slave node, requesting that the write message be written to at least one slave node.

[0087] S108: After at least one slave node returns a storage success message, return a message indicating that the write message has been successfully stored to the sidecar.

[0088] The embodiments of the present invention employ a method of simultaneous writing to sharded master nodes and slave nodes to back up write messages, which helps to avoid data loss.

[0089] Preferably, the proxy bridging method for the time-series queue may further include:

[0090] S109: For each read message in the time-series queue, based on the consumer's lease, the sidecar determines within the coordination center node whether the consumer has the permission to obtain the read message, wherein the consumer is located within the business agent. The coordination center node uses a scheduled task to assign different consumers to different partitions of a time-series queue to ensure that the partitions of each time-series queue are distributed as evenly as possible among the effective consumers, thereby improving the efficiency of the consumers.

[0091] S110: A consumer with permission to read the message reads the corresponding message content from the storage space of the existing product; wherein, when the storage space of the existing product is a Redis storage space, the consumer with permission to read the message reads the storage offset of the message reads, sends the storage offset of the message reads to the target proxy bridge node and the slave node, sends the storage offset of the message reads to the Redis storage space through the message middleware bridging interface of the target proxy bridge node, and receives the message content returned by the Redis storage space based on the storage offset of the message reads.

[0092] Preferably, the storage space of the existing product also includes at least one storage space that runs parallel to the Redis storage space;

[0093] In S104, sending the rewrite log to the storage space of the existing product includes:

[0094] The rewrite logs are sent to any one or more of the existing Redis storage space or other storage spaces for storage as needed through the message middleware bridging interface of the target proxy bridge node.

[0095] Other storage spaces are extensions of Redis storage spaces, enabling existing platforms to operate independently of Redis storage spaces.

[0096] Preferably, the proxy bridging method for the time-series queue may further include:

[0097] S120: Set the interface call protocol used between servers to an interface call protocol other than the Redis protocol; the server includes at least: an application server, a message broker bridge node, a coordination center node, a Redis storage space, and a storage space parallel to the Redis storage space;

[0098] S130: Sets up Redis protocol-compatible calls between internal server processes, and the calls do not go through the network card, making it impossible for external parties to access the internal server.

[0099] This embodiment does not expose the Redis port externally, and the interface call protocol for external access has been changed. All Redis protocol communication is implemented through inter-process communication of UDS (Unix Domain Socket) inside the server, without going through the network.

[0100] like Figure 2 As shown, in conjunction with an embodiment of the present invention, a proxy bridging system for time-series queues is provided, including an application server 21 and a message proxy bridge node 22. The application server and the message proxy bridge node 22 interface with the storage space of existing products, including Redis storage space. The application server 21 includes a business proxy terminal 211 and a sidecar 212, wherein:

[0101] The business agent 211 is used to receive the time-series queue, and for each write message in the time-series queue, to transform the write message to form new parameters for the write message;

[0102] The sidecar 212 is used to generate a call instruction based on the new parameters for writing messages, call the message writing interface of the message broker bridge node according to the call instruction, and take the called message broker bridge node as the target message broker bridge node.

[0103] The target message broker bridge node in message broker bridge node 22 is used to receive the write message through the message write interface, write the write message into the rewrite log of the target message broker bridge node; and asynchronously send the rewrite log to the storage space of the existing product for related subsequent operations based on the rewrite log.

[0104] For existing systems, i.e., systems currently using Redis storage versions lower than 5.0, the existing Redis storage system is not used directly. Instead, the time-series queue is received directly through the business agent, and the messages in the time-series queue are written directly to the rewrite log of the target message broker bridge node via a sidecar call. This satisfies the storage requirements of messages in the time-series queue with strict time-series requirements. Furthermore, the rewrite log is asynchronously sent to the existing Redis storage space. This process uses the UDS (Unix Domain Socket) inter-process communication protocol, bypassing the network, thus enabling subsequent operations based on the rewrite log (i.e., based on the stored messages with strict time-series requirements).

[0105] The proxy bridging method of this invention stores messages with strict timing requirements. Redis storage assists in tasks other than storing these messages, thus preventing Redis from being exposed to the network. Instructions are only sent to Redis after being audited by the proxy bridge. This eliminates the need to replace existing Redis storage systems (versions lower than 5.0) or use other storage solutions, thereby reducing the time and economic costs of resolving issues arising from using existing Redis storage.

[0106] Preferably, the business agent 211 is specifically used to parse the write message for each write message in the time-series queue through the producer in the business agent to obtain the message content, and write the message content to the sidecar.

[0107] The sidecar 212 is specifically used to call the message writing interface of the message broker bridge node, take the called message broker bridge node as the target message broker bridge node, and send the message content to the target message broker bridge node.

[0108] The target message broker bridge node is used to determine the storage offset of the written message through a consumer in the business broker based on the message timestamp generated by the message content. The new parameters of the written message include the message content and the storage offset.

[0109] The sidecar is located on the application server, and write information is sent to subsequent processes through inter-process communication within the node. Therefore, the Redis port is not exposed externally. All Redis protocol communication is achieved through inter-process communication of UDS (Unix DomainSocket) within the server, without going through the network. This avoids related remote code execution vulnerabilities.

[0110] Preferably, the sidecar 212 is specifically used for:

[0111] Generate a message ID based on the storage offset and the message timestamp;

[0112] Based on the message content and the message ID, a call instruction is generated, and the message writing interface of the target message broker bridge node is called according to the call instruction.

[0113] The purpose is to transform the write information in the time-series queue, which is originally stored in Redis storage space, into a unified data format that can be called by the target message broker bridge node's message write interface.

[0114] Preferably, the proxy bridging system for the time-series queue further includes:

[0115] The distributed lock acquisition unit is used to acquire a distributed lock before writing the message content and lock the sidecar using the distributed lock;

[0116] The distributed lock acquisition unit is specifically used for:

[0117] If the existing product's storage space only includes Redis storage space, a distributed lock is acquired after multi-layer verification before transmitting write messages to the Redis storage space;

[0118] The acquired distributed lock includes 4-5 layers of verification. The specific verification layers are: the first layer is a timestamp, the second layer is the lock queue, the third layer is the GETSET operation of the lock value, the fourth layer is the timeout setting of the lock value, and the fifth layer is a lease lock, which can be merged with the fourth layer depending on the situation. Through multi-layered verification, a distributed exclusive lock can be simultaneously effective in three dimensions in non-atomic operations: multiple clients, multiple partition concurrency, and concurrent read / write operations on the same partition. This allows for locking the sidecar using the distributed lock.

[0119] Preferably, if the existing product's storage space only includes Redis storage space, a distributed lock is acquired after multi-level verification before transmitting write messages to the Redis storage space, including:

[0120] If Redis storage space is used as the storage space, for the write message, the Redis STREAM data structure is used as the lock queue, the corresponding timestamp obtained from the lock queue is used to generate a distributed instruction, and the initial timestamp is returned;

[0121] Convert the initial timestamp into an initial intermediate timestamp suitable for Redis, represented using double-precision fractions;

[0122] The client ID that sends the write message and the initial intermediate timestamp to be locked are written into a lock queue in Redis that is a ZSET data structure named with a specific distributed lock name. The specific distributed lock queue includes the corresponding initial timestamps generated for the lock requirements of different data transmission tools, including sidecars.

[0123] Obtain the head of the queue in the lock queue, wherein the word of the head of the queue includes the client ID and a double-precision fraction of the initial intermediate timestamp;

[0124] The success of acquiring a specific distributed lock is determined based on the head of the queue.

[0125] Specifically, when using Redis storage, the message content received by the target message broker bridge node uses the Redis STREAM data structure as a lock queue. The lock queue is used to obtain timestamps and generate distributed instructions, returning the initial timestamp. The data written to the STREAM data structure by Redis is generated according to a time sequence, with the following rules: milliseconds (starting from 00:00 on January 1, 1970), hyphens "-", and a 64-bit binary sequence number (determining the order in which the same millisecond occurs, with a maximum of 2^32). 63 Redis's actual performance is 100,000 to 200,000 requests per second, so retaining only 4 digits for the sequence number (corresponding to the generation of millions of data per second) is sufficient for its use.

[0126] The initial timestamp is converted into an initial intermediate timestamp represented as a double-precision fraction; specifically, the rules for converting a timestamp into a double-precision number (score) are as follows:

[0127] 1) For the integer part, subtract 20000000000000 from the milliseconds (because the precision of double-precision numbers is insufficient to support 13 integer digits plus 4 decimal digits, the first digit needs to be truncated, so the supported time range only extends to 2033. Consider using its negative range. Taking the first two digits of the current time point's timestamp as 17, the maximum supported timestamp is 2700000000000, which is 2055. Subsequently, as needed, after 2033, i.e., after reaching 2000000000000, the time can be changed to 3000000000000 through system switching, which can support up to 2065; after reaching 3000000000000 in 2065, it can be changed to 40000000000000, which can support up to 2096; and so on).

[0128] 2) For the decimal part, retain 4 digits of the serial number as 4 decimal places. Depending on the sign of the integer part: if the integer part is negative, subtract 1; if the integer part is positive, directly use it as the decimal part.

[0129] The initial timestamp is converted into an initial intermediate timestamp represented using double-precision fractions; the client ID of the sender of the write message to be locked and the initial intermediate timestamp are written into a lock queue in Redis named after the distributed lock. This distributed lock queue includes the corresponding initial timestamps generated for the lock requirements of different business agents; the lock queue adopts a ZSET data structure, which is an existing technology.

[0130] Obtain the head of the lock queue, where the head includes the client ID and a double-precision fraction; determine whether the acquisition of a specific distributed lock was successful based on the head.

[0131] When the client ID of the head of the queue is the same as the client ID of the sender of the write message to be locked, and the double-precision fraction of the head of the queue is equal to the initial intermediate timestamp, an atomic operation is performed to set the lock value of the distributed lock command. The lock value is the current intermediate timestamp represented by a string with 3 decimal places, and the old timestamp of the original lock value of the initial specific distributed lock command is returned.

[0132] When the old timestamp does not exist, all data in the lock queue that is less than the initial intermediate timestamp by a preset duration is cleared. Then, the head of the queue is retrieved again. If the value of the head of the queue is equal to the initial intermediate timestamp, it indicates that the distributed lock acquisition was successful; otherwise, the distributed lock acquisition failed. Because it is concurrent processing, extreme cases (such as Java garbage collection and severe network latency occurring between two instructions, during which another client performs cleanup of expired data) may cause the initial intermediate timestamp to not exist.

[0133] When the old timestamp is greater than the initial intermediate timestamp, it indicates that the distributed lock has been acquired by another client (the client with the later timestamp). The client then relinquishes the lock acquisition and clears its own data from the lock queue.

[0134] When the old timestamp is less than the initial intermediate timestamp or there is no initial intermediate timestamp, the head of the queue is retrieved again. If the value of the head of the queue is equal to the initial intermediate timestamp, it indicates that the distributed lock has been successfully acquired. Otherwise, the distributed lock acquisition fails, and the acquisition of the lock is abandoned and the data in the lock queue is cleared.

[0135] After successfully acquiring the distributed lock, use SETE to update the lock value based on the latest timestamp and simultaneously set the timeout for the distributed lock.

[0136] After successfully acquiring the distributed lock, use SETE to update the lock value based on the latest timestamp and simultaneously set the timeout for the distributed lock.

[0137] When the lock's effective scope is limited to multiple clients, the lock value is converted into a lease lock, and its key name follows the atomic instruction required by the API, meaning the lock value's key name has business meaning. When the lock's effective scope also needs to be effective across concurrent read / write operations or across multiple API instructions in terms of time dimension, the lock value does not contain business meaning. Instead, a separate lease lock is added to carry the business meaning, provided the lock value is effective.

[0138] When Redis storage is not used, an interface is added to the coordination center to implement read and write operations on the lock queue through a linked list in the virtual machine memory (JVM memory) of the single master node of the coordination center. Specifically, it uses the timestamp, lock queue, and lock value of the data structure within the single JVM of the Nacos master node; the atomicity of the timestamp is ensured by the singleness of the Nacos master node; the lock queue uses a lock table for data storage; and the lock value uses the JVM's locking mechanism (synchronized) to ensure the exclusivity of the read and write processes.

[0139] Preferably, the proxy bridging system for the time-series queue further includes:

[0140] The distributed lock verification unit is used to determine the storage offset of the write message through the consumer in the business agent terminal and before submitting the storage offset to the side vehicle, after confirming that the lease of the distributed lock is valid, execute the lease of the distributed lock, lock the action of submitting the storage offset through the distributed lock, and ensure that the action of submitting the offset holds the lease.

[0141] The distributed lock verification unit is specifically used for:

[0142] Based on the initial intermediate timestamp, check whether the initial intermediate timestamp exists in the lock queue and the lock value respectively;

[0143] If the initial intermediate timestamp is not present in both the lock queue and the lock value, the lease of the distributed lock expires, and the sidecar does not perform the action of submitting the storage offset.

[0144] If the initial intermediate timestamp exists in both the lock queue and the lock value, then the lease of the distributed lock is valid.

[0145] Only locks / leases with expiration times are valid. The expiration time is set to prevent deadlocks from being released. Whether a lock can be successfully acquired depends on the verification of the head of the queue in the ZSET.

[0146] Preferably, the target message broker bridge node is specifically used for:

[0147] The message content and the message ID are received through the message writing interface;

[0148] The message is written to the rewrite log of the target message broker bridge node. Within the message log of the rewrite log, the message ID and message content are sequentially and intermittently written to the last line of the rewrite log based on the message ID, and the line number of the last line is obtained. Simultaneously, the message filename is determined. The message filename refers to the segment number in the partition directory where the message is located. The segment number refers to the sequence number assigned to each segment when the message is divided into segments with a fixed number of message lines in the partition directory.

[0149] Within the write index log of the rewrite log, a write index log filename for the write message is constructed, wherein the write index log filename includes the message filename and the storage offset.

[0150] By using the redo log of the message broker bridge node to store the write messages of the time-series queue, high availability, reliability and consistency are achieved, thus avoiding the use of Redis for storage. Redis is only used for related subsequent operations based on the redo log. In other words, high availability, reliability and consistency do not depend on the performance of Redis.

[0151] Preferably, the proxy bridging system for the time-series queue further includes:

[0152] The disk is used to write the rewrite log to this disk, and the queue name of the time-series queue is used as the first-level storage directory, and the partition number corresponding to the partition directory is used as the second-level storage directory.

[0153] Because the rewrite log of the message broker bridge node is statically stored, the rewrite log itself is only temporary storage and will not exist forever, so it cannot be used for persistent storage. Therefore, it needs to be written to disk for persistent storage.

[0154] Preferably, the proxy bridging system for the time-series queue further includes a coordination center node, specifically used for:

[0155] In the sharding mechanism, one node is periodically selected from multiple message broker bridge nodes as the sharding master node, and the other nodes are designated as slave nodes. The sharding master node and the slave nodes are then broadcast to the sidecar and the configuration of each node for routing purposes. Here, the target message broker bridge node refers to the sharding master node.

[0156] As a slave message broker bridge node, when sending the write message to the target message broker bridge node through the message write interface and writing the write message to the rewrite log of the target message broker bridge node, the target broker node simultaneously sends the write message write request to at least one slave node, requesting that the write message be written to at least one slave node; when at least one slave node returns storage success information, the write message storage success information is returned to the sidecar.

[0157] By employing a method of simultaneous writing to both master and slave nodes, write messages are backed up to prevent data loss.

[0158] Preferably, the proxy bridging system for the time-series queue further includes:

[0159] The sidecar 212 is used to determine, within the coordination center node, whether a consumer has permission to retrieve a read message for each read message in the time-series queue, based on the consumer's lease. The consumer is located within the business agent. The coordination center node uses a scheduled task to assign different consumers to different partitions of a time-series queue to ensure that each partition of the time-series queue is distributed as evenly as possible among effective consumers, thereby improving consumer efficiency.

[0160] A consumer with permission to access the read messages is used to read the corresponding message content from the storage space of the existing product. When the storage space of the existing product is a Redis storage space, the consumer with permission to access the read messages obtains the storage offset of the read messages, sends the storage offset of the read messages to the target proxy bridge node and the slave node, sends the storage offset of the read messages to the Redis storage space through the message middleware bridging interface of the target proxy bridge node, and receives the message content returned by the Redis storage space based on the storage offset of the read messages.

[0161] Preferably, the storage space of the existing product also includes at least one storage space parallel to the Redis storage space;

[0162] The target message broker bridge node in message broker bridge node 22 is also used to send the rewritten log to any one or more spaces of the existing Redis storage space or other storage spaces for storage as needed through the message middleware bridging interface of the target broker bridge node.

[0163] Other storage spaces are extensions of Redis storage spaces, enabling existing platforms to operate independently of Redis storage spaces.

[0164] Preferably, the proxy bridging system for the time-series queue further includes a protocol setting unit, used for:

[0165] The interface call protocol used between servers is set to an interface call protocol other than the Redis protocol; the server includes at least: an application server, a message broker bridge node, a coordination center node, a Redis storage space, and a storage space parallel to the Redis storage space;

[0166] The Redis protocol-compatible calls are set up to be made between processes within the server, and the calls do not go through the network card, so that the outside cannot access the internal server.

[0167] This embodiment does not expose the Redis port externally. All Redis protocol communication is achieved through inter-process communication within the server's internal UDS (Unix Domain Socket), without going through the network. All commands that may be sent to Redis are audited by the proxy bridge to avoid remote code vulnerabilities.

[0168] In conjunction with embodiments of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing one or more programs, which, when executed by a computer device, cause the computer device to perform the aforementioned proxy bridging method for a time-series queue.

[0169] like Figure 3 As shown, in conjunction with embodiments of the present invention, a computer device is provided, comprising:

[0170] A processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the aforementioned proxy bridging method for a time-series queue.

[0171] The technical solutions of the present invention will be described in detail below with reference to specific application examples. For technical details not described in the implementation process, please refer to the relevant descriptions above.

[0172] like Figure 4 The diagram shows the complete logical structure of the time-series queue proxy bridging system used in the implementation of the time-series queue proxy bridging method according to an embodiment of the present invention. It includes an application server, a Digestion Coordinator Cluster (DCC) server (a cluster), and a message broker server. The application server includes a business agent (a Java application) and a sidecar. The DCC server includes at least one DCC node. The message broker server includes multiple message broker bridge nodes and storage space for existing products (also called message broker nodes). The storage space for existing products includes Redis storage space and other parallel storage spaces, such as Kafka, RockMQ, Cassandra, Doris BE, etc.

[0173] 1. Set the interface call protocol used between servers to an interface call protocol other than the Redis protocol; the server includes at least: an application server, a message broker bridge node, a coordination center node, a Redis storage space and a storage space parallel to the Redis storage space; set the Redis-compatible call to be made between internal processes of the server, and the call does not go through the network card so that the outside cannot access the internal server.

[0174] 2. The Coordination Center (DCC) node is used to manage and coordinate the cluster of message broker bridge nodes. It verifies consumer permissions, and if the permission verification fails, the consumer will not be able to find the message broker bridge node to access. The Coordination Center (DCC) node can help with routing and forwarding, and also has embedded permission management capabilities.

[0175] Message broker bridge nodes (including physical and logical server nodes) exist as portals for accessing the interfaces exposed by nodes in the storage pathway.

[0176] In the sharding mechanism of the coordination center node, one node is periodically selected from multiple message broker bridge nodes as the sharding master node, and the other nodes are designated as slave nodes. The sharding master node and the slave nodes are then broadcast to the sidecar and each node's respective configuration for routing purposes. The target message broker bridge node refers to the sharding master node.

[0177] The Coordination Center (DCC) node acts as the first-level distributed lock, using a lease mechanism to locate the dynamic routing rules of the message broker bridge node.

[0178] Third, the message broker bridge node is deployed in a distributed cluster. Multiple unified interfaces are abstracted on the message broker bridge node, treating it as a microservice for edge vehicles to call. The main function is to abstract and standardize the methods to be executed by the instructions sent by the edge vehicles, thereby enabling the rewriting of logs and message middleware drivers through interface implementation. The message broker bridge node then achieves reliability, distributed transaction consistency, and request uniqueness through a redo log mechanism.

[0179] IV. The steps to compose a message are as follows:

[0180] 1. For each write message in the time-series queue, the producer in the business agent parses the write message to obtain the message content, and writes the message content to the sidecar through the "write message" operation;

[0181] The message content is sent to the target message proxy bridge node by calling the message writing interface of the message proxy bridge node through the side vehicle.

[0182] Based on the message timestamp generated by the target message broker bridge node according to the message content, the storage offset of the written message is determined by the consumer in the business agent end. The storage offset of the written message is submitted to the message writing interface of the target message broker bridge node by submitting the storage offset operation to record the rewrite log. The new parameters for writing the message include the message content and the storage offset.

[0183] The role of the sidecar is to facilitate inter-process communication within the server without exposing the Redis port externally. Furthermore, the interface call protocol for external access has been changed, and access to Redis is only possible between the proxy bridge and the Redis process.

[0184] 2. Based on the storage offset and the message timestamp, generate a message ID using the sidecar;

[0185] Based on the message content and the message ID, a call instruction (including the time-series queue name, message key name, and message body / content) is generated by the sidecar. The call instruction then invokes the message writing interface of the target message broker bridge node to record rewrite logs. This ensures the atomicity of the message.

[0186] 3. Send the message content and message ID to the target message broker bridge node through the message writing interface; write the message to the rewrite log of the target message broker bridge node (i.e., record the rewrite log), wherein, in the message log of the rewrite log, based on the message ID, the message ID and the message content are sequentially written to the last line of the rewrite log at intervals to obtain the line number of the last line; that is, each message is written sequentially as one line, and the message ID (the message ID is a parameter passed by the message writing interface) and the message content are written respectively, separated by three "@" characters. At the same time, the message file name is determined, which refers to the segment number under the partition directory where the message is located. The segment number refers to the sequence number set for the segment when the message is segmented with a fixed number of message lines under the partition directory; each segment is an independent file. And in the write index log of the rewrite log, construct the write index log file name of the message, which includes the message file name and the storage offset. The storage file where the storage offset is located has the storage file name format seek.idx. The storage file containing the segment number has the filename format segment.idx.

[0187] When the write message is sent to the target message broker bridge node through the message write interface and written to the rewrite log of the target message broker bridge node, the target broker node simultaneously sends a write message request to at least one slave node, requesting that the write message be written to at least one slave node; when at least one slave node returns a storage success message, it returns a write message storage success message to the sidecar, i.e., a high-availability write confirmation, thereby ensuring that both the sharded master node and slave nodes are written and backed up, avoiding data loss.

[0188] The log rewriting mechanism relies on mmap to ensure a disk flush is performed with each write operation. If a timeout occurs, the write is considered a failure, and the line is removed from the mmap. mmap (memory map) maps a file or other object into memory. Files are mapped to multiple pages, and mmap establishes a mapping from virtual memory to disk space. It maps a virtual memory address to a disk file. If this address is not set, the system automatically sets it. The function returns the corresponding memory address (virtual address). When this address is accessed, the contents of the disk need to be copied into memory for reading or writing. Finally, mmap swaps the data back to disk, effectively unmapping the virtual memory from the disk. This is one method for reading and writing disk files, and also a method for processes to share data and memory. The operating system schedules disk writes in the background at appropriate times, or disk writes can be actively performed by calling system functions.

[0189] Virtual memory pages are a mechanism for memory management in Linux. Virtual memory space can be considered as being mapped to disk space, with the mapping location recorded in a page table. When an address is accessed, the valid bit in the page table indicates whether the data is in memory. If not, a page fault occurs, and the corresponding data is copied from disk to memory. If there is no free memory, the page is sacrificed to replace another page.

[0190] 4. Write the rewrite log to the disk to realize the storage of the rewrite log. In the disk, the queue name of the time-series queue is used as the first-level storage directory, and the partition number corresponding to the partition directory is used as the second-level storage directory.

[0191] Because the message broker bridge node's redo log is statically stored, it is only temporary and cannot be used for persistent storage. The broker bridge uses a scheduled task cleanup mechanism to remove all data with segment numbers less than the read segment number from memory. Before being removed, this data can be persisted to disk according to the configuration.

[0192] In addition, Redolog is not designed to have the random retrieval capability for querying specific information like relational databases; it only performs sequential read and write operations. Storage components generally need to provide this capability in order to locate the message body, i.e., the message content, by consuming the offset.

[0193] 5. The rewrite logs are sent to the storage space of the existing Redis product, or any one or more other storage spaces, through the message middleware bridging interface of the target proxy bridge node as required for storage.

[0194] V. The steps for reading messages are as follows:

[0195] For each read message in the time-series queue, based on the consumer's lease, the sidecar determines within the coordination center node whether the consumer has the permission to obtain the read message, wherein the consumer is located within the business agent.

[0196] Consumers with permission to read the messages read from the existing product's storage space. When the existing product's storage space is a Redis storage space, the consumer with permission to read the messages obtains the storage offset of the messages, sends the storage offset of the messages to the target proxy bridge node and slave node through the message read interface (carrying the queue name, consumer name, and offset), sends the consumption offset of the messages to the Redis storage space through the message middleware bridging interface of the target proxy bridge node, and receives the message content returned by the Redis storage space based on the consumption offset of the messages.

[0197] Specifically, a permission management layer is set up at the Coordination Center (DCC) node. When a consumer starts, it needs to send "verify lease and obtain offset" to the sidecar. After being integrated by the message filter, it sends "lease scheduling and obtain offset / session number" to the Coordination Center (DCC) node. Within the Coordination Center (DCC) node, in the "Permission Management and Visibility of Proxy Bridge Service" service, if a consumer does not have permission to access the time-series queue, it returns a lease acquisition failure. That is, for reading messages, the consumer has no right to read them; otherwise, the lease acquisition is successful. Then, in the "Consumer Self-Balancing and Message Broker Type Management Scheduled Task" service, it is determined that the consumer can obtain the offset / session number. The reason is that the Coordination Center (DCC) node has a health check mechanism. In addition to the standard client liveness detection (actively probing whether consumers are alive) configured in the Coordination Center (DCC) server center, consumers of the time-series queue service are registered through the consumer registration interface to maintain a consumer list for different queues. When registering, consumers carry the queue name and consumer name and register with the existing message broker bridge node and middleware. The Coordination Center (DCC) node internally uses scheduled tasks to assign different consumers to different partitions of a time-series queue, ensuring that the distribution within each partition of the time-series queue is as evenly distributed as possible among the effective consumers. If the consumer is a signed consumer, the offset / session ID can be obtained. Specifically, the message broker bridge node can be accessed, which sends a message read request. After being processed by the message broker bridge node, the message read interface of the existing product's storage space is called. Depending on the storage space used, the consumption offset or session ID is returned. For Redis storage space, the consumption offset is returned; for Kafka, since it manages the offset itself, only the consumer group and consumer name are required to automatically return messages, and the consumer group will be returned as the session ID. Finally, the corresponding message content is obtained.

[0198] Specifically, for the obtained consumption offset, the storage file will be named reading.idx, and its content will be the storage offset submitted when writing the message.

[0199] For reading messages, the on-demand call when it is determined that the offset needs to be obtained means that the interface for obtaining the consumption offset (which carries the queue name and consumer name when called) is an on-demand interface. Whether it is called depends on whether it is a required parameter when reading messages (required for Redis storage, not required for Kafka). It needs to call the storage to obtain the current consumption progress.

[0200] The acknowledgment interface is used to record the completion of message consumption when a consumer successfully executes the relevant business logic based on the read message request and the retrieved message. The information carried in the acknowledgment includes the queue name, consumer name, offset, and commit offset or ACK. Different storage spaces have different definitions of acknowledgment, such as commit offset (Redis), commit ACK (RabbitMQ), and commit (Kafka time-series queue).

[0201] Additionally, during the establishment of the message queue connection pool when a consumer starts up, the registration client (used to manage consumer information) and marking client (used to filter invalid consumer information (only retaining active connections)) operations of the message queue connection pool are intercepted by the sidecar, transformed into service registration, and sent to the Coordination Center (DCC) server. The registration client and marking client interfaces are mechanisms used to test the liveness of consumers (actively probing whether consumers are alive) when using Redis storage.

[0202] VI. In the event of any of the following situations: system crash, data center power outage, hardware failure, server migration, or cloud server drift, the message broker bridge node needs to be restarted. After the message broker bridge node restarts, the processing progress related to storage is read from the rewrite log, and the storage locations that have not been written are determined. The write progress is located from segment.idx and seek.idx; the lines not written to storage are those following seek.idx. The message ID is obtained from reading.idx (this is used to cache in memory to handle invalid offset conversions in read requests), thus continuing the parameter values ​​in the methods calling storage / middleware. The lines not written to storage, i.e., those following seek.idx, refer to: seek.idx is updated after the broker bridge successfully writes to storage; it records the write log offset, so messages after this offset are not written.

[0203] The beneficial technical effects of the embodiments of the present invention are as follows:

[0204] 1. For existing systems, from the system operation and maintenance and platform level, Redis can be replaced with Kafka, Doris, etc., but at the application and product level, it remains almost unchanged, achieving minimal system adjustments. If modifications are made due to changes in architecture (replacing Redis), the one-time investment in application and product level would be calculated in "hundreds of person-years". Therefore, it avoids the workload and potential risks brought about by reconstruction.

[0205] Based on the sharding and routing of sidecar and coordination center (DCC) nodes, a unified interface is used on the storage side (message broker bridge node) to adapt to different existing storage spaces (message middleware) (Kafka, RocketMQ) or distributed storage (Cassandra, Doris BE). In existing technologies, message middleware is based on a proxy-registration center mechanism to write messages to message middleware nodes (Broker). However, in this embodiment of the invention, the message middleware only serves as storage, but high availability, reliability, and consistency do not depend on the middleware used as storage.

[0206] 2. By rewriting the redo log, atomic transactions in the time-series queue are saved before being written to storage for recovery in case of failure. This no longer relies on the distributed mechanism of the storage itself, so the choice of storage is not limited to message middleware and is superior to the reliability mechanism of most message middleware itself.

[0207] 3. The message broker bridge node adopts an abstract unified interface, which allows it to connect to self-designed database tables, ETL, etc. when choosing distributed storage, thereby connecting real-time (asynchronous) system processes and data warehouses, shortening the original links of multiple subsystems of "message middleware-ETL or stream-data warehouse storage" (although the internal transmission link is slightly longer, considering the platformization, the performance loss is negligible compared to the cross-subsystem ingestion).

[0208] 4. It interfaces with various mainstream storage systems, improving the versatility of time-series queues.

[0209] 5. Multi-tenant access control has been implemented, thereby extending the application scenarios from the application level to the platform level.

[0210] 6. Since the connected storage includes mainstream distributed databases, it can further support bypassing the connection between real-time data and data warehouses.

[0211] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to the specific order or hierarchy described.

[0212] In the above detailed description, various features are combined together in a single embodiment to simplify this disclosure. This approach to disclosure should not be construed as reflecting an intention that embodiments of the claimed subject matter require more features than are explicitly stated in each claim. Rather, as reflected in the appended claims, the invention is presented with fewer features than all of the features of the single disclosed embodiment. Therefore, the appended claims are hereby explicitly incorporated into the detailed description, wherein each claim stands alone as a preferred embodiment of the invention.

[0213] The disclosed embodiments have been described above to enable any person skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit and scope of this disclosure. Therefore, this disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0214] The foregoing description includes examples of one or more embodiments. It is certainly impossible to describe all possible combinations of components or methods in order to describe the above embodiments, but those skilled in the art will recognize that further combinations and arrangements of the various embodiments are possible. Therefore, the embodiments described herein are intended to cover all such changes, modifications, and variations that fall within the scope of the appended claims. Furthermore, the term "comprising" as used in the specification or claims is interpreted in a manner similar to the term "including," as interpreted when used as a conjunction in the claims. Additionally, the use of any term "or" in the specification of the claims is intended to mean "non-exclusive or."

[0215] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A proxy bridging method for a time-series queue, characterized in that, Implemented based on application servers and message broker bridge nodes, the system interfaces with the storage space of existing products, including Redis storage. The application server includes a business proxy and a sidecar. The proxy bridging method for the time-series queue includes: The service agent receives a time-series queue, and for each write message in the time-series queue, it transforms the write message to form new parameters for the write message. The sidecar generates a call instruction based on the new parameters for writing messages, and calls the message writing interface of the message broker bridge node according to the call instruction, and uses the called message broker bridge node as the target message broker bridge node. The write message is sent to the target message broker bridge node through the message write interface, and the write message is written to the rewrite log of the target message broker bridge node; The rewrite log is asynchronously sent to the storage space of the existing product for subsequent operations based on the rewrite log.

2. The proxy bridging method for time-series queues according to claim 1, characterized in that, The step of transforming each write message in the time-series queue to form new parameters for the write message includes: For each write message in the time-series queue, the producer in the business agent parses the write message to obtain the message content, and writes the message content to the sidecar. The message content is sent to the target message proxy bridge node by calling the message writing interface of the message proxy bridge node through the side vehicle. Based on the message timestamp generated by the target message broker bridge node according to the message content, the storage offset of the written message is determined by the consumer in the business broker end, wherein the new parameters of the written message include the message content and the storage offset.

3. The proxy bridging method for time-series queues according to claim 2, characterized in that, The step of generating a call instruction based on new parameters for writing messages using the sidecar, and then calling the message writing interface of the message broker bridge node according to the call instruction, includes: Based on the storage offset and the message timestamp, a message ID is generated through the sidecar; Based on the message content and the message ID, a call instruction is generated by the sidecar, and the message writing interface of the target message broker bridge node is called according to the call instruction.

4. The proxy bridging method for time-series queues according to claim 3, characterized in that, The step of sending the write message to the target message broker bridge node through the message write interface and writing the write message to the rewrite log of the target message broker bridge node includes: The message content and the message ID are sent to the target message broker bridge node through the message writing interface; The message is written to the rewrite log of the target message broker bridge node. Within the message log of the rewrite log, the message ID and message content are sequentially written to the last line of the rewrite log at intervals, based on the message ID, and the line number of the last line is obtained. Simultaneously, the message filename is determined. The message filename refers to the segment number in the partition directory where the message is located. The segment number refers to the sequence number assigned to each segment when the message is divided into segments with a fixed number of message lines in the partition directory. Within the write index log of the rewrite log, a write index log filename for the write message is constructed, wherein the write index log filename includes the message filename and the storage offset.

5. The proxy bridging method for time-series queues according to claim 4, characterized in that, Also includes: The rewrite log is written to the disk, where the queue name of the time-series queue is used as the first-level storage directory, and the partition number corresponding to the partition directory is used as the second-level storage directory.

6. The proxy bridging method for time-series queues according to claim 1, characterized in that, Also includes: In the sharding mechanism of the coordination center node, one node is periodically selected from multiple message broker bridge nodes as the sharding master node, and the other nodes are designated as slave nodes. The sharding master node and the slave nodes are then broadcast to the sidecar and each node's respective configuration for routing purposes. The target message broker bridge node refers to the sharding master node. The proxy bridging method for the time-series queue further includes: When the write message is sent to the target message broker bridge node through the message write interface and the write message is written into the rewrite log of the target message broker bridge node, the target broker node simultaneously sends the write message write request to at least one slave node, requesting that the write message be written to at least one slave node. Once at least one slave node returns a successful storage message, the write message is returned to the sidecar with a successful storage message.

7. The proxy bridging method for time-series queues according to claim 1, characterized in that, Also includes: For each read message in the time-series queue, based on the consumer's lease, the sidecar determines within the coordination center node whether the consumer has the permission to obtain the read message, wherein the consumer is located within the business agent. Consumers with permission to read the messages read from the existing product's storage space. When the existing product's storage space is a Redis storage space, the consumer with permission to read the messages obtains the storage offset of the messages, sends the storage offset of the messages to the target proxy bridge node and the slave node, sends the storage offset of the messages to the Redis storage space through the message middleware bridging interface of the target proxy bridge node, and receives the message content returned by the Redis storage space based on the storage offset of the messages.

8. The proxy bridging method for time-series queues according to claim 6, characterized in that, The existing product's storage space also includes at least one storage space that runs parallel to the Redis storage space; Sending the rewrite log to the storage space of the existing product includes, The rewrite logs are sent to the storage space of the existing Redis product, or any one or more other storage spaces, through the message middleware bridging interface of the target proxy bridge node, as needed.

9. The proxy bridging method for time-series queues according to claim 6, characterized in that, Also includes: Set the interface call protocol used between servers to an interface call protocol other than the Redis protocol; The server includes at least: an application server, a message broker bridge node, a coordination center node, a Redis storage space, and a storage space running parallel to the Redis storage space; The Redis protocol-compatible calls are set up to be made between processes within the server, and the calls do not go through the network card, so that the outside cannot access the internal server.

10. A proxy bridging system for time-series queues, characterized in that, This includes an application server and a message broker bridge node, which interface with the storage space of existing products, including Redis storage. The application server includes a business proxy and a sidecar, wherein: The business agent is used to receive the time-series queue, and for each write message in the time-series queue, to transform the write message into new parameters for the write message; The sidecar is used to generate a call instruction based on the new parameters for writing messages, call the message writing interface of the message broker bridge node according to the call instruction, and take the called message broker bridge node as the target message broker bridge node. The target message broker bridge node in the message broker bridge node is used to receive the write message through the message write interface, write the write message into the rewrite log of the target message broker bridge node; and asynchronously send the rewrite log to the storage space of the existing product for related subsequent operations based on the rewrite log.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which, when executed by a computer device, cause the computer device to perform a proxy bridging method for a time-series queue as described in any one of claims 1-9.

12. A computer device, characterized in that, include: processor; And a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform a proxy bridging method for a timing queue as described in any one of claims 1-9.

Images