Anti-loopback bidirectional message synchronization method and system between message queue clusters

Abstract

The invention relates to the technical field of distributed data processing, and discloses an anti-loopback bidirectional message synchronization method and system between message queue clusters, and the method comprises the steps that a message producer client determines a current cluster identity identifier according to network connection configuration, and the message producer client sets a source cluster identifier attribute in a message object; the synchronization component pulls a message object from the first message queue cluster, and adds a synchronization source mark attribute before forwarding the message object to the second message queue cluster; the synchronization component runs the reverse synchronization task, and if it is detected that the message object contains the synchronization source mark attribute, the message object is discarded; and the message consumer client compares the source cluster identifier attribute with the local cluster identifier, and sends consumption confirmation and discards the message object if the source cluster identifier attribute is inconsistent with the local cluster identifier. According to the invention, the self-contained routing judgment is realized through the message object metadata without depending on an external global state database, so that the architecture complexity is reduced, and the throughput performance of a synchronous link is improved.

Description

Technical Field

[0001] This invention relates to the field of distributed data processing technology, specifically to a method and system for preventing loopbacks in bidirectional message synchronization between message queue clusters. Background Technology

[0002] In large-scale distributed systems, in order to ensure high availability and disaster recovery capabilities of data, multiple independent message queue clusters need to be deployed in physical resource pools in different geographical locations. The first message queue cluster in the first physical resource pool and the second message queue cluster in the second physical resource pool need to perform bidirectional message synchronization to ensure that business data remains consistent across different regions.

[0003] When a bidirectional synchronization channel is established between the first and second message queue clusters, message objects synchronized from the first to the second message queue cluster are easily retrieved again by the second message queue cluster's synchronization mechanism and sent back to the first message queue cluster, creating a message routing deadlock. Existing anti-loop technologies rely on maintaining an external global state database in the synchronization link to record message transmission trajectories, or on complex distributed lock mechanisms to coordinate the synchronization state. The external global state database introduces additional architectural complexity and maintenance costs, and in high-concurrency scenarios, frequent access to the external global state database increases network latency, limiting the throughput performance of the message synchronization link.

[0004] Existing message production and transmission mechanisms lack the ability to solidify physical ownership information at the source of message object generation. During cross-network transmission or storage, message objects lack self-contained initial origin credentials, making it difficult for subsequent processing steps to determine the original source of the message object based on its own data content. When network topology changes or configurations drift, message objects lacking solidified credentials can cause the anti-loopage judgment logic to fail.

[0005] In scenarios where full data backup is implemented, both the first and second message queue clusters contain all message data through synchronization. If the message consumer client cannot distinguish whether a message object is a locally generated native message or a copy message synchronized from another location, the message consumer client will perform business logic processing on all message objects. The message consumer client's repeated business operations on message objects synchronized from another location will cause resource waste, lead to abnormal business status, and ultimately disrupt the eventual consistency of business data between the first and second message queue clusters. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and system for preventing loopbacks in bidirectional message synchronization between message queue clusters. It solves the problems of existing technologies relying on external global state databases, which leads to complex architecture and limited throughput performance of synchronization links; message objects lacking fixed initial origin credentials, making it difficult to determine their original source; and message consumer clients being unable to distinguish between synchronized messages from different locations, resulting in duplicate consumption and thus disrupting data consistency.

[0007] To achieve the above objectives, the first aspect of the present invention provides a method for preventing loopback and synchronizing bidirectional messages between message queue clusters, comprising the following steps: S100: During the generation phase, the message producer client sends a message object to the first message queue cluster. The message producer client sets the source cluster identifier attribute in the message object through a customized software development kit. The attribute value of the source cluster identifier attribute is set to the unique identifier of the first message queue cluster. The message object is stored in the message topic of the first message queue cluster. S200: During the synchronization phase, the synchronization component initiates a message synchronization task from the first message queue cluster to the second message queue cluster. The synchronization component pulls message objects from the first message queue cluster. Before forwarding the message objects to the second message queue cluster, the synchronization component performs attribute enhancement operations on the message objects. The attribute enhancement operations include reading the attribute value of the source cluster identifier attribute in the message object and adding a synchronization source marker attribute to the message object. The synchronization component sets the attribute value of the synchronization source marker attribute to be the same as the attribute value of the source cluster identifier attribute. The synchronization component writes the message objects that have undergone attribute enhancement operations to the second message queue cluster. S300: During the loopback detection phase, the synchronization component simultaneously runs a reverse synchronization task from the second message queue cluster to the first message queue cluster. The reverse synchronization task pulls message objects from the second message queue cluster. The reverse synchronization task executes loopback detection logic on the pulled message objects. The loopback detection logic checks whether the message object carries a synchronization source marker attribute. If the message object already contains the synchronization source marker attribute, the reverse synchronization task executes the abort synchronization operation and discards the message object.

[0008] During the generation phase, the customized software development kit (SDK) reads the client configuration information during the message producer client startup initialization phase. The SSD extracts the name server address parameter from the client configuration information. The name server address parameter represents the target service endpoint for the message producer client to establish a Transmission Control Protocol (TCP) connection. The SSD loads a pre-set cluster address mapping table and performs matching calculations between the name server address parameter and the cluster address mapping table, outputting the current cluster identity identifier.

[0009] The customized software development kit responds to the message sending command from the message producer client, generates an instance of the message object, registers the source cluster identifier field in the user-defined attribute dictionary contained in the message object, obtains the current cluster identity identifier, and writes the current cluster identity identifier into the entry in the user-defined attribute dictionary with the key name "source cluster identifier attribute".

[0010] The method for preventing loopbacks and bidirectional message synchronization between message queue clusters also includes local environment determination steps and same-origin consumption strategy steps executed by the message consumer client: The message consumer client reads the configuration information of the currently active network connection, extracts the target server access point address from the configuration information, matches the target server access point address with the pre-set cluster address list, and determines the local cluster identifier based on the matching result.

[0011] The message consumer client subscribes to and pulls message objects from either the first or second message queue cluster. The message consumer client reads the attribute value of the source cluster identifier attribute in the message object. The message consumer client compares the attribute value of the source cluster identifier attribute with the local cluster identifier. If the attribute value of the source cluster identifier attribute matches the local cluster identifier, the message consumer client triggers the business logic to consume the message object. If the attribute value of the source cluster identifier attribute does not match the local cluster identifier, the message consumer client discards the message object locally.

[0012] A second aspect of the present invention provides a loop-prevention bidirectional message synchronization system between message queue clusters, comprising a first message queue cluster, a second message queue cluster, a message producer client, a message consumer client, and a synchronization component, wherein: The message producer client integrates a message production module. The message production module is used to: read the network connection configuration of the message producer client to determine the physical cluster identity of the current client when generating a message object, initialize the user attribute set in the data structure of the message object, and write the physical cluster identity into the attribute item with the key name source cluster identity attribute.

[0013] The synchronization component integrates a message synchronization module, which is specifically used for: Maintain the consumption point and continuously pull message objects to be synchronized from the source message queue cluster; scan the pulled message objects to check if there is an attribute item with the key name "synchronization source flag" in the user attribute set. If it exists, issue a discard command; otherwise, issue a release command; respond to the release command, extract the attribute value with the key name "source cluster identifier" from the message object, construct an attribute with the key name "synchronization source flag" in the message object, and assign the extracted attribute value of the source cluster identifier to the attribute with the key name "synchronization source flag"; encapsulate the message object after attribute enhancement and send it to the destination message queue cluster.

[0014] The message consumer client integrates a message consumption module, which is specifically used for: After retrieving the message object, the user attribute set is parsed and the attribute value with the key name "source cluster identifier" is extracted. By analyzing the current network connection context of the message consumer client, the identity identifier of the local cluster is calculated and obtained. The attribute value of the extracted source cluster identifier is compared with the identity identifier of the local cluster. If the attribute value of the source cluster identifier matches the identity identifier of the local cluster, the business logic is allowed to consume the message object. If the attribute value of the source cluster identifier does not match the identity identifier of the local cluster, the message object is intercepted and discarded locally.

[0015] This invention provides a method and system for preventing loopback and synchronizing bidirectional messages between message queue clusters. It has the following beneficial effects: 1. This invention performs attribute enhancement operations on message objects before forwarding them to the second message queue cluster through a synchronization component. The attribute enhancement operation reads the source cluster identifier attribute and generates a synchronization source marker attribute. The reverse synchronization task identifies and discards loopback messages by detecting whether the message object contains the synchronization source marker attribute. The anti-loopback bidirectional message synchronization method between message queue clusters uses the metadata carried inside the message object for self-contained routing determination, without relying on an external global state database. This reduces the architectural complexity of the anti-loopback bidirectional message synchronization system between message queue clusters and improves the throughput performance of the message synchronization link.

[0016] 2. This invention utilizes a customized software development kit integrated into the message producer client to perform environment identification and attribute injection during the message generation stage. The customized software development kit determines the current cluster identity based on the network connection configuration and writes the current cluster identity into the source cluster identity attribute of the message object. The message producer client solidifies the physical ownership information of the message object at the source, ensuring that the message object can still retain the initial place of origin certificate after cross-network transmission or storage, providing a data foundation for subsequent anti-loop judgment logic and same-source consumption strategy.

[0017] 3. This invention executes a same-origin consumption strategy through a message consumer client. The message consumer client compares the source cluster identifier attribute in the message object with the local cluster identifier. When the source cluster identifier attribute is inconsistent with the local cluster identifier, the message object is discarded locally and a consumption confirmation data packet is sent to the first message queue cluster or the second message queue cluster. The same-origin consumption strategy achieves logical isolation at the application layer, enabling the first message queue cluster and the second message queue cluster to maintain full data backup, while preventing message objects synchronized from different locations from triggering duplicate business logic processing, thus ensuring the eventual consistency of business data between the first message queue cluster and the second message queue cluster. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a system architecture block diagram of the present invention. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See attached document Figure 2 This invention provides a loop-prevention bidirectional message synchronization system between message queue clusters, including a first message queue cluster, a second message queue cluster, a message producer client, a message consumer client, and a synchronization component; The message producer client, synchronization component, and message consumer client execute this invention collaboratively through their internally integrated message production module, message synchronization module, and message consumption module, respectively.

[0021] The first message queue cluster is deployed in the first physical resource pool, and the second message queue cluster is deployed in the second physical resource pool. The first message queue cluster and the second message queue cluster are physically isolated from each other, and the first message queue cluster and the second message queue cluster communicate with each other through the network.

[0022] The first and second physical resource pools are geographically separated internet data centers, each with its own independent power supply and network switching facilities, ensuring physical independence between them. A dedicated wide area network (WAN) provides cross-regional Transmission Control Protocol (TCP) data transmission channels, configured with Secure Sockets Layer (SSL) encryption to guarantee data confidentiality during cross-regional message transmission. Both the first and second message queue clusters consist of multiple message broker server nodes. These nodes employ a master-slave replication architecture for high availability and are responsible for persistent storage and forwarding of message data.

[0023] The customized software development kit (SDK) registers a pre-send interceptor in the message producer client's runtime environment. This pre-send interceptor is a hook function implementation within the aspect-oriented programming (AOP) model. When the message producer client calls the message sending interface, the pre-send interceptor pauses the main message sending process, obtains a reference to the current message object, calls the environment identification unit to retrieve the current cluster identity, and then calls the attribute injection unit to inject the current cluster identity into the message object's user attribute set. The main message sending process only resumes after the pre-send interceptor has completed its execution.

[0024] Before being transmitted over the network, message objects need to be processed by a serialization encoder, which converts the message object into a sequence of binary bytes. The serialization encoder follows a predefined message transmission protocol format, which defines the magic number, version number, message length, header length, header data, and the order of the message body data within the binary byte sequence. The source cluster identifier and synchronization source flag attributes are stored as key-value pairs in the header data area. The message broker server node only needs to parse the header data to read the source cluster identifier and synchronization source flag attributes; the message broker server node does not need to decode the message body data, thus reducing the computational load on the message broker server node's CPU.

[0025] Internally, each message broker server node runs a name service registration client. This client periodically sends heartbeat packets to the name server cluster. These heartbeat packets contain the message broker server node's internet address, port number, and a list of message topics carried by the node. The name server cluster maintains a global routing table that records the mapping between message topics and message broker server nodes. Message producer and consumer clients retrieve metadata information for either the first or second message queue cluster by querying the name server cluster.

[0026] The message broker server node employs sequential disk write technology to write message objects to the physical hard drive. Sequential disk write reduces disk head seek time, thus improving input / output performance. The message broker server node utilizes memory-mapped file technology to map disk files to virtual memory space. Memory-mapped file technology allows the operating system to directly manipulate file data in memory, avoiding the overhead of frequent data copying between user-mode and kernel-mode memory spaces. The message broker server node is configured with either asynchronous or synchronous flushing strategies. The asynchronous flushing strategy accumulates a certain amount of message data in the memory buffer before performing the disk write operation, while the synchronous flushing strategy returns an acknowledgment signal after each message object is written to disk.

[0027] The message producer client integrates a customized software development kit (SDK). The message producer client generates message objects and sends them to either a first or second message queue cluster. When generating a message object, the SSD obtains the identity identifier of the message queue cluster the client is currently connected to. The SSD writes this identity identifier into the message object's user attribute set. For message objects, The message object contains a collection of user attributes. The attribute key name is used to represent the source identifier of the message. For the identity string of the currently connected message queue cluster, the attribute injection logic executed by the message producer client satisfies the following relationship: ; in: This indicates that the key name in the user attribute collection is... The corresponding attribute values, in this embodiment, Defined as the string constant SEND_AZ, when a message producer client connects to the first message queue cluster, The value is assigned to R1 when the message producer client connects to the second message queue cluster. It is assigned the value R2.

[0028] The synchronization component is deployed between the first message queue cluster and the second message queue cluster. It performs bidirectional message synchronization tasks between these two clusters. The component includes a message retrieval unit, a message filtering unit, a message attribute enhancement unit, and a message forwarding unit. The message retrieval unit retrieves message objects from the source message queue cluster. The message filtering unit performs loopback detection on the retrieved message objects. The attribute key name is used to represent the synchronization source tag. This is a collection of all attribute key names in the user attribute collection of the message object. For the operation instructions executed by the synchronization component, the anti-loop judgment logic executed by the message filtering unit satisfies the following formula: ; Wherein: DISCARD indicates discarding the message object and terminating subsequent processing of the message object, and PROCESS indicates allowing the message object to enter the subsequent processing stage. In this embodiment, It is defined as the string constant ORIGIN_SEND_UNIT.

[0029] When the operation instruction is PROCESS, the message attribute enhancement unit performs attribute enhancement operations on the message object, and the message attribute enhancement unit reads the key name in the message object. The original attribute values, the message attribute enhancement unit adds a key named in the user attribute collection. The attribute item is assigned the original attribute value, and the attribute item is assigned the original attribute value. The key name in the message object The corresponding attribute values ​​and attribute enhancement operations satisfy the following logic: ; ; The message forwarding unit sends the message object, after attribute enhancement, to the destination message queue cluster.

[0030] The message consumer client integrates a customized software development kit. It subscribes to and pulls message objects from either the first or second message queue cluster. The client reads the message objects from the clusters whose keys are... The message consumer client compares the attribute value with the identity identifier of the message queue cluster it is currently connected to. If the attribute value matches the identity identifier, the message consumer client triggers business logic to consume the message object. If the attribute value does not match the identity identifier, the message consumer client discards the message object locally.

[0031] A message object consists of message body data and a set of user attributes. The set of user attributes is stored using a key-value pair data structure. The set of user attributes includes a source cluster identifier attribute and a synchronization source marker attribute. The key name of the source cluster identifier attribute is configured as SEND_AZ. The source cluster identifier attribute is used to record the name of the message queue cluster to which the message object belongs at the time of initial creation. The key name of the synchronization source marker attribute is configured as ORIGIN_SEND_UNIT. The synchronization source marker attribute is used to mark the original source cluster name of the message object during cross-cluster synchronization.

[0032] The message body data is a binary byte stream, carrying specific business load information. The user attribute set is stored using a hash table data structure, where both keys and values ​​are string types. The hash table supports dynamic expansion to accommodate other business attributes besides the source cluster identifier and synchronization source flag attributes. The user attribute set maintains data integrity during message object serialization and deserialization. As metadata, the user attribute set is transmitted over the network along with the message body data. The message queue cluster's server-side storage engine supports independent indexing of the user attribute set to facilitate attribute-based message filtering.

[0033] The synchronization component acts as middleware connecting the first message queue cluster and the second message queue cluster. The synchronization component is internally configured with a message converter. The message converter is used to modify the user attribute set during the process of message objects flowing from the source message queue cluster to the destination message queue cluster. The message converter has attribute extraction and attribute injection functions. The attribute extraction function is used to read the attribute value of the source cluster identifier attribute, and the attribute injection function is used to generate the synchronization source marker attribute based on the attribute value of the source cluster identifier attribute.

[0034] See attached document Figure 1 This invention provides a method for preventing loopback and synchronizing bidirectional messages between message queue clusters, comprising the following steps: S100: During the generation phase, the message producer client sends a message object to the first message queue cluster. The message producer client sets the source cluster identifier attribute in the message object using a customized software development kit. The attribute value of the source cluster identifier attribute is set to the unique identifier of the first message queue cluster. The message object is stored in the message topic of the first message queue cluster.

[0035] Customized software development kits (SDKs) embed the message sending chain of the message producer client using aspect-oriented programming (AOP) or interceptor design patterns. The SSD takes over the message sending process without modifying the business layer code, thus achieving non-intrusive integration with existing business systems. Before sending a message object, the SSD serializes the message object, encoding the source cluster identifier attribute as an extended field into the serialized binary data packet, ensuring that the source cluster identifier attribute is transmitted to the message broker server node along with the message body data.

[0036] S200: During the synchronization phase, the synchronization component initiates a message synchronization task from the first message queue cluster to the second message queue cluster. The synchronization component pulls message objects from the first message queue cluster. Before forwarding the message objects to the second message queue cluster, the synchronization component performs attribute enhancement operations on the message objects. The attribute enhancement operations include reading the attribute value of the source cluster identifier attribute in the message object and adding a synchronization source marker attribute to the message object. The synchronization component sets the attribute value of the synchronization source marker attribute to be the same as the attribute value of the source cluster identifier attribute. The synchronization component writes the message objects that have undergone attribute enhancement operations to the second message queue cluster.

[0037] S300: During the loopback detection phase, the synchronization component simultaneously runs a reverse synchronization task from the second message queue cluster to the first message queue cluster. The reverse synchronization task pulls message objects from the second message queue cluster and executes loopback detection logic on the pulled message objects. The loopback detection logic determines whether a message object is a synchronized message by checking whether the message object carries a synchronization source marker attribute.

[0038] set up This represents the message object to be processed. This indicates the key name of the synchronization source tag attribute. Represents the collection of properties of a message object, and defines functions. Used to detect whether a key name exists in a collection of attributes, defined For the execution decision of the reverse synchronization task, the loop closure detection logic satisfies the following formula: ; Where: ABORT means to abort the synchronization operation and discard the message object, and TRANSMIT means to continue the message forwarding operation.

[0039] Since the message objects synchronized from the first message queue cluster to the second message queue cluster already contain the synchronization source marker attribute, when the reverse synchronization task processes the message objects... If the conditions are met, the reverse synchronization task executes the ABORT decision. The reverse synchronization task intercepts the message object on the second message queue cluster side. The message object will not be sent back to the first message queue cluster. The data consistency between the first message queue cluster and the second message queue cluster is maintained through a unidirectional flow path, and the message routing dead loop link is physically cut off.

[0040] Step S101 is performed by a custom software development kit integrated within the message producer client. Step S101 aims to determine the current physical network environment of the message producer client by detecting network connection configuration.

[0041] During the initialization phase of the message producer client, the customized software development kit reads the client configuration information. The customized software development kit extracts the name server address parameter from the client configuration information. The name server address parameter represents the target server endpoint for the message producer client to establish a Transmission Control Protocol connection. The customized software development kit loads a pre-set cluster address mapping table, which contains the address set of the first message queue cluster and the address set of the second message queue cluster.

[0042] The customized software development kit matches and calculates the name server address parameters against the cluster address mapping table to output the current cluster identity. This serves as the current cluster identity identifier. For the name server address parameter, This is the set of pre-defined addresses for the first message queue cluster. Given the pre-defined address set for the second message queue cluster, the matching calculation logic executed by the customized software development kit satisfies the following formula: ; Where R1 represents the unique character code of the first message queue cluster, and R2 represents the unique character code of the second message queue cluster.

[0043] The customized software development kit stores the calculated current cluster identity in the local memory space of the message producer client. The current cluster identity is used as a global constant for subsequent message attribute construction steps. The customized software development kit ensures that every message object sent to the message queue cluster can be associated with the correct physical cluster affiliation during the lifecycle of the message producer client.

[0044] Step S102 is performed by a custom software development kit integrated within the message producer client. Step S102 aims to pre-define an attribute container in the message object's data structure to store origin information.

[0045] The customized software development kit responds to the message sending command from the message producer client. The customized software development kit generates an instance of a message object. The message object instance contains a message body payload and a user-defined attribute dictionary. The user-defined attribute dictionary adopts a key-value pair mapping structure. The customized software development kit registers the source cluster identifier field in the user-defined attribute dictionary.

[0046] The customized software development kit establishes the source cluster identifier field as one of the standard metadata of the message object instance through the above operations. After processing in step S102, the message object instance has the data structure foundation for storing physical cluster ownership information. The customized software development kit ensures that all message object instances sent through the message producer client contain the source cluster identifier field.

[0047] Step S103 is executed by a customized software development kit integrated within the message producer client. Step S103 assigns differentiated values ​​to the source cluster identifier attribute based on the physical network environment in which the message producer client is located.

[0048] The customized software development kit (SDK) obtains the current cluster identity determined in step S101, locates the user-defined attribute dictionary inside the message object, and determines the specific string value to be written into the user-defined attribute dictionary based on the target cluster object currently connected to by the message producer client. If the connection is to the first cluster, the value is assigned to R1; if the connection is to the second cluster, the value is assigned to R2.

[0049] When the message producer client establishes a connection with the first message queue cluster, the custom software development kit writes the string R1 into the entry with the key name SEND_AZ in the user-defined attribute dictionary. When the message producer client establishes a connection with the second message queue cluster, the custom software development kit writes the string R2 into the entry with the key name SEND_AZ in the user-defined attribute dictionary. After completing step S103, the message object carries an immutable initial place of origin certificate.

[0050] The message production module is integrated into the customized software development kit (SDK) of the message producer client. It executes steps S101 to S103 above, ensuring that the sent messages carry the correct initial source information. The message production module specifically includes an environment identification unit, an attribute injection unit, and an attribute injection unit, wherein: The environment identification unit is used to perform the above-mentioned S101 step. The environment identification unit reads the network connection configuration of the message producer client and determines the physical cluster identity of the current client by matching the name server address with a preset cluster address mapping table.

[0051] The attribute injection unit performs steps S102 and S103 as described above. In response to the message sending request, the attribute injection unit initializes the user attribute set in the message object's data structure and writes the identity identifier output by the environment identification unit into the attribute field with the key name SEND_AZ.

[0052] See attached document Figure 1In step S200, the message retrieval operation is performed first. The synchronization component continuously listens to the message topics of the source message queue cluster. The synchronization component retrieves the message objects to be synchronized from the source message queue cluster. The message objects contain a set of user attributes.

[0053] Step S200 then performs loopback detection and filtering operations. The synchronization component reads the user attribute set of the message object and checks whether the synchronization source marker attribute exists in the user attribute set. For the processing decisions of the synchronization component, The key name for the attribute of the synchronization source. Given a list of keys for the user attribute set in the message object, the detection logic executed by the synchronous component satisfies the following formula: ; in: This indicates that the message object should be discarded and the synchronization process should be terminated. This indicates that the message object is allowed to proceed to subsequent processing steps. In this embodiment, the key name of the synchronization source tag attribute is used. It is defined as the string constant ORIGIN_SEND_UNIT.

[0054] When the decision is When the synchronization component recognizes the message object as a loopback message, it releases the message object from memory, and the message object will not be written to the destination message queue cluster.

[0055] When the decision is At that time, the synchronization component performs attribute enhancement operations. The synchronization component locates the entry in the user attribute set whose key is "Source Cluster Identifier," extracts the attribute value of the "Source Cluster Identifier," creates a new "Synchronization Source Marker" attribute in the user attribute set, and assigns the extracted attribute value to the "Synchronization Source Marker" attribute. For the collection of user attributes of the message object, The key name for the source cluster identifier attribute. For the value passed to the synchronization source tag attribute, the attribute enhancement operation satisfies the following formula: ; ; in: The corresponding string constant is SENDAZ.

[0056] The final step in step S200 is message forwarding. The synchronization component sends the message object, which has undergone attribute enhancement, to the destination message queue cluster. The destination message queue cluster receives and stores the message object, which carries both the source cluster identifier attribute and the synchronization source marker attribute.

[0057] Step S201 is performed by a synchronization component deployed in a heterogeneous network environment. Step S201 ensures the continuous acquisition of newly generated message data from the source message queue cluster.

[0058] The synchronization component initializes a network communication session with the source message queue cluster. The synchronization component registers a dedicated synchronization consumer group in the source message queue cluster. The dedicated synchronization consumer group subscribes to the message topics to be synchronized. The synchronization component initiates a batch pull request to the source message queue cluster based on the consumption position recorded locally.

[0059] set up This is a list of message objects returned by a single fetch operation. For the target message subject, For the current consumption location, Define a function to set the maximum number of messages to fetch in a single session. For the message fetch operation function, the message fetch logic executed by the synchronous component satisfies the following formula: ; in: It contains zero or more message objects.

[0060] The synchronization component receives a list of message objects and temporarily stores it in its internal memory buffer. After confirming that the list of message objects has been successfully received, the synchronization component updates the value of the consumption point recorded locally. The synchronization component achieves real-time tracking of the message stream in the source message queue cluster by repeatedly executing the message retrieval step from the source cluster.

[0061] The message retrieval unit employs a multi-threaded concurrent retrieval mode. It launches an independent retrieval thread for each physical partition of the source message queue cluster to maximize network bandwidth utilization. The unit maintains a consumption point for each message queue partition, which is persistently stored in an external relational database or local file system to prevent data loss or duplicate synchronization due to restarting the synchronization component. The synchronization component maintains sessions with both the first and second message queue clusters via long-lived connections. It utilizes a batch processing mechanism to retrieve multiple message objects at once, reducing network interactions and increasing message synchronization throughput.

[0062] The synchronization component internally maintains a thread pool resource, which contains multiple concurrently executing worker threads. The synchronization component dynamically adjusts the number of worker threads based on the number of message topic partitions in the source message queue cluster. Each worker thread is bound to one or more message topic partitions, and is responsible for message retrieval, message filtering, and message forwarding tasks for the bound message topic partition. The execution states of worker threads do not interfere with each other.

[0063] The synchronization component is configured with a consumer position manager, which is responsible for recording the synchronization progress of each message topic partition. The consumer position manager periodically writes the synchronization progress to an external configuration center or a local persistent file. When the synchronization component fails and restarts, it reads the synchronization progress from the external configuration center or local persistent file and resumes the message fetching task based on the read progress, thus avoiding duplicate or missed message transmissions. The synchronization component also features a graceful shutdown mechanism; when it receives a stop command, it waits for the currently processed message batch to complete before closing the network connection and saving the final consumer position.

[0064] Step S202 is executed immediately by the synchronization component after obtaining the message object. Step S202 aims to block illegal routing loops by identifying characteristic attributes in the message object.

[0065] The synchronization component reads the user attribute set from the message object, iterates through the user attribute set to obtain all attribute key names, checks whether the attribute key names contain the predefined synchronization source marker attribute key name, and determines the subsequent processing flow of the message object based on the detection result.

[0066] set up For the operation instructions executed by the synchronization component on the message object, A collection of user attribute key names carried by the message object. For the synchronization source marker attribute key name used to detect loopbacks, the detection and judgment logic executed by the synchronization component satisfies the following formula: ; Wherein: DISCARD indicates that the message object should be discarded immediately and the current synchronization process of the message object should be terminated; NEXT indicates that the message object should be allowed to pass the detection and enter the attribute enhancement step. In this embodiment, the synchronization source is marked with the attribute key name. It is set to the string constant ORIGIN_SEND_UNIT.

[0067] When the synchronization component detects a set of user attribute keys Includes When the synchronization component determines that the message object is a loopback message that has undergone at least one cross-cluster synchronization, the synchronization component executes the DISCARD instruction, destroys the message object in memory, and the message object is not written to the destination message queue cluster. When the synchronization component detects the user attribute key name set... Not included When the synchronization component determines that the message object is a native message generated by the source cluster, the synchronization component executes the NEXT instruction and passes the message object to the subsequent attribute enhancement module for processing.

[0068] When executing detection and judgment logic, the message filtering unit only reads the header metadata of the message object. It does not need to deserialize or parse the specific content of the message body data. Therefore, the message filtering unit supports business data of any format, including text, binary, or encrypted data. When the operation command is DISCARD, the synchronization component records detailed discard logs for auditing by operations and maintenance personnel. The discard logs include the message's unique identifier, the source cluster identifier, and the discard timestamp. The synchronization component then updates the consumption point to skip the current message object, ensuring continuous processing of subsequent message streams.

[0069] Step S203 is executed by the synchronization component after confirming that the message object has passed the loopback detection. Step S203 is designed to add a route tracing tag to the message object for subsequent loopback identification.

[0070] The synchronization component locates the set of user attributes within the message object, retrieves the source cluster identifier attribute from the set of user attributes, and stores the initial origin information of the message object. The synchronization component extracts the attribute value corresponding to the source cluster identifier attribute.

[0071] The synchronization component constructs a synchronization source marker attribute in the user attribute set, and assigns the attribute value of the extracted source cluster identifier attribute to the synchronization source marker attribute.

[0072] After the synchronization component completes the attribute conversion operation, the message object carries both the source cluster identifier attribute and the synchronization source marker attribute. The synchronization component then passes the attribute-enhanced message object to the message forwarding module. The message forwarding module is responsible for sending the message object to the destination message queue cluster. When the destination message queue cluster receives the message object, the message object already has the feature data required to trigger the next loopback detection.

[0073] Step S204 is performed by the synchronization component. Step S204 aims to persistently store the attribute-enhanced message object to the destination message queue cluster. The synchronization component connects to the destination message queue cluster using a pre-established network channel. The synchronization component encapsulates the message object processed in step S203 into a data sending request. At this point, the message object already contains the source cluster identifier attribute and the synchronization source marker attribute.

[0074] The synchronous component calls the message sending interface to transmit the data sending request to the destination message queue cluster. The result of the write operation. For the destination message queue cluster, For a message object that carries a synchronization source flag attribute, For the message delivery function, the message writing operation performed by the synchronous component satisfies the following formula: ; in: This indicates whether the message object has been successfully written to the destination message queue cluster. The synchronous component receives the execution result. When the execution result When the write operation is successful, the synchronization component updates the consumption progress cursor of the source message queue cluster. When a write operation fails, the synchronization component executes retry sending logic on the message object, and the destination message queue cluster receives the message object and stores it in persistent storage.

[0075] The message forwarding unit adopts an asynchronous sending mode. When writing a message object to the destination message queue cluster, the unit can process the next message object without waiting for a write acknowledgment signal, thereby reducing synchronization latency. The message forwarding unit has a built-in fault tolerance mechanism. When it receives a write failure signal from the destination message queue cluster, it executes an exponential backoff retry strategy. This strategy gradually increases the retry interval to avoid network congestion, ensuring that the message object is eventually successfully written to the destination message queue cluster, thus guaranteeing data transmission reliability.

[0076] The message forwarding unit maintains a dead-letter queue mechanism. When the number of retries for a message object reaches a preset threshold and still fails to write it to the destination message queue cluster, the message forwarding unit delivers this message object to the local dead-letter queue. The local private message queue is used to isolate unprocessable abnormal messages, preventing them from blocking the message sending channel. Operations personnel can manually intervene or redeliver abnormal messages by querying the local private message queue.

[0077] The synchronization component integrates a performance indicator (PI) monitoring probe, which collects real-time operational status data. This data includes synchronization latency, message processing per second, network bandwidth utilization, heap memory usage, and garbage collection frequency. The PI monitoring probe pushes this data to a unified monitoring platform, which then provides a visual representation of the data. The unified monitoring platform is configured with alarm rules; when the synchronization latency exceeds an alarm threshold, it notifies operations personnel via SMS or email. The synchronization component not only enables bidirectional message data flow but also provides comprehensive observability support, ensuring the stability of the cross-regional message synchronization system.

[0078] The message synchronization module corresponds to the synchronization components deployed between clusters, used to execute steps S201 to S204, and is the core of implementing loopback prevention. The message synchronization module specifically includes a message retrieval unit, a loopback detection and filtering unit, an attribute enhancement unit, and a message forwarding unit, wherein: The message retrieval unit is used to execute step S201. The message retrieval unit maintains the consumption position and continuously retrieves message objects to be synchronized from the source message queue cluster through the network connection.

[0079] The loopback detection and filtering unit performs step S202 above. It scans the retrieved message objects and checks if a synchronization source marker attribute with the key name ORIGIN_SEND_UNIT exists in the user attribute set. If it exists, the loopback detection and filtering unit issues a discard command; otherwise, it issues a allow command.

[0080] The attribute enhancement unit is used to execute step S203. In response to the release command, the attribute enhancement unit extracts the attribute value of SEND_AZ from the message object, constructs the ORIGIN_SEND_UNIT attribute in the message object, and assigns the extracted value to the ORIGIN_SEND_UNIT attribute.

[0081] The message forwarding unit is used to perform the above S204 step. The message forwarding unit encapsulates the message object after attribute enhancement processing and sends it to the destination message queue cluster.

[0082] Step S301 is executed by the message consumer client. Step S301 aims to obtain the initial physical location information pre-defined in the message object.

[0083] The message consumer client pulls message objects from the message queue cluster via a network connection. The message consumer client locates the user attribute mapping table stored in the message object and uses a pre-defined source cluster identifier key to access the user attribute mapping table.

[0084] Step S302 is executed by the message consumer client. Step S302 aims to determine the physical cluster environment in which the message consumer client is currently located by using the network connection context.

[0085] The message consumer client reads the configuration information of the currently active network connection, extracts the target server access point address from the configuration information, matches the target server access point address with the pre-set cluster address list, and determines the specific value of the local cluster identifier based on the matching result. If the address matches the first cluster address, the local identifier is R1; otherwise, it is R2.

[0086] The message consumer client stores the calculated local cluster identifier in the runtime memory of the message consumer client. The local cluster identifier serves as the baseline data for determining whether the message source is locally generated in subsequent business logic.

[0087] The S303 step is executed immediately by the message consumer client after it retrieves the message object. The S303 step is designed to enforce a strict same-origin consumption strategy by comparing location information.

[0088] The message consumer client obtains the source cluster identifier attribute value extracted in step S301 and the local cluster identifier determined in step S302. The message consumer client performs a string equality comparison between the source cluster identifier attribute value and the local cluster identifier. If the source cluster identifier attribute value matches the local cluster identifier, the message consumer client triggers the business callback function to consume the message object normally. If the source cluster identifier attribute value does not match the local cluster identifier, the message consumer client determines that the message object is remotely synchronized data and does not meet the same-source consumption condition. The message consumer client directly discards the message object locally and skips processing.

[0089] The message consumer client supports dynamically refreshing the cluster address list at runtime to adapt to network topology changes caused by message queue cluster expansion or migration, ensuring that the local environment determination unit always performs calculations based on the latest network configuration. The message consumer client's local message object discarding operation means that the client sends a consumption success acknowledgment signal to the message queue cluster but does not call the business layer processing function, thus logically filtering out remotely synchronized messages and preventing remote data from polluting the local database or triggering duplicate business operations. This process is completely transparent to the upper-layer business logic; the business logic code does not need to be aware of the specific implementation details of message filtering.

[0090] When a message consumer client performs a local discard operation, it sends a consumption acknowledgment packet to the message broker server node in the message queue cluster. This acknowledgment packet informs the message broker server node that the current message object has been successfully processed. Upon receiving the acknowledgment packet, the message broker server node advances the consumption progress pointer maintained on the server side. If the message consumer client does not send a acknowledgment packet, the message broker server node will assume the current message object has not been consumed and will re-deliver it to the message consumer client after a timeout, resulting in unnecessary network traffic consumption. Therefore, sending a consumption acknowledgment packet by the message consumer client is a crucial step in implementing logical filtering without affecting the server's state.

[0091] The custom software development kit (SDK) implements a consumer filter adapter within the message consumer client. This adapter contains user-written business logic listeners. When the message consumer client receives a message object from the network layer, the consumer filter adapter takes over control. It executes attribute comparison logic. Only if the attribute comparison logic succeeds will the consumer filter adapter invoke the message receiving function of the business logic listener, passing the message object to the business logic code. If the attribute comparison logic fails, the consumer filter adapter directly returns a success status, and the business logic listener code is not executed. The consumer filter adapter pattern completely decouples message filtering logic from business logic, improving code maintainability and scalability.

[0092] The message consumption module is integrated into the customized software development kit (SDK) of the message consumer client. It executes steps S301 to S303 to implement a strict same-origin consumption strategy. The message consumption module specifically includes an attribute parsing unit, a local environment determination unit, and a consumption filtering unit.

[0093] The attribute parsing unit is used to execute step S301. After the consumer retrieves the message object, the attribute parsing unit parses the user attribute set and extracts the attribute value of SEND_AZ.

[0094] The local environment determination unit is used to execute step S302. The local environment determination unit calculates and obtains the identity of the local cluster in real time by analyzing the current network connection context of the consumer client.

[0095] The consumption filtering unit is used to execute step S303. The consumption filtering unit compares the attribute value extracted by the attribute parsing unit with the local identifier obtained by the local environment determination unit. If the attribute value and the local identifier are consistent, the business logic is allowed to consume the message object; if the attribute value and the local identifier are inconsistent, the message object is intercepted and discarded locally.

Claims

1. A method for preventing loopback and synchronizing bidirectional messages between message queue clusters, characterized in that, Includes the following steps: S100: The message producer client sends a message object to the first message queue cluster. The message producer client sets a source cluster identifier attribute in the message object through a customized software development kit. The attribute value of the source cluster identifier attribute is set to a unique identifier of the first message queue cluster. The message object is stored in the message topic of the first message queue cluster. S200: The synchronization component initiates a message synchronization task from the first message queue cluster to the second message queue cluster. The synchronization component pulls the message object from the first message queue cluster. Before forwarding the message object to the second message queue cluster, the synchronization component performs an attribute enhancement operation on the message object. The attribute enhancement operation includes reading the attribute value of the source cluster identifier attribute in the message object, adding a synchronization source marker attribute to the message object, setting the attribute value of the synchronization source marker attribute to be the same as the attribute value of the source cluster identifier attribute, and then writing the message object to the second message queue cluster. S300: The synchronization component runs a reverse synchronization task from the second message queue cluster to the first message queue cluster. The reverse synchronization task pulls the message object from the second message queue cluster and executes loopback detection logic. If the message object already contains the synchronization source marker attribute, the reverse synchronization task stops the synchronization operation and discards the message object.

2. The method for preventing loopback and bidirectional message synchronization between message queue clusters according to claim 1, characterized in that, In step S100, the customized software development kit (SDK) reads client configuration information during the message producer client startup initialization phase. The SSD extracts name server address parameters from the client configuration information. The name server address parameters represent the target service endpoint for the message producer client to establish a Transmission Control Protocol (TCP) connection. The SSD loads a pre-set cluster address mapping table. The SSD matches and calculates the name server address parameters with the cluster address mapping table to output the current cluster identity.

3. The method for preventing loopback and bidirectional message synchronization between message queue clusters according to claim 2, characterized in that, In step S100, the customized software development kit responds to the message sending instruction from the message producer client, the customized software development kit generates an instance of the message object, and the customized software development kit registers a source cluster identifier field in the user-defined attribute dictionary contained in the message object; The customized software development kit obtains the current cluster identity identifier and writes the current cluster identity identifier into the entry with the key name SEND_AZ in the user-defined attribute dictionary.

4. The method for preventing loopback and bidirectional message synchronization between message queue clusters according to claim 1, characterized in that, In step S200, the synchronization component initializes a network communication session with the first message queue cluster, the synchronization component registers a dedicated synchronization consumer group in the first message queue cluster, the dedicated synchronization consumer group subscribes to the message topics to be synchronized, and the synchronization component initiates a batch pull request to the first message queue cluster based on the locally recorded consumption position. The synchronization component receives a list of message objects, temporarily stores the list of message objects in its internal memory buffer, and updates the value of the consumption position recorded locally after confirming that the list of message objects has been successfully received.

5. The method for preventing loopback and bidirectional message synchronization between message queue clusters according to claim 1, characterized in that, In step S200, after the synchronization component retrieves the message object and before performing the attribute enhancement operation, it executes detection and judgment logic: The synchronization component reads the user attribute set in the message object, the synchronization component traverses the user attribute set to obtain all attribute key names, and the synchronization component detects whether the attribute key names contain a predefined synchronization source marker attribute key name; When the synchronization component detects that the user attribute set contains the synchronization source marker attribute key name, the synchronization component determines that the message object is a loopback message, and the synchronization component destroys the message object in memory; When the synchronization component detects that the user attribute set does not contain the synchronization source marker attribute key name, the synchronization component determines that the message object is a native message, and the synchronization component allows the message object to enter the attribute enhancement operation.

6. The method for preventing loopback and bidirectional message synchronization between message queue clusters according to claim 1, characterized in that, In step S200, the synchronization component connects to the second message queue cluster using a pre-established network channel. The synchronization component encapsulates the message object that has undergone the attribute enhancement operation into a data sending request. The synchronization component calls the message sending interface to transmit the data sending request to the second message queue cluster. The synchronization component receives the execution result. When the execution result indicates that the write was successful, the synchronization component updates the consumption progress cursor of the first message queue cluster. When the execution result indicates that the write failed, the synchronization component executes retry sending logic for the message object.

7. The method for preventing loopback and bidirectional message synchronization between message queue clusters according to claim 1, characterized in that, The method also includes a local environment determination performed by the message consumer client, specifically including the following steps: The message consumer client reads the configuration information of the currently active network connection, extracts the target server access point address from the configuration information, matches the target server access point address with a pre-set list of cluster addresses, and determines the local cluster identifier based on the matching result.

8. A method for preventing loopback and bidirectional message synchronization between message queue clusters according to claim 7, characterized in that, The method also includes a same-origin consumption strategy executed by the message consumer client, specifically comprising the following steps: The message consumer client subscribes to and pulls the message object from the first message queue cluster or the second message queue cluster. The message consumer client reads the attribute value with the key name SEND_AZ in the message object. The message consumer client compares the attribute value with the local cluster identifier. If the attribute value matches the local cluster identifier, the message consumer client triggers business logic to consume the message object. If the attribute value does not match the local cluster identifier, the message consumer client discards the message object locally.

9. A method for preventing loopback and bidirectional message synchronization between message queue clusters according to claim 8, characterized in that, The message consumer client discards the message object locally, specifically including the following steps: The message consumer client sends a consumption confirmation data packet to the message broker server node of the first message queue cluster or the second message queue cluster. The consumption confirmation data packet informs the message broker server node that the message object has been successfully processed, but the message consumer client does not call the business layer processing function.

10. A bidirectional message synchronization system for preventing loopbacks between message queue clusters, characterized in that, A method for preventing loopback and bidirectional message synchronization between message queue clusters, applicable to any one of claims 1-9, includes a first message queue cluster, a second message queue cluster, a message producer client, a message consumer client, and a synchronization component; The message producer client integrates a message production module, which is used to read the network connection configuration to determine the physical cluster identity of the current client, and write the identity into the attribute item with the key name SEND_AZ in the user attribute set when generating the message object. The synchronization component integrates a message synchronization module. The message synchronization module is used to pull the message object from the source message queue cluster, check whether there is a synchronization source marker attribute with the key name ORIGIN_SEND_UNIT in the user attribute set. If it exists, it is discarded. If it does not exist, the attribute value with the key name SEND_AZ is extracted from the message object, an attribute with the key name ORIGIN_SEND_UNIT is constructed, and the extracted attribute value is assigned to the attribute with the key name ORIGIN_SEND_UNIT. The processed message object is then sent to the destination message queue cluster. The message consumer client integrates a message consumption module. The message consumption module is used to extract the attribute value with the key name SEND_AZ from the message object to obtain the identity identifier of the local cluster. If the attribute value is inconsistent with the identity identifier of the local cluster, the message object is intercepted and discarded locally.

Images