Distributed data change synchronization method and system under unidirectional isolation network

Abstract

The invention particularly relates to a distributed data change synchronization method and system under a one-way isolation network. According to the distributed data change synchronization method and system under the one-way isolation network, a customized Rocket MQ producer SDK and a local standby library are deployed in a node area, and the node area sends a data change message to a non-node area through the one-way isolation network; if the sending fails, storing the message to a local standby library; the node service acquires the Rocket MQ cluster state information regularly, and adjusts a message sending strategy according to the Rocket MQ cluster state information; and if the fault is recovered, firstly taking out the message from the local standby library, and resending the message to the Rocket MQ cluster. According to the distributed data change synchronization method and system under the one-way isolation network, efficient and reliable synchronization of node area data change can be achieved on the premise that network safety isolation is guaranteed, even if a Rocket MQ cluster breaks down, it can be guaranteed that data are not lost through a local standby library mechanism, and message retransmission is automatically completed after the fault is recovered.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a distributed data change synchronization method and system in a unidirectional isolated network. Background Art

[0002] In large-scale distributed systems, data consistency and real-time performance are crucial, especially in scenarios involving sensitive data or high security requirements, where one-way isolation of the network environment has become the norm. However, this network environment poses a huge challenge to data change synchronization, and traditional message queue solutions are difficult to apply directly. Therefore, a solution is needed that can handle Rocket MQ cluster failures in a one-way isolated network environment and ensure the safe and orderly synchronization of data change messages.

[0003] In order to achieve efficient and secure data change synchronization in a complex network environment, the present invention proposes a distributed data change synchronization method and system in a unidirectional isolated network. Summary of the invention

[0004] In order to overcome the defects of the prior art, the present invention provides a simple and efficient distributed data change synchronization method and system in a unidirectional isolated network.

[0005] The present invention is achieved through the following technical solutions:

[0006] A distributed data change synchronization method in a unidirectional isolated network, characterized in that it includes the following steps:

[0007] Step S1: Deploy customized Rocket MQ producer SDK and local backup library in the node area;

[0008] Step S2: The node area sends a data change message to the non-node area through a unidirectional isolation network;

[0009] When the data in the node area changes, the message is sent to the Rocket MQ cluster in the non-node area through the customized Rocket MQ producer SDK;

[0010] If the sending fails, the message is stored in the local backup database and a successful response is returned to the caller;

[0011] Step S3: The node service periodically obtains Rocket MQ cluster status information from the status database of the non-node area, and adjusts the message sending strategy (such as suspending sending, resuming sending, etc.) according to the Rocket MQ cluster status information;

[0012] If the Rocket MQ cluster recovers, it will first retrieve the message from the local standby database and resend the message to the Rocket MQ cluster;

[0013] Step S4: After all local standby database messages are sent, the newly generated data change message is directly sent through the Rocket MQ cluster.

[0014] In step S2, a message priority mechanism is introduced to ensure that high-priority critical business data can be sent and processed first after the Rocket MQ cluster is restored;

[0015] At the same time, the message deduplication logic is implemented to avoid message redundancy caused by repeated sending during Rocket MQ cluster failure;

[0016] Finally, a message confirmation mechanism is introduced to ensure that each message can be safely deleted from the local backup library after it is successfully sent to the Rocket MQ cluster and consumed, avoiding data backlog and wasting storage space.

[0017] In step S2, when sending a message to a Rocket MQ cluster in a non-node area, if a network problem or a Rocket MQ cluster failure causes the message to fail, the customized Rocket MQ producer SDK will automatically capture the exception and retry according to a preset retry interval;

[0018] If the message still cannot be sent successfully after reaching the preset retry threshold, the customized Rocket MQ producer SDK will serialize the message and store it in the local backup library of the node, and immediately return a successful response to the caller to ensure the smooth operation of the business system is not affected.

[0019] In step S3, the customized Rocket MQ producer SDK also maintains a status monitoring thread, and regularly requests the latest status information of the Rocket MQ cluster from the non-node area through the unidirectional isolated network at a custom period (such as every minute).

[0020] A distributed data change synchronization system in a unidirectional isolated network, comprising:

[0021] The node area includes multiple node databases and node services. Each node service is deployed with a customized Rocket MQ producer SDK, which is responsible for processing data change messages.

[0022] The node area sends data change messages to the non-node area through a unidirectional isolated network. If the Rocket MQ cluster fails to send the message, the message is written to the local backup library. The node service adjusts the message sending strategy according to the Rocket MQ cluster status information provided by the non-node area to ensure the safe and orderly synchronization of data change messages.

[0023] One-way isolated network, used to ensure that the node area can access the non-node area, but the non-node area cannot directly access the node area;

[0024] The non-node area includes the Rocket MQ cluster, cluster status monitoring module, and status database;

[0025] Deploy a highly available Rocket MQ cluster in the non-node area to ensure that it can receive and process messages from the node area.

[0026] The Rocket MQ cluster is responsible for receiving and processing messages from the node area, and the cluster status monitoring module is responsible for real-time monitoring of the Rocket MQ cluster status (such as cluster node status, message queue status, etc.), and persisting the status information to the status database of the non-node area;

[0027] The customized Rocket MQ producer SDK integrates the native sending interface of the Rocket MQ cluster, and is pre-set with an intelligent retry mechanism, local standby library writing logic, and a state synchronization callback interface. It is responsible for storing messages in the local standby library when the Rocket MQ cluster fails, and automatically resending messages after the Rocket MQ cluster recovers.

[0028] The local standby database is responsible for storing messages that failed to be sent successfully due to Rocket MQ cluster failures. It supports fast reading and writing, transactional operations, and message deduplication.

[0029] The customized Rocket MQ producer SDK is provided with a message priority mechanism, which is responsible for sending high-priority business data first after the Rocket MQ cluster is restored;

[0030] At the same time, the customized Rocket MQ producer SDK also has built-in intelligent retry mechanism, node backup library writing logic and status synchronization callback interface;

[0031] When sending messages to a Rocket MQ cluster in a non-node zone, if a network problem or a Rocket MQ cluster failure causes the message to fail, the customized Rocket MQ producer SDK will automatically capture the exception and retry according to the preset retry interval;

[0032] If the message still cannot be sent successfully after reaching the preset retry threshold, the customized Rocket MQ producer SDK will serialize the message and store it in the local backup library of the node, and immediately return a successful response to the caller to ensure the smooth operation of the business system is not affected.

[0033] The customized Rocket MQ producer SDK also maintains a status monitoring thread, and the cluster status monitoring module in the non-node area is responsible for providing a status query interface for the customized Rocket MQ producer SDK in the node area to call regularly to obtain the latest status information of the Rocket MQ cluster.

[0034] The local standby library uses a database (such as MySQL, SQLite, etc.) or a high-performance key-value storage system (such as Redis) as the underlying storage medium, and supports fast reading and writing, transactional operations, message deduplication logic and message confirmation mechanism.

[0035] A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the above methods.

[0036] A computing device comprising:

[0037] One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0038] The beneficial effects of the present invention are as follows: the distributed data change synchronization method and system under a one-way isolated network can realize efficient and reliable synchronization of node area data changes under the premise of ensuring network security isolation. Even when the Rocket MQ cluster fails, the local backup library mechanism can ensure that data is not lost and automatically complete the retransmission of messages after the failure is recovered. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Attached Figure 1 It is a schematic diagram of the distributed data change synchronization method in a unidirectional isolation network of the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] The distributed data change synchronization method under the unidirectional isolation network includes the following steps:

[0043] Step S1: Deploy customized Rocket MQ producer SDK and local backup library in the node area;

[0044] Step S2: The node area sends a data change message to the non-node area through a unidirectional isolation network;

[0045] When the data in the node area changes, the message is sent to the Rocket MQ cluster in the non-node area through the customized Rocket MQ producer SDK;

[0046] If the sending fails, the message is stored in the local backup database and a successful response is returned to the caller;

[0047] Step S3: The node service periodically obtains Rocket MQ cluster status information from the status database of the non-node area, and adjusts the message sending strategy (such as suspending sending, resuming sending, etc.) according to the Rocket MQ cluster status information;

[0048] If the Rocket MQ cluster recovers, it will first retrieve the message from the local standby database and resend the message to the Rocket MQ cluster;

[0049] Step S4: After all local standby database messages are sent, the newly generated data change message is directly sent through the Rocket MQ cluster.

[0050] In step S2, a message priority mechanism is introduced to ensure that high-priority critical business data can be sent and processed first after the Rocket MQ cluster is restored;

[0051] At the same time, the message deduplication logic is implemented to avoid message redundancy caused by repeated sending during Rocket MQ cluster failure;

[0052] Finally, a message confirmation mechanism is introduced to ensure that each message can be safely deleted from the local backup library after it is successfully sent to the Rocket MQ cluster and consumed, avoiding data backlog and wasting storage space.

[0053] In step S2, when sending a message to a Rocket MQ cluster in a non-node area, if a network problem or a Rocket MQ cluster failure causes the message to fail, the customized Rocket MQ producer SDK will automatically capture the exception and retry according to a preset retry interval;

[0054] If the message still cannot be sent successfully after reaching the preset retry threshold, the customized Rocket MQ producer SDK will serialize the message and store it in the local backup library of the node, and immediately return a successful response to the caller to ensure the smooth operation of the business system is not affected.

[0055] In step S3, the customized Rocket MQ producer SDK also maintains a status monitoring thread, and regularly requests the latest status information of the Rocket MQ cluster from the non-node area through the unidirectional isolated network at a custom period (such as every minute).

[0056] The distributed data change synchronization system under the unidirectional isolation network includes:

[0057] The node area includes multiple node databases and node services. Each node service is deployed with a customized Rocket MQ producer SDK, which is responsible for processing data change messages.

[0058] The node area sends data change messages to the non-node area through a unidirectional isolated network. If the Rocket MQ cluster fails to send the message, the message is written to the local backup library. The node service adjusts the message sending strategy according to the Rocket MQ cluster status information provided by the non-node area to ensure the safe and orderly synchronization of data change messages.

[0059] One-way isolated network, used to ensure that the node area can access the non-node area, but the non-node area cannot directly access the node area;

[0060] The non-node area includes the Rocket MQ cluster, cluster status monitoring module, and status database;

[0061] Deploy a highly available Rocket MQ cluster in the non-node area to ensure that it can receive and process messages from the node area.

[0062] The Rocket MQ cluster is responsible for receiving and processing messages from the node area, and the cluster status monitoring module is responsible for real-time monitoring of the Rocket MQ cluster status (such as cluster node status, message queue status, etc.), and persisting the status information to the status database of the non-node area;

[0063] The customized Rocket MQ producer SDK integrates the native sending interface of the Rocket MQ cluster, and is pre-set with an intelligent retry mechanism, local standby library writing logic, and a state synchronization callback interface. It is responsible for storing messages in the local standby library when the Rocket MQ cluster fails, and automatically resending messages after the Rocket MQ cluster recovers.

[0064] The local standby database is responsible for storing messages that failed to be sent successfully due to Rocket MQ cluster failures. It supports fast reading and writing, transactional operations, and message deduplication.

[0065] The customized Rocket MQ producer SDK is provided with a message priority mechanism, which is responsible for sending high-priority business data first after the Rocket MQ cluster is restored;

[0066] At the same time, the customized Rocket MQ producer SDK also has built-in intelligent retry mechanism, node backup library writing logic and status synchronization callback interface;

[0067] When sending messages to a Rocket MQ cluster in a non-node zone, if a network problem or a Rocket MQ cluster failure causes the message to fail, the customized Rocket MQ producer SDK will automatically capture the exception and retry according to the preset retry interval;

[0068] If the message still cannot be sent successfully after reaching the preset retry threshold, the customized Rocket MQ producer SDK will serialize the message and store it in the local backup library of the node, and immediately return a successful response to the caller to ensure the smooth operation of the business system is not affected.

[0069] The customized Rocket MQ producer SDK also maintains a status monitoring thread, and the cluster status monitoring module in the non-node area is responsible for providing a status query interface for the customized Rocket MQ producer SDK in the node area to call regularly to obtain the latest status information of the Rocket MQ cluster.

[0070] The local standby library uses a database (such as MySQL, SQLite, etc.) or a high-performance key-value storage system (such as Redis) as the underlying storage medium, and supports fast reading and writing, transactional operations, message deduplication logic and message confirmation mechanism.

[0071] The computer-readable storage medium stores one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device performs any one of the above methods.

[0072] The computing device comprises:

[0073] One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0074] In summary, the distributed data change synchronization method and system in a unidirectional isolated network has the following characteristics:

[0075] 1) High security: Under the restriction of one-way isolation, the system can ensure that data change messages are safely and orderly transmitted from the restricted area (node ​​area) to the Rocket MQ cluster in the unrestricted area (non-node area), effectively ensuring the security of data flow and preventing unauthorized access.

[0076] 2) High reliability: Even if the Rocket MQ cluster is temporarily unavailable, the built-in node backup library mechanism can temporarily store unsuccessfully sent messages and automatically resend them after the cluster is restored, thus ensuring the integrity and real-time nature of data changes.

[0077] 3) Efficiency: Through customized producer SDK and node backup library mechanism, rapid response and synchronization of data changes are achieved, improving the overall efficiency of the system.

[0078] 4) Flexibility: The system architecture is flexible, and the number of nodes and Rocket MQ cluster configuration can be adjusted according to actual needs to adapt to application scenarios of different scales.

[0079] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed data change synchronization method in a unidirectional isolated network, characterized in that: The following steps are involved: Step S1: Deploy customized Rocket MQ producer SDK and local backup library in the node area; Step S2: The node area sends a data change message to the non-node area through a unidirectional isolation network; When the data in the node area changes, the message is sent to the Rocket MQ cluster in the non-node area through the customized Rocket MQ producer SDK; If the sending fails, the message is stored in the local backup database and a successful response is returned to the caller; Step S3: The node service periodically obtains Rocket MQ cluster status information from the status database of the non-node area, and adjusts the message sending strategy according to the Rocket MQ cluster status information; If the RocketMQ cluster recovers, it will first retrieve the message from the local standby database and resend the message to the RocketMQ cluster; Step S4: After all local standby database messages are sent, the newly generated data change message is directly sent through the Rocket MQ cluster.

2. The distributed data change synchronization method in a one-way isolated network according to claim 1 is characterized in that: In step S2, a message priority mechanism is introduced to ensure that high-priority critical business data can be sent and processed first after the Rocket MQ cluster is restored; At the same time, the message deduplication logic is implemented to avoid message redundancy caused by repeated sending during Rocket MQ cluster failure; Finally, a message confirmation mechanism is introduced to ensure that each message can be safely deleted from the local backup library after it is successfully sent to the Rocket MQ cluster and consumed, avoiding data backlog and wasting storage space.

3. The distributed data change synchronization method in a one-way isolated network according to claim 2 is characterized in that: In step S2, when sending a message to a Rocket MQ cluster in a non-node area, if a network problem or a RocketMQ cluster failure causes the message to fail, the customized Rocket MQ producer SDK will automatically capture the exception and retry according to a preset retry interval; If the message still cannot be sent successfully after reaching the preset retry threshold, the customized Rocket MQ producer SDK will serialize the message and store it in the local backup library of the node, and immediately return a successful response to the caller to ensure the smooth operation of the business system is not affected.

4. The distributed data change synchronization method in a one-way isolated network according to claim 1 or 2, characterized in that: In step S3, the customized Rocket MQ producer SDK also maintains a status monitoring thread, and regularly requests the latest status information of the Rocket MQ cluster from the non-node area through the unidirectional isolated network according to a custom period.

5. A distributed data change synchronization system in a unidirectional isolated network, characterized in that: include: The node area includes multiple node databases and node services. Each node service is deployed with a customized Rocket MQ producer SDK, which is responsible for processing data change messages. The node area sends data change messages to the non-node area through a unidirectional isolated network. If the Rocket MQ cluster fails to send the message, the message is written to the local backup library. The node service adjusts the message sending strategy according to the Rocket MQ cluster status information provided by the non-node area. One-way isolated network, used to ensure that the node area can access the non-node area, but the non-node area cannot directly access the node area; The non-node area includes the Rocket MQ cluster, cluster status monitoring module, and status database; The Rocket MQ cluster is responsible for receiving and processing messages from the node area, and the cluster status monitoring module is responsible for real-time monitoring of the Rocket MQ cluster status and persisting the status information to the status database of the non-node area; The customized Rocket MQ producer SDK integrates the native sending interface of the Rocket MQ cluster, and is pre-set with an intelligent retry mechanism, local standby library writing logic, and a state synchronization callback interface. It is responsible for storing messages in the local standby library when the Rocket MQ cluster fails, and automatically resending messages after the Rocket MQ cluster recovers. The local standby database is responsible for storing messages that failed to be sent successfully due to Rocket MQ cluster failures. It supports fast reading and writing, transactional operations, and message deduplication.

6. The distributed data change synchronization system in a one-way isolated network according to claim 5, characterized in that: The customized Rocket MQ producer SDK is provided with a message priority mechanism, which is responsible for sending high-priority business data first after the Rocket MQ cluster is restored; At the same time, the customized Rocket MQ producer SDK also has built-in intelligent retry mechanism, node backup library writing logic and status synchronization callback interface; When sending messages to a Rocket MQ cluster in a non-node zone, if a network problem or a Rocket MQ cluster failure causes the message to fail, the customized Rocket MQ producer SDK will automatically capture the exception and retry according to the preset retry interval; If the message still cannot be sent successfully after reaching the preset retry threshold, the customized Rocket MQ producer SDK will serialize the message and store it in the local backup library of the node, and immediately return a successful response to the caller to ensure the smooth operation of the business system is not affected.

7. The distributed data change synchronization system in a one-way isolated network according to claim 5 or 6, characterized in that: The customized Rocket MQ producer SDK also maintains a status monitoring thread, and the cluster status monitoring module in the non-node area is responsible for providing a status query interface for the customized Rocket MQ producer SDK in the node area to call regularly to obtain the latest status information of the Rocket MQ cluster.

8. The distributed data change synchronization system in a one-way isolated network according to claim 5, characterized in that: The local standby library uses a database or a high-performance key-value storage system as the underlying storage medium, and supports fast reading and writing, transactional operations, message deduplication logic and message confirmation mechanism.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions which, when executed by a computing device, cause the computing device to perform any one of the methods according to claims 1 to 4.

10. A computing device, characterized in that include: One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods according to claims 1 to 4.

Images