Transaction monitoring IDC synchronization method under 5G enhanced call operation management platform disaster recovery architecture

Abstract

The invention discloses a transaction monitoring IDC synchronization method under a 5G enhanced call operation management platform disaster recovery architecture, and the method comprises the steps: building a private line connection between a source data center and a target data center, and guaranteeing the designated theme of each other in a message queue; the source data center triggers data operation and adds corresponding synchronous annotations to form a data operation event, and the data operation event is recorded in a database of the source data center; when the event is successfully submitted, the transaction monitor monitors the data operation event through the synchronous annotation; obtaining an operation object of the data operation event; reversely generating an SQL statement according to the operation pair of the data operation event; the source data center sends an SQL statement and routing information to a message queue; and the target data center consumes from the message queue to obtain the corresponding SQL statement, judges whether the SQL statement is synchronized or not according to the timestamp and the component ID in the SQL statement, and compensates and synchronizes the SQL statement which is not synchronized in the local database to ensure final consistency. According to the transaction monitoring mechanism and the message queue, cross-data center data synchronization is realized through a reverse generation SQL algorithm.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a transaction monitoring IDC synchronization method under a 5G enhanced call operation management platform disaster recovery architecture. Background Art

[0002] In distributed systems, cross-data center (IDC) data synchronization has always been a key issue, especially when dealing with businesses that require high availability and consistency. Traditional cross-data center synchronization methods usually rely on the database's own mechanism or manually use message queues for the database. When the database's own mechanism is perfect, the problem is not very obvious. However, when the database itself does not provide a friendly synchronization solution point, it becomes difficult to use message queues to ensure data consistency and handle distributed transactions. With the popularity of microservice architecture, Spring framework, and message queue middleware, it has become possible to use Spring's transaction monitoring mechanism and aspect-oriented programming to achieve more efficient and reliable cross-data center synchronization. Summary of the invention

[0003] The purpose of the present invention is to provide a transaction monitoring IDC synchronization method under the disaster recovery architecture of a 5G enhanced call operation management platform, so as to solve the problems that manual synchronization cannot well guarantee data consistency, cannot uniformly control transactions, and cannot unify data comparison.

[0004] The technical solution adopted by the present invention is: The transaction monitoring IDC synchronization method under the disaster recovery architecture of the 5G enhanced call operation management platform includes the following steps: Step 1: Establish a dedicated line connection between the source data center and the target data center to at least ensure that they consume the specified topic of the message queue specified by each other; Step 2: The source data center triggers the data operation and adds the corresponding synchronization annotation to form a data operation event, which is recorded in the database of the source data center; Step 3: When the event is successfully submitted, the transaction listener configured on the source data center listens to the data operation event through the synchronization annotation; Step 4, obtaining the operation object of the data operation event; Step 5: Generate SQL statements in reverse order according to the operation pair of the data operation event, save the data operation event in the local database and mark the local database log as unsynchronized; Step 6: The source data center sends the SQL statement and routing information to the message queue, waiting for the target data center to consume it; at the same time, the local database log is modified to be synchronized. Step 7: The target data center consumes the corresponding SQL statement from the message queue, determines whether it is synchronized based on the timestamp and component ID in the SQL statement, and performs compensatory synchronization on the SQL statement that is not synchronized in the local database to ensure eventual consistency.

[0005] Furthermore, the data operation in step 2 includes data update, addition and deletion operations.

[0006] Furthermore, in step 2, the current transaction object is converted into a json string and passed to the transaction listener. Specifically, the current transaction object is converted into a json string and passed to the transaction listener through AOP (Aspect-Oriented Programming).

[0007] Furthermore, in step 7, the target data center uses a program to generate a snowflake algorithm consumption SQL to ensure that the primary key is unique.

[0008] Specifically, because the present invention prohibits reliance on database self-increment and uses a program to generate a snowflake algorithm, it can ensure that the primary key is unique and avoid conflicts.

[0009] Furthermore, in step 7, the SQL statements that are not synchronized with the local database are compensated and synchronized through a scheduled task, and the SQL logs synchronized before the specified date are deleted to avoid excessive storage pressure on the local database.

[0010] Specifically, as a feasible implementation, in step 7, the synchronized SQL logs before yesterday are deleted.

[0011] The present invention adopts the above technical solution, by combining the event-driven mechanism with transaction management, the present invention will eventually be encapsulated into a jar package, providing multiple custom annotations, which can be transplanted everywhere, and a few lines of code can synchronize the data to the message queue in real time and reliably after the data in the source data center is changed. The target data center consumes the message queue and guarantees idempotence and orderliness, and then synchronizes and updates to the target data center database, and only performs synchronization operations after the transaction is successfully submitted to ensure data consistency. At the same time, the present invention also designs a fault recovery mechanism, which automatically retries when synchronization fails, and adds a message queue retry mechanism. Multiple failures will be written to the local log library, and scanning and resending according to the rules or notifying the operation and maintenance personnel to intervene greatly improves the reliability and maintainability of the system, and is suitable for distributed systems with high high availability requirements. The present invention also supports regular comparison of data differences between the data centers on both sides to ensure the final consistency of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments; Figure 1 The schematic diagram of the principle architecture of the transaction monitoring IDC synchronization method under the disaster recovery architecture of the 5G enhanced call operation management platform of the present invention. Implementation

[0013] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0014] like Figure 1 As shown, the present invention discloses a transaction monitoring IDC synchronization method under a 5G enhanced call operation management platform disaster recovery architecture, which includes the following steps: Step 1: Establish a dedicated line connection between the source data center and the target data center to at least ensure that they consume the specified topic of the message queue specified by each other; Step 2: The source data center triggers the data operation and adds the corresponding synchronization annotation to form a data operation event, which is recorded in the database of the source data center; Furthermore, the data operation in step 2 includes data update, addition and deletion operations.

[0015] Furthermore, in step 2, the current transaction object is converted into a json string and passed to the transaction listener. Specifically, the current transaction object is converted into a json string and passed to the transaction listener through AOP (Aspect-Oriented Programming).

[0016] Step 3: When the event is successfully submitted, the transaction listener configured on the source data center listens to the data operation event through the synchronization annotation; Specifically, the source data center triggers data operations to form corresponding transaction objects, adds synchronization annotations to the transaction objects, and automatically triggers transaction listening through synchronization annotations to pass the current transaction object to the transaction listener. Specifically, the transaction listener listens to events such as commit and rollback of the transaction object. The present invention uses the commit event. After the source data center operation transaction is committed, the SQL statements of insert, update and delete are reversely generated through the json string, and the local database writes the synchronization log, the status is marked as synchronized, and the key information is saved.

[0017] Step 4, obtaining the operation object of the data operation event; Step 5: Generate SQL statements in reverse order according to the operation pair of the data operation event, save the data operation event in the local database and mark the local database log as unsynchronized; Step 6: The source data center sends the SQL statement and routing information to the message queue, waiting for the target data center to consume it; at the same time, the local database log is modified to be synchronized. Step 7: The target data center consumes the corresponding SQL statement from the message queue, determines whether it is synchronized based on the timestamp and component ID in the SQL statement, and performs compensatory synchronization on the SQL statement that is not synchronized in the local database to ensure eventual consistency.

[0018] Furthermore, in step 7, the target data center uses a program to generate a snowflake algorithm to consume SQL to ensure that the primary key is unique. Specifically, because the present invention prohibits relying on database auto-increment, using a program to generate a snowflake algorithm can ensure that the primary key is unique and avoid conflicts.

[0019] Furthermore, in step 7, the SQL statements that are not synchronized with the local database are compensated and synchronized through a scheduled task, and the SQL logs synchronized before the specified date are deleted to avoid excessive storage pressure on the local database. Specifically, as a feasible implementation method, in step 7, the synchronized SQL logs before yesterday are deleted.

[0020] The specific principle of the present invention is described in detail below: Taking the 5G enhanced call operation management platform as an example, the platform needs to be deployed in two data centers in different locations. The two data centers are independent of each other and the applications are deployed bilaterally, but the network can be connected.

[0021] 1) Open a dedicated line between the two IDCs to at least ensure that both sides can consume each other's designated message queues and designated topics.

[0022] 2) When the source data center triggers an update, add, or delete operation, the annotation automatically triggers a transaction listener, and converts the current object into a JSON string through the aspect and passes it to the transaction listener.

[0023] 3) The transaction listener can monitor transaction commit, rollback and other events. We use the commit event. After the transaction is committed in the source data center, we reverse the insert, update and delete SQL statements through the JSON string, write the synchronization log to the local database, mark the status as synchronized, and save key information.

[0024] 4) Deliver SQL, timestamp, component ID, etc. to the message queue, modify the local database log to be synchronized, and wait for consumption by the target data center.

[0025] 5) After the target data center consumes SQL, it uses timestamps, component IDs, etc. to determine whether it is synchronized to avoid duplication. Because we prohibit relying on database auto-increment, we use a program-generated snowflake algorithm to ensure that the primary key is unique and avoid conflicts.

[0026] 6) Compare bilateral data regularly through scheduled tasks and ensure eventual consistency through compensation mechanism.

[0027] 7) Use scheduled tasks to compensate for the SQL that has not been synchronized with the local database, and delete the SQL logs that have been synchronized before yesterday to avoid excessive storage pressure on the local database.

[0028] The present invention ensures the final consistency of data: by triggering the cross-data center data synchronization operation after the transaction is committed, the present invention significantly improves data consistency. Only when the source data center transaction is successfully committed, the data will be synchronized to the target data center, avoiding the data inconsistency problem caused by transaction rollback, and ensuring the consistency of data in each data center in the system.

[0029] The present invention improves real-time performance and efficiency: The present invention adopts an event-driven mechanism, which can quickly synchronize data across data centers after data changes, significantly reducing synchronization delays. Events are triggered immediately after a transaction is successfully submitted, which improves the real-time performance of data synchronization and the response efficiency of the system, allowing data to quickly reflect changes between different data centers.

[0030] The present invention enhances system reliability and fault tolerance: The fault recovery mechanism and automatic retry function ensure that the system can automatically handle and recover to normal state in the event of a synchronization operation failure. This mechanism improves the system's fault tolerance and reliability in the face of network problems or target data center failures, reducing the need for manual intervention.

[0031] The present invention supports idempotent operations and avoids data duplication: The idempotent design of the present invention ensures that even if repeated operations occur during the synchronization process, the state of the data remains consistent and does not cause data duplication or errors. This feature increases the stability of the system and the security of data processing.

[0032] The present invention simplifies usage and extension: By customizing the annotation @SyncEventPublisher, the present invention simplifies application coding work, and by adding a line of annotation to the data operation method, it can be ensured that the data is normally synchronized to another data center after being successfully submitted.

[0033] The present invention improves system availability: High consistency, real-time synchronization and reliable fault recovery mechanism improve the overall availability of the system, are suitable for high-demand distributed environments, and ensure that data in different data centers are always synchronized and the system runs stably.

[0034] The present invention adopts the above technical solution, based on the cross-data center data synchronization mechanism of transaction event monitoring, and realizes cross-data center data synchronization by reversely generating SQL algorithms through the ApplicationEvent and @TransactionalEventListener transaction monitoring mechanisms and message queues of the Spring framework. After the data change operation in the source data center successfully submits the transaction, the mechanism automatically publishes and monitors custom events to ensure that the data synchronization operation is only performed when the transaction is successful.

[0035] Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians of the art without making creative work are within the scope of protection of the present application.

Claims

The transaction monitoring IDC synchronization method under the disaster recovery architecture of the 1.5G enhanced call operation management platform is characterized by: It includes the following steps: Step 1: Establish a dedicated line connection between the source data center and the target data center to at least ensure that they consume the specified topic of the message queue specified by each other; Step 2: The source data center triggers the data operation and adds the corresponding synchronization annotation to form a data operation event, which is recorded in the database of the source data center; Step 3: When the event is successfully submitted, the transaction listener configured on the source data center listens to the data operation event through the synchronization annotation; Step 4, obtaining the operation object of the data operation event; Step 5: Generate SQL statements in reverse order according to the operation pair of the data operation event, save the data operation event in the local database and mark the local database log as unsynchronized; Step 6: The source data center sends the SQL statement and routing information to the message queue, waiting for the target data center to consume it; at the same time, the local database log is modified to be synchronized. Step 7: The target data center consumes the corresponding SQL statement from the message queue, determines whether it is synchronized based on the timestamp and component ID in the SQL statement, and performs compensatory synchronization on the SQL statement that is not synchronized in the local database to ensure eventual consistency.

2. According to claim 1, the transaction monitoring IDC synchronization method under the disaster recovery architecture of the 5G enhanced call operation management platform is characterized in that: The data operations in step 2 include data update, addition and deletion operations.

3. According to claim 1, the transaction monitoring IDC synchronization method under the disaster recovery architecture of the 5G enhanced call operation management platform is characterized in that: In step 2, convert the current transaction object into a JSON string and pass it to the transaction listener.

4. The transaction monitoring IDC synchronization method under the disaster recovery architecture of the 5G enhanced call operation management platform according to claim 3 is characterized in that: The current transaction object is converted into a json string and passed to the transaction listener through the AOP aspect.

5. According to claim 1, the transaction monitoring IDC synchronization method under the disaster recovery architecture of the 5G enhanced call operation management platform is characterized in that: In step 7, the target data center uses a program to generate the snowflake algorithm consumption SQL to ensure that the primary key is unique.

6. The transaction monitoring IDC synchronization method under the disaster recovery architecture of the 5G enhanced call operation management platform according to claim 1 is characterized in that: In step 7, the scheduled task is used to compensate for the synchronization of the SQL statements that have not been synchronized in the local database, and the SQL logs that have been synchronized before the specified date are deleted.

7. The transaction monitoring IDC synchronization method under the disaster recovery architecture of the 5G enhanced call operation management platform according to claim 6 is characterized in that: In step 7, delete the synchronized SQL logs before yesterday.

Images