Cross-network database synchronization method and system of cloud native architecture

By introducing synchronization management services, resource control, and exchange transmission engines under a cloud-native architecture, the reliability and efficiency issues of traditional database synchronization methods in cross-network scenarios are solved, achieving flexible and scalable real-time database synchronization.

CN118972434AActive Publication Date: 2024-11-15SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411082127.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-15
Estimated Expiration
2044-08-08

AI Technical Summary

Technical Problem

Traditional database synchronization methods are difficult to adapt to the flexibility and dynamism of cloud-native architectures, especially in cross-network, weak network, or edge computing scenarios, making it difficult to guarantee the reliability, efficiency, real-time performance, consistency, and security of database synchronization.

Method used

A cloud-native architecture is adopted for cross-network database synchronization, including synchronization management service, synchronization resource control, scheduling management service, task execution and exchange transmission. It utilizes a custom resource controller and exchange transmission engine in Kubernetes to provide a unified synchronization service interface and data transmission capability, and supports multi-replica deployment and dynamic scaling.

Benefits of technology

It achieves flexibility, scalability, and easy-to-manage real-time synchronization capabilities for cross-network database synchronization under a cloud-native architecture, improving the reliability and efficiency of database synchronization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972434A_ABST
    Figure CN118972434A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of databases, in particular to a cross-network database synchronization method and system for a cloud native architecture, and the method comprises the following steps: synchronous management service, synchronous resource control, scheduling management service, task execution and exchange transmission. The cross-network database synchronization method and system of the cloud native architecture have the advantages that the advantages of the cloud native technology, the lightweight data transmission technology and the data collection technology are combined, and the real-time synchronization capacity which is more flexible, extensible, lightweight and easy to manage is provided for heterogeneous database synchronization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a cross-network database synchronization method and system for a cloud native architecture. Background Art

[0002] With the continuous development and popularization of cloud computing technology, cloud-native architecture has gradually become the mainstream choice for enterprises to build and operate applications. Cloud-native architecture emphasizes the use of technical means such as containerization, microservices, automated deployment and operation and maintenance to improve the elasticity, scalability and maintainability of applications. In a cloud-native environment, database is one of the core components of the application, and its cross-network synchronization has become an important technical challenge.

[0003] Traditional database synchronization methods often rely on fixed network environments and complex configurations, and are difficult to adapt to the flexibility and dynamism of cloud-native architectures. Especially in cross-network, weak network or edge computing scenarios, the reliability and efficiency of database synchronization are even more difficult to guarantee. In addition, with the continuous expansion of enterprise business, the requirements for real-time, consistency and security of database synchronization are becoming increasingly higher. Summary of the invention

[0004] The purpose of the present invention is to provide a cross-network database synchronization method and system for a cloud native architecture to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a cross-network database synchronization method of a cloud native architecture, the method comprising the following steps:

[0006] Synchronous management service, providing a unified synchronization service interface and synchronization operation interface to the outside world;

[0007] Synchronous resource control provides the control of K8s custom resources required for synchronization, including instance resource controller, tenant resource controller, and task resource controller, which respectively control custom instance resources, tenant resources, and task resources;

[0008] Scheduling management service, responsible for task execution plan generation and scheduling, and maintaining the running status of related meta-information;

[0009] Task execution: receiving the task execution plan issued by the scheduling management service, executing the corresponding logic, and providing database collection and writing capabilities;

[0010] Switching transmission provides data transmission and caching capabilities, and enables data transmission in network isolation, weak network, and edge scenarios. It is divided into local switching transmission engine and central switching transmission engine.

[0011] Preferably, the specific operations of synchronous resource control include:

[0012] Synchronous resource control uses a set of K8s custom controllers to create and manage custom resources. The controller monitors and responds to the creation, update, and deletion operations of custom resources, provides lifecycle management of related resources, and can provide automatic scaling, creation, and upgrade of related service modules according to needs; the synchronous resource controller includes instance resource controller, tenant resource controller, and task resource controller; the instance resource controller corresponds to the synchronization service instance resource, describes the specifications of the basic service and its related configuration, and is responsible for the creation, expansion, and status monitoring of the synchronization service subject; the tenant resource controller corresponds to the tenant class custom resource, describes a class of tenant-isolated computing and transmission resource specifications, and its configuration, and the tenant resource controller is responsible for the creation, expansion, and destruction of the execution unit and the local exchange transmission engine; the task resource controller corresponds to the task custom resource, describes the basic elements of a synchronization task, including the source of data in the task, the flow of data, and the topological relationship information of the data. Through the task controller, the creation, update, and deletion operations of the synchronization task are realized, and the task status is modified through custom attributes, and the task entity associated with the resource is started, paused, and stopped.

[0013] Preferably, the specific operations of the scheduling management service include:

[0014] The scheduling management service is the core of database synchronization. It supports multi-copy deployment and is responsible for managing the system's metadata and maintaining the system's operating status. The system's metadata and operating status are divided into two categories: task information and tenant information, which are uniformly aggregated and transmitted through the exchange transmission engine. Tenant information specifically refers to the information of the execution unit, workload indicators, and its operating status, which are used to guide the scaling of tenant resources. Task information corresponds to the configuration information, execution results and status of subtasks associated with the synchronization task, and supports the distribution, migration, and scheduling decisions of the synchronization task.

[0015] Preferably, the specific operations performed by the task include:

[0016] Utilize the execution unit, which is the actual execution unit of the synchronization task, including database collection and database writing capabilities. The execution unit is a logical execution unit, a stateless component, and supports horizontal expansion. A single execution unit contains a basic front-end agent and a public runtime environment. The agent plug-in is issued through the synchronization service according to the type of task, providing the ability of tailoring and hot update. The database data collected by the collection service in the execution unit is cached in the local exchange engine, and then transmitted to the local exchange engine of the write service through the central exchange engine, realizing stable cross-network data transmission. When the tenant execution resources in the system are insufficient or idle, the dynamic scaling of the execution unit is realized by updating the associated custom resources.

[0017] Preferably, the specific operations of the exchange transmission include:

[0018] The switching transmission engine is the communication base of the system and the exchange center of the entire synchronization system data and events. It provides the ability to route, transmit, and persist cache messages and data between components within the system. A single switching transmission engine is composed of multiple service nodes. The nodes are connected and communicated with each other through cluster protocols to build a message transmission network. The switching transmission engine includes a central switching transmission engine and a local switching transmission engine. The switching transmission engine mainly provides core capabilities based on NATS.

[0019] A cross-network database synchronization system with a cloud native architecture, the system comprising a synchronization management module, a resource control module, a scheduling management module, an execution module and a transmission module;

[0020] The synchronization management module provides a unified synchronization service interface and synchronization operation interface to the outside world;

[0021] The resource control module provides management and control of K8s custom resources required for synchronization, including instance resource controller, tenant resource controller, and task resource controller, which respectively manage and control custom instance resources, tenant resources, and task resources;

[0022] The scheduling management module is responsible for generating and scheduling task execution plans and maintaining the running status of related meta-information;

[0023] The execution module receives the task execution plan issued by the scheduling management service, executes the corresponding logic, and provides database collection and writing capabilities;

[0024] The transmission module provides data transmission and caching capabilities, and supports data transmission in network isolation, weak network, and edge scenarios. It is divided into local switching transmission engine and central switching transmission engine.

[0025] Preferably, the resource control module and the synchronous resource control utilize a group of K8s custom controllers, which are responsible for creating and managing custom resources. The controller monitors and responds to the creation, update, and deletion operations of custom resources, provides lifecycle management of related resources, and can provide automatic scaling, creation, and upgrade of related service modules according to needs; the synchronous resource controller includes an instance resource controller, a tenant resource controller, and a task resource controller; the instance resource controller corresponds to the synchronous service instance resources, describes the specifications of the basic services and their related configurations, and is responsible for the creation, expansion, and status monitoring of the synchronization service subject; the tenant resource controller corresponds to the tenant class custom resources, describes the specifications of a class of tenant-isolated computing and transmission resources, and their configurations, and the tenant resource controller is responsible for the creation, expansion, and destruction of the execution unit and the local switching transmission engine; the task resource controller corresponds to the task custom resources, describes the basic elements of a synchronization task, including the source of data in the task, the flow of data, and the topological relationship information of the data. Through the task controller, the creation, update, and deletion operations of the synchronization task are realized, and the task status is modified through custom attributes, and the task entity associated with the resource is started, paused, and stopped.

[0026] Preferably, the scheduling management module and the scheduling management service are the core of database synchronization, support multi-copy deployment, are responsible for managing the system's metadata, and maintain the system's operating status. The system metadata and operating status are classified into two categories, namely, task information and tenant information, which are uniformly aggregated and transmitted through an exchange transmission engine; tenant information specifically refers to the information of the execution unit, workload indicators, and its operating status, which is used to guide the scaling of tenant resources; task information corresponds to the configuration information, execution results and status of subtasks associated with the synchronization task, and supports the distribution, migration, and scheduling decisions of the synchronization task.

[0027] Preferably, the execution module utilizes an execution unit, which is the actual execution unit of the synchronization task, including database collection and database writing capabilities. The execution unit is a logical execution unit, a stateless component, and supports horizontal expansion; a single execution unit includes a basic front-end agent and a public runtime environment, and the agent plug-in is issued through the synchronization service according to the type of task, and provides the capabilities of tailoring and hot update. The database data collected by the collection service in the execution unit is cached in the local switching engine, and then transmitted to the local switching engine of the writing service through the central switching engine, so as to realize stable transmission of data across the network; when the execution resources of the tenants in the system are insufficient or idle, the dynamic scaling of the execution unit is realized by updating the associated custom resources.

[0028] Preferably, the transmission module, the switching transmission engine is the communication base of the system, the switching center of the entire synchronization system data and events, providing the ability to route, transmit, and persistently cache messages and data between components within the system. A single switching transmission engine is composed of multiple service nodes, and the nodes are interconnected and communicated through a cluster protocol to build a message transmission network; the switching transmission engine includes a central switching transmission engine and a local switching transmission engine, and the switching transmission engine mainly provides core capabilities based on NATS.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The cross-network database synchronization method and system of the cloud-native architecture proposed in the present invention, combining the advantages of cloud-native technology, lightweight data transmission technology, and data acquisition technology, provides a more flexible, scalable, lightweight, and easy-to-manage real-time synchronization capability for heterogeneous database synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is the overall architecture diagram of the present invention;

[0032] Figure 2 This is a diagram of the architecture of the switching transmission engine of the present invention;

[0033] Figure 3 FIG. 2 is a diagram of the execution unit architecture of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Embodiment 1

[0036] See also Figures 1 to 3 The present invention provides a technical solution: a cross-network database synchronization method of a cloud native architecture, the method comprising the following steps:

[0037] It consists of five parts: synchronization management service, synchronization resource controller, scheduling management service, execution unit, and switching transmission engine.

[0038] The synchronization management service is the synchronization service management layer, which provides a unified synchronization service interface and synchronization operation interface to the outside world; the synchronization resource controller provides management and control of the K8s custom resources required for synchronization, including the instance resource controller, tenant resource controller, and task resource controller, which respectively manage and control custom instance resources, tenant resources, and task resources. The scheduling management service is responsible for task execution plan generation and scheduling, and maintains related metadata and the operating status of the system. The execution unit is the actual execution unit of the task, receives the task execution plan issued by the scheduling management service, executes the corresponding logic, and provides database collection and writing capabilities. The exchange transmission engine provides data transmission and caching capabilities, and provides data transmission in network isolation, weak network, and edge scenarios. It is divided into local exchange transmission engine and central exchange transmission engine. The overall architecture is as follows Figure 1 shown.

[0039] Database synchronization is mainly divided into two stages, namely the creation of synchronization task-related resources and the scheduling, distribution, and execution of synchronization tasks. The specific process is as follows:

[0040] (1) The user inputs the metadata of the created synchronization task to the synchronization management service through the interface or interface, including the source / destination database address, user name, password; synchronized database table information; synchronization task attribute configuration and other basic information.

[0041] (2) After receiving the user information, the synchronization management service calls the scheduling management service interface to determine whether there is an execution unit that meets the requirements. If there is an available execution unit, it goes to step 4, otherwise it goes to step 3.

[0042] (3) Synchronization management service, which calls the tenant resource controller in the synchronization resource controller according to the task configuration information (source and destination database information) to create the associated local switching transmission engine and execution unit.

[0043] (4) Synchronous management service: calls the task resource controller in the synchronous resource controller according to the task configuration information (source, destination), and the task resource controller calls the relevant interface of the scheduling management service component to create a task.

[0044] (5) After receiving the created task, the scheduling management service generates a task execution plan, including an extraction end execution plan or a storage end execution plan, and sends each execution plan to the corresponding execution unit through the exchange transmission engine.

[0045] (6) The execution unit receives the task execution plan, builds the extraction end or storage segment execution environment according to the specific type, and feeds back the execution results, and collects or writes database data according to the needs.

[0046] 1Synchronous Resource Controller

[0047] The synchronous resource controller is a set of K8s custom controllers responsible for creating and managing custom resources. The controller monitors and responds to the creation, update, and deletion operations of custom resources, provides lifecycle management for related resources, and can provide automatic scaling, creation, and upgrade of related service modules according to demand.

[0048] The synchronization resource controller includes instance resource controller, tenant resource controller, and task resource controller.

[0049] The instance resource controller corresponds to the synchronization service instance resource. It describes the specifications of the basic service and its related configuration, and is responsible for the creation, expansion, and status monitoring of the synchronization service subject (scheduling management service, central switching transmission engine).

[0050] The tenant resource controller corresponds to the tenant-class custom resources. It mainly describes the specifications and configuration of a class of tenant-isolated computing and transmission resources. The tenant resource controller is responsible for the creation, expansion, and destruction of execution units and local switching transmission engines.

[0051] The task resource controller corresponds to the task custom resource. It describes the basic elements of a synchronization task, including the source of data in the task, the direction of data flow, the topological relationship of data, and other information. Through the task controller, we can create, update, and delete synchronization tasks. At the same time, we can modify the status of the task through custom properties, and start, pause, and stop the task entity associated with the resource.

[0052] 2Scheduling management services

[0053] The scheduling management service is the core of database synchronization and supports multi-copy deployment. It is responsible for managing the system's metadata and maintaining the system's operating status. System metadata and operating status can be divided into two categories: task information and tenant information, which are uniformly aggregated and transmitted through the exchange transmission engine.

[0054] The tenant information specifically refers to the information of the execution unit, workload indicators, and its running status, which is used to guide the scaling of tenant resources.

[0055] Task class information corresponds to the configuration information, execution results and status of subtasks associated with the synchronization task, and supports the decision-making of the distribution, migration and scheduling of synchronization tasks.

[0056] 3Switch Transfer Engine

[0057] The exchange transmission engine is the communication base of the system and the exchange center of the entire synchronous system data and events. It provides the ability to route, transmit, and persist cache messages (control instructions, monitoring information) and data (DDL events, DML events) between components within the system. A single exchange transmission engine (central, local) is composed of multiple service nodes (nodes). The nodes are connected and communicated with each other through cluster protocols to build a message transmission network. The exchange transmission engine includes a central exchange transmission engine and a local exchange transmission engine. The exchange transmission engine mainly provides core capabilities based on NATS. The network topology relationship is as follows: Figure 2 shown.

[0058] The central switching transmission engine mainly provides cross-cloud and cross-regional network collaboration capabilities. Multiple central switching transmission engines are connected using a gateway protocol to achieve message transmission between different central switching engines.

[0059] The local switching transmission engine provides basic message transmission capabilities, as well as local message persistence and caching capabilities, supporting secure and reliable transmission of data messages between working computing nodes. Multiple local switching transmission engines can actively connect to the central switching transmission engine through the edge extension protocol to achieve routing, forwarding, and switching transmission of messages between local switching engines with network isolation.

[0060] The switching transmission engine also supports data encryption, flow control and other functions, providing a controllable and secure transmission channel for the system. The switching transmission engine decouples the relationship between the components in the system, simplifies the network topology, provides the underlying abstraction of the control and transmission channels, and improves the overall flexibility, stability and scalability of the system.

[0061] 4 Execution Unit

[0062] The execution unit is the actual execution unit of the synchronization task, including database collection and database writing capabilities. The execution unit is a logical execution unit, a stateless component, and supports horizontal expansion.

[0063] A single execution unit contains the basic front-end agent and the public runtime environment (the common logic of heterogeneous database synchronization and sharing). The agent plug-in is issued through the synchronization service according to the type of task, providing the ability to tailor and hot update. The database data collected by the collection service in the execution unit is cached in the local exchange engine, and then transmitted to the local exchange engine of the write service through the central exchange engine to achieve stable data transmission across the network. Figure 3 As shown, the execution unit receives control instructions through the front-end agent and performs related operations.

[0064] When the tenant execution resources in the system are insufficient or idle, the execution unit can be dynamically scaled by updating the associated custom resources. The execution unit is stateless, which means that tasks can be easily migrated between different computing units, and the synchronization service can be reasonably scheduled based on the availability and load of each execution unit.

[0065] Embodiment 2

[0066] Based on the first embodiment, a cross-network database synchronization system with a cloud native architecture is proposed, wherein the system is composed of a synchronization management module, a resource control module, a scheduling management module, an execution module, and a transmission module;

[0067] The synchronization management module provides a unified synchronization service interface and synchronization operation interface to the outside world;

[0068] The resource control module provides the control of K8s custom resources required for synchronization, including instance resource controller, tenant resource controller, and task resource controller, which respectively control custom instance resources, tenant resources, and task resources; the synchronization resource control uses a group of K8s custom controllers to create and manage custom resources. The controller monitors and responds to the creation, update, and deletion operations of custom resources, provides life cycle management of related resources, and can provide automatic scaling, creation, and upgrade of related service modules according to needs; the synchronization resource controller includes instance resource controller, tenant resource controller, and task resource controller; the instance resource controller corresponds to the synchronization service instance resource, and describes the basic The tenant resource controller corresponds to the tenant class custom resources, describes the specifications of a class of tenant-isolated computing and transmission resources, and their configuration. The tenant resource controller is responsible for the creation, expansion, and destruction of execution units and local switching transmission engines. The task resource controller corresponds to the task custom resources, describes the basic elements of a synchronization task, including the source of data in the task, the direction of data flow, and the topological relationship information of the data. Through the task controller, the creation, update, and deletion of synchronization tasks are implemented. At the same time, the task status is modified through custom attributes, and the task entity associated with the resource can be started, paused, and stopped.

[0069] The scheduling management module is responsible for generating and scheduling task execution plans and maintaining the running status of related meta-information. The scheduling management service is the core of database synchronization, supports multi-copy deployment, and is responsible for managing the system's meta-information and maintaining the system's running status. The system meta-information and running status are divided into two categories, namely task information and tenant information, which are uniformly aggregated and transmitted through the exchange transmission engine. Tenant information specifically refers to the information of the execution unit, workload indicators, and its running status, which is used to guide the scaling of tenant resources. Task information corresponds to the configuration information, execution results and status of sub-tasks associated with the synchronization task, and supports the distribution, migration, and scheduling decisions of synchronization tasks.

[0070] The execution module receives the task execution plan issued by the scheduling management service, executes the corresponding logic, and provides database collection and writing capabilities; uses the execution unit, which is the actual execution unit of the synchronization task, including database collection and database writing capabilities. The execution unit is a logical execution unit, a stateless component, and supports horizontal expansion; a single execution unit contains a basic front-end agent and a public runtime environment. The agent plug-in is issued through the synchronization service according to the type of task, and provides the ability to cut down and hot update. The database data collected by the collection service in the execution unit is cached in the local exchange engine, and then transmitted to the local exchange engine of the write service through the central exchange engine, so as to realize stable cross-network data transmission; when the tenant execution resources in the system are insufficient or idle, the dynamic scaling of the execution unit is realized by updating the associated custom resources.

[0071] The transmission module provides data transmission and caching capabilities, and provides data transmission in network isolation, weak network, and edge scenarios. It is divided into local switching transmission engine and central switching transmission engine. The switching transmission engine is the communication base of the system and the exchange center of the entire synchronization system data and events. It provides the ability to route, transmit, and persist cache messages and data between components within the system. A single switching transmission engine is composed of multiple service nodes. The nodes are connected and communicated with each other through a cluster protocol to build a message transmission network. The switching transmission engine includes a central switching transmission engine and a local switching transmission engine. The switching transmission engine mainly provides core capabilities based on NATS.

[0072] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A cross-network database synchronization method for a cloud native architecture, characterized by: The method comprises the following steps: Synchronous management service, providing a unified synchronization service interface and synchronization operation interface to the outside world; Synchronous resource control provides the control of K8s custom resources required for synchronization, including instance resource controller, tenant resource controller, and task resource controller, which respectively control custom instance resources, tenant resources, and task resources; Scheduling management service, responsible for task execution plan generation and scheduling, and maintaining the running status of related meta-information; Task execution: receiving the task execution plan issued by the scheduling management service, executing the corresponding logic, and providing database collection and writing capabilities; Switching transmission provides data transmission and caching capabilities, and enables data transmission in network isolation, weak network, and edge scenarios. It is divided into local switching transmission engine and central switching transmission engine.

2. According to the cloud native architecture cross-network database synchronization method of claim 1, it is characterized by: The specific operations of synchronous resource control include: Synchronous resource control uses a set of K8s custom controllers to create and manage custom resources. The controller monitors and responds to the creation, update, and deletion operations of custom resources, provides lifecycle management of related resources, and can provide automatic scaling, creation, and upgrade of related service modules according to needs; the synchronous resource controller includes instance resource controller, tenant resource controller, and task resource controller; the instance resource controller corresponds to the synchronization service instance resource, describes the specifications of the basic service and its related configuration, and is responsible for the creation, expansion, and status monitoring of the synchronization service subject; the tenant resource controller corresponds to the tenant class custom resource, describes a class of tenant-isolated computing and transmission resource specifications, and its configuration, and the tenant resource controller is responsible for the creation, expansion, and destruction of the execution unit and the local exchange transmission engine; the task resource controller corresponds to the task custom resource, describes the basic elements of a synchronization task, including the source of data in the task, the flow of data, and the topological relationship information of the data. Through the task controller, the creation, update, and deletion operations of the synchronization task are realized, and the task status is modified through custom attributes, and the task entity associated with the resource is started, paused, and stopped.

3. According to a cloud native architecture cross-network database synchronization method according to claim 1, it is characterized by: The specific operations of the scheduling management service include: The scheduling management service is the core of database synchronization. It supports multi-copy deployment and is responsible for managing the system's metadata and maintaining the system's operating status. The system's metadata and operating status are divided into two categories: task information and tenant information, which are uniformly aggregated and transmitted through the exchange transmission engine. Tenant information specifically refers to the information of the execution unit, workload indicators, and its operating status, which are used to guide the scaling of tenant resources. Task information corresponds to the configuration information, execution results and status of subtasks associated with the synchronization task, and supports the distribution, migration, and scheduling decisions of the synchronization task.

4. According to a cloud native architecture cross-network database synchronization method according to claim 1, it is characterized in that: The specific operations performed by the task include: Utilize the execution unit, which is the actual execution unit of the synchronization task, including database collection and database writing capabilities. The execution unit is a logical execution unit, a stateless component, and supports horizontal expansion. A single execution unit contains a basic front-end agent and a public runtime environment. The agent plug-in is issued through the synchronization service according to the type of task, providing the ability of tailoring and hot update. The database data collected by the collection service in the execution unit is cached in the local exchange engine, and then transmitted to the local exchange engine of the write service through the central exchange engine, realizing stable cross-network data transmission. When the tenant execution resources in the system are insufficient or idle, the dynamic scaling of the execution unit is realized by updating the associated custom resources.

5. According to a cloud native architecture cross-network database synchronization method according to claim 1, it is characterized in that: The specific operations of exchange transmission include: The switching transmission engine is the communication base of the system and the exchange center of the entire synchronization system data and events. It provides the ability to route, transmit, and persist cache messages and data between components within the system. A single switching transmission engine is composed of multiple service nodes. The nodes are connected and communicated with each other through cluster protocols to build a message transmission network. The switching transmission engine includes a central switching transmission engine and a local switching transmission engine. The switching transmission engine mainly provides core capabilities based on NATS.

6. A cross-network database synchronization system of a cloud native architecture according to the cross-network database synchronization method of a cloud native architecture according to any one of claims 1 to 5, characterized in that: The system is composed of a synchronization management module, a resource control module, a scheduling management module, an execution module and a transmission module; The synchronization management module provides a unified synchronization service interface and synchronization operation interface to the outside world; The resource control module provides management and control of K8s custom resources required for synchronization, including instance resource controller, tenant resource controller, and task resource controller, which respectively manage and control custom instance resources, tenant resources, and task resources; The scheduling management module is responsible for generating and scheduling task execution plans and maintaining the running status of related meta-information; The execution module receives the task execution plan issued by the scheduling management service, executes the corresponding logic, and provides database collection and writing capabilities; The transmission module provides data transmission and caching capabilities, and supports data transmission in network isolation, weak network, and edge scenarios. It is divided into local switching transmission engine and central switching transmission engine.

7. The cross-network database synchronization system of a cloud native architecture according to claim 6, characterized in that: The resource control module, the synchronous resource control uses a group of K8s custom controllers, which are responsible for creating and managing custom resources. The controller monitors and responds to the creation, update, and deletion operations of custom resources, provides lifecycle management of related resources, and can provide automatic scaling, creation, and upgrade of related service modules according to demand; the synchronous resource controller includes an instance resource controller, a tenant resource controller, and a task resource controller; the instance resource controller corresponds to the synchronous service instance resources, describes the specifications of the basic services and their related configurations, and is responsible for the creation, expansion, and status monitoring of the synchronous service subject; the tenant resource controller corresponds to the tenant class custom resources, describes a class of tenant-isolated computing and transmission resource specifications, and their configuration, and the tenant resource controller is responsible for the creation, expansion, and destruction of the execution unit and the local switching transmission engine; The task resource controller corresponds to the task custom resource and describes the basic elements of a synchronization task, including the source of data in the task, the direction of data flow, and the topological relationship information of the data. Through the task controller, the creation, update, and deletion of synchronization tasks are implemented. At the same time, the task status can be modified through custom attributes, and the task entity associated with the resource can be started, paused, and stopped.

8. The cross-network database synchronization system of a cloud native architecture according to claim 6, characterized in that: The scheduling management module and scheduling management service are the core of database synchronization, support multi-copy deployment, are responsible for managing the system's metadata, and maintain the system's operating status. The system metadata and operating status are classified into two categories, namely task information and tenant information, which are uniformly aggregated and transmitted through the exchange transmission engine; tenant information specifically refers to the information of the execution unit, workload indicators, and its operating status, which are used to guide the scaling of tenant resources; task information corresponds to the configuration information, execution results and status of subtasks associated with the synchronization task, and supports the distribution, migration, and scheduling decisions of the synchronization task.

9. The cross-network database synchronization system of a cloud native architecture according to claim 6, characterized in that: The execution module utilizes an execution unit, which is the actual execution unit of the synchronization task, including database collection and database writing capabilities. The execution unit is a logical execution unit, a stateless component, and supports horizontal expansion. A single execution unit includes a basic front-end agent and a public runtime environment. The agent plug-in is issued through the synchronization service according to the type of task, and provides the capabilities of tailoring and hot update. The database data collected by the collection service in the execution unit is cached in the local exchange engine, and then transmitted to the local exchange engine of the write service through the central exchange engine, so as to realize stable cross-network transmission of data. When the tenant execution resources in the system are insufficient or idle, the dynamic scaling of the execution unit is realized by updating the associated custom resources.

10. The cross-network database synchronization system of a cloud native architecture according to claim 6, characterized in that: The transmission module, the exchange transmission engine is the communication base of the system, the exchange center of the entire synchronization system data and events, and provides the ability to route, transmit, and persist cache messages and data between components within the system. A single exchange transmission engine is composed of multiple service nodes, and the nodes are connected and communicated with each other through the cluster protocol to build a message transmission network; The switching and transmission engine includes the central switching and transmission engine and the local switching and transmission engine. The switching and transmission engine mainly provides core capabilities based on NATS.

Citation Information

Patent Citations

  • Database synchronization method and device, electronic equipment and storage medium

    CN114510531A

  • File synchronization system of cloud edge scene based on NATS

    CN115514726A

  • Cross-network data sharing and exchanging system

    CN117041185A

  • System and method for coordinating and executing complex communication tasks using structured messaging and off-line synchronization

    WO2023081432A1