Database synchronization method and system supporting virtual machine and container deployment

Through Nomad scheduler and containerization or virtualization technology, database synchronization tools are efficiently deployed and managed in virtual machines and container environments, solving the problems of cumbersome deployment and poor scalability, and providing efficient, reliable and secure database synchronization plan.

CN119025593AActive Publication Date: 2024-11-26SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411049451.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-26
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

The existing database synchronization tools are cumbersome and poorly scalable in virtual machines or container environments, and it is difficult to adapt to rapidly changing business needs.

Method used

Using Nomad as the core scheduler, the database synchronization tasks are abstracted into configurable service units by defining Nomad task specifications, and the automated deployment function of Nomad automatically creates and manages containers or virtual machine instances, combining service discovery, monitoring and security mechanisms to achieve Efficient and reliable database synchronization.

Benefits of technology

It realizes efficient and reliable execution of database synchronization tasks, simplifies the deployment process, improves resource utilization and system reliability, enhances security, and is suitable for small-scale deployment and simple scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025593A_ABST
    Figure CN119025593A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of database synchronization, in particular to a database synchronization method and system supporting virtual machine and container deployment, and the method comprises the following steps: architecture design, task definition and configuration, automatic deployment and intelligent scheduling, service discovery and load balancing, monitoring and alarm, and security guarantee. The database synchronization method and system supporting virtual machine and container deployment have the advantages that the advantages of Nomad distributed scheduling and containerization or virtualization technologies are combined, and efficient and reliable execution of database synchronization tasks is achieved through automatic deployment, intelligent scheduling, elastic expansion and other means.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of database synchronization, and in particular to a database synchronization method and system supporting virtual machine and container deployment. Background Art

[0002] With the rapid development of informatization today, database synchronization is widely used in various business scenarios as an important means to ensure data consistency and reliability. However, the traditional deployment method of database synchronization tools usually relies on a specific hardware environment or operating system, resulting in a cumbersome deployment process, poor scalability, and difficulty in adapting to rapidly changing business needs.

[0003] In recent years, with the rise of containerization and virtualization technologies, the way database synchronization tools are deployed and managed has changed significantly. Containerization technology packages applications and their dependencies into portable container images, allowing applications to be quickly deployed and run in any environment that supports container technology. Virtualization technology provides a higher level of resource isolation and flexibility by creating a virtual computing environment.

[0004] As a distributed, high-availability, data center-aware scheduler, Nomad has excellent scheduling algorithms, service discovery mechanisms, and dynamic expansion capabilities, making it ideal for building elastic and scalable distributed systems. However, there is currently no mature solution that can fully utilize the advantages of Nomad to achieve efficient deployment and management of database synchronization tools in virtual machine or container environments. Summary of the invention

[0005] The purpose of the present invention is to provide a database synchronization method and system supporting virtual machine and container deployment to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a database synchronization method supporting virtual machine and container deployment, the method comprising the following steps:

[0007] Architecture design, using Nomad as the core scheduler, responsible for scheduling and managing database synchronization tasks;

[0008] Task definition and configuration: By defining Nomad task specifications, database synchronization tasks are abstracted into configurable service units;

[0009] Automated deployment and intelligent scheduling: Use Nomad's automated deployment function to automatically create and start corresponding containers or virtual machine instances based on task configurations.

[0010] Service discovery and load balancing: Automatically register and discover database synchronization service instances through Nomad’s service discovery mechanism;

[0011] Monitoring and alarming: real-time monitoring of the execution status of database synchronization tasks, resource usage, and cluster health;

[0012] Security assurance: By isolating container or virtual machine environments, security risks between different tasks are reduced; data isolation between different jobs is enhanced through network layer policies.

[0013] Preferably, the specific operations of the architecture design include:

[0014] Use containerization or virtualization technology to encapsulate database synchronization tools and their dependent environments to form portable deployment units. Through Nomad's client and server architecture, dynamic task allocation and unified management of cluster resources are achieved.

[0015] Nomad supports multiple types of drivers, through which tasks can be run and scheduled on different platforms. It supports running tasks in Docker containers, running binary files or scripts directly on the host, and running Java applications. It can choose the most appropriate deployment method to implement tasks without limiting the implementation and deployment methods of application services.

[0016] A Nomad cluster consists of a server and one or more client hosts. It runs tasks according to policy scheduling within the entire cluster and exits after the function is completed.

[0017] Preferably, the specific operations of task definition and configuration include:

[0018] Task configuration includes synchronization strategy, data source information, target database information, required synchronized library tables, and key parameters for task unique identification. This configuration is recorded in the syncmgr task, which modifies the number of parallel reader / writer task instances based on the task progress. In addition to specific configuration parameters, all tasks also need to configure the driver type and resource restrictions of the task. After the configuration is determined, the Nomad job is started, and then the running of each task is scheduled.

[0019] Preferably, the specific operations of automated deployment and intelligent scheduling include:

[0020] Through Nomad's scheduling algorithm, virtual machine or container resources are intelligently allocated according to the cluster's resource usage and task requirements to ensure efficient execution of synchronization tasks. Nomad also supports dynamic migration and fault recovery of tasks to ensure continuous availability of services. Nomad supports multiple scheduling strategies, including least used, best location, and random, to help tasks run on the best nodes in the cluster, thereby optimizing resource utilization and performance. The reader is deployed on a host close to the source database, and the writer is deployed on a host close to the destination database. Multiple reader and writer task instances are started during the synchronization of library table structures and full data to maximize resource usage with concurrent efficiency.

[0021] Preferably, the specific operations of service discovery and load balancing include:

[0022] When new tasks are added or existing tasks exit, Nomad will automatically update the service list and distribute requests to appropriate service instances based on the load balancing strategy to achieve automatic expansion and failover of services; at different stages of task operation, change the Reader / writer configuration to one or more instances, and Nomad will automatically expand or shrink the actual running task instances.

[0023] A database synchronization system supporting virtual machine and container deployment, the system consists of an architecture design module, a task configuration module, a deployment module, a service discovery module, a monitoring module and a security module;

[0024] The architecture design module uses Nomad as the core scheduler, responsible for the scheduling and management of database synchronization tasks;

[0025] The task configuration module is used for task definition and configuration. By defining the Nomad task specification, the database synchronization task is abstracted into a configurable service unit.

[0026] The deployment module is used for automated deployment and intelligent scheduling. It uses Nomad's automated deployment function to automatically create and start corresponding containers or virtual machine instances according to task configurations.

[0027] The service discovery module is used for service discovery and load balancing. It automatically registers and discovers database synchronization service instances through Nomad's service discovery mechanism.

[0028] The monitoring module is used for monitoring and alarming, and real-time monitoring of the execution status of database synchronization tasks, resource usage, and cluster health status;

[0029] The security module is used for security assurance. It reduces security risks between different tasks by isolating container or virtual machine environments and enhances data isolation between different jobs through network layer policies.

[0030] Preferably, the architecture design module uses containerization or virtualization technology to encapsulate the database synchronization tool and its dependent environment to form a portable deployment unit, and realizes dynamic allocation of tasks and unified management of cluster resources through Nomad's client and server architecture;

[0031] Nomad supports multiple types of drivers, through which tasks can be run and scheduled on different platforms. It supports running tasks in Docker containers, running binary files or scripts directly on the host, and running Java applications. It can choose the most appropriate deployment method to implement tasks without limiting the implementation and deployment methods of application services.

[0032] A Nomad cluster consists of a server and one or more client hosts. It runs tasks according to policy scheduling within the entire cluster and exits after the function is completed.

[0033] Preferably, in the task configuration module, the task configuration includes synchronization strategy, data source information, target database information, library tables to be synchronized, and key parameters for task unique identification. This configuration is recorded in the syncmgr task, and the syncmgr modifies the number of parallel reader / writer task instances according to the task progress. In addition to specific configuration parameters, all tasks also need to configure the Driver type and resource restrictions of the task. After the configuration is determined, the Nomad job is started, and then the operation of each task is scheduled.

[0034] Preferably, the deployment module, through Nomad's scheduling algorithm, intelligently allocates virtual machine or container resources according to the resource usage and task requirements of the cluster to ensure efficient execution of synchronization tasks. Nomad also supports dynamic migration and fault recovery of tasks to ensure continuous availability of services. Nomad supports multiple scheduling strategies, including least used, best location and random, to help tasks run on the best nodes of the cluster, thereby optimizing resource utilization and performance, setting the reader to be deployed on a host close to the source database, setting the writer to be deployed on a host close to the destination database, and starting multiple reader and writer task instances in the synchronization of library table structure and full data stages to maximize resource usage with concurrent efficiency.

[0035] Preferably, in the service discovery module, when a new task is added or an existing task is exited, Nomad will automatically update the service list and distribute the request to the appropriate service instance according to the load balancing strategy, so as to realize automatic expansion and failover of the service; at different stages of task operation, the Reader / writer configuration is changed to one or more instances, and Nomad will automatically expand or shrink the task instance actually running.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The database synchronization method and system that supports virtual machine and container deployment proposed in the present invention combines the advantages of Nomad's distributed scheduling and containerization or virtualization technology, and realizes efficient and reliable execution of database synchronization tasks through automated deployment, intelligent scheduling, and elastic expansion. This is the advantage of Nomad compared to k8s. It not only supports containerized applications, but also supports other types of workloads, which makes it easier to use existing application systems instead of having to run them after container transformation; and Nomad's architecture is relatively lightweight, suitable for small-scale deployment and simple scenarios. Compared with k8s, Nomad can achieve more flexible deployment methods and a higher degree of customization. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Embodiment 1

[0041] See also Figure 1 The present invention provides a technical solution: a database synchronization method supporting virtual machine and container deployment, the method comprising the following steps:

[0042] 1. Architecture Design

[0043] The present invention uses Nomad as the core scheduler, which is responsible for the scheduling and management of database synchronization tasks. At the same time, the database synchronization tool and its dependent environment are encapsulated using containerization or virtualization technology to form a portable deployment unit. Through Nomad's client and server architecture, dynamic allocation of tasks and unified management of cluster resources are achieved.

[0044] Nomad supports multiple types of drivers, through which tasks can be run and scheduled on different platforms, support running tasks in Docker containers, support running binary files or scripts directly on the host, support running Java applications, etc. Therefore, you can choose the most appropriate deployment method to implement the task, without limiting the implementation and deployment method of the application service.

[0045] A Nomad cluster consists of a server and one or more client hosts, which can schedule and run tasks in the entire cluster according to policies. Tasks can be long-running tasks, such as databases, or short-run tasks, such as starting to run and completing a specific function and then exiting after the function is completed.

[0046] The overall framework design is as follows:

[0047] Registration service: As a registration and discovery server, other tasks are registered here after running.

[0048] Configuration library: records the running status of synchronization tasks, such as whether the library table structure is collected, whether it is stored, whether the full amount is collected, whether the full amount is stored, etc.

[0049] Prometheus monitoring system: is an open source monitoring and alerting tool that is mainly used to collect, store and query various types of time series data. It can monitor the performance indicators of servers, applications and network services so that administrators can detect problems in time and take appropriate measures.

[0050] MQ Cluster: A message middleware cluster in which data topics can be created to achieve decoupling, asynchronous communication, traffic peak shaving, data collision prevention, and data reliability assurance, thus improving the maintainability, performance, and stability of the system.

[0051] Reader: Provides database collection functions through APIs, executes tasks scheduled by syncmgr, obtains data from the source database, and sends it to the MQ Cluster.

[0052] Writer: Provides database storage function through API, executes tasks scheduled by syncmgr, obtains data from MQ Cluster, and writes to the destination database.

[0053] Syncmgr: responsible for scheduling specific synchronization tasks and scheduling readers / writers to execute tasks according to the task progress.

[0054] The reader, writer, and syncmgr tasks of the synchronization task are all configured with a unique task identifier. The task status is not saved, and the task information is recorded in the configuration library. The task can start one or more instances as needed. After the task is started, it is registered with the configuration and discovery server. Syncmgr calls the reader / writer API interface as needed to perform collection and storage tasks. Reader implements a series of collection function APIs, such as collecting the structure of a table and sending it to a topic in MQ; obtaining the data sharding of a table; collecting the data of a shard of a table and sending it to a topic in MQ, etc. Writer implements a series of storage-related APIs, such as pulling data from a topic and processing it. Syncmgr provides a stateful upload API, which the reader / writer can call to upload the task status, such as the completion of the collection of a data shard, the change of the incremental point on the collection end, etc. for task status monitoring. The amount of data collected and stored, data packets, and other information are sent to the monitoring system.

[0055] 2. Task Definition and Configuration

[0056] The present invention abstracts database synchronization tasks into configurable service units by defining Nomad task specifications. Task configuration includes key parameters such as synchronization strategy, data source information, target database information, required synchronized library tables, and task unique identifiers. This configuration is recorded in the syncmgr task. The syncmgr modifies the number of parallel reader / writer task instances according to the task progress.

[0057] In addition to specific configuration parameters, all tasks also need to configure the task's Driver type, resource limits (CPU, memory, disk, etc. set by the task), etc.

[0058] After the configuration is determined, you can start the Nomad job and then schedule the running of each task.

[0059] 3. Automated Deployment and Intelligent Scheduling

[0060] The present invention uses Nomad's automated deployment function to automatically create and start corresponding containers or virtual machine instances according to task configuration. Through Nomad's scheduling algorithm, virtual machine or container resources are intelligently allocated according to cluster resource usage and task requirements to ensure efficient execution of synchronization tasks. At the same time, Nomad also supports dynamic migration and fault recovery of tasks to ensure the continuous availability of services.

[0061] Nomad supports multiple scheduling strategies, including least used, best location, and random, to help tasks run on the best nodes in the cluster, thereby optimizing resource utilization and performance. Set the reader to be deployed on a host close to the source database, and set the writer to be deployed on a host close to the destination database. During the synchronization of the library table structure and the full data, multiple reader and writer task instances can be started to maximize resource utilization with concurrent efficiency.

[0062] 4. Service Discovery and Load Balancing

[0063] The present invention automatically registers and discovers database synchronization service instances through Nomad's service discovery mechanism. When new tasks are added or existing tasks are exited, Nomad will automatically update the service list and distribute the requests to the appropriate service instances according to the load balancing strategy, realizing automatic expansion and failover of services.

[0064] Since the syncmgr, reader, and writer of the synchronization task have no status, they are registered to the service registration and discovery server after the task is run, and the status of the task operation is saved in the configuration database. You can change the reader / writer configuration to one or more instances at different stages of the task operation, and Nomad will automatically expand or shrink the actual running task instance.

[0065] V. Monitoring and Alerting

[0066] The present invention provides a complete monitoring and alarm mechanism to monitor the execution status of database synchronization tasks, resource usage, and cluster health in real time. When an abnormality or failure occurs, the system will automatically trigger an alarm notification and take corresponding treatment measures to ensure the stable operation of the service.

[0067] Nomad provides an API to obtain the running status of jobs and tasks, and can also send monitoring and alarm information to Prometheus, etc. Syncmgr can obtain data from the monitoring system to generate job status, such as the progress of collection and storage, the amount of data cached by MQ Cluster, the amount of data collected or stored, etc., and provide it to the outside world in the form of API.

[0068] 6. Security

[0069] Nomad itself has taken a number of measures in terms of security. By isolating container or virtual machine environments, it reduces security risks between different tasks; and enhances data isolation between different jobs through network layer strategies.

[0070] In summary, the present invention provides an efficient, reliable, and scalable database synchronization tool deployment solution by integrating Nomad's distributed scheduling and containerization or virtualization technology. This solution not only simplifies the deployment process and improves resource utilization, but also enhances the reliability and security of the system, providing strong technical support for distributed database synchronization scenarios.

[0071] Embodiment 2

[0072] Based on the first embodiment, a database synchronization system supporting virtual machine and container deployment is proposed, the system consists of an architecture design module, a task configuration module, a deployment module, a service discovery module, a monitoring module and a security module;

[0073] The architecture design module uses Nomad as the core scheduler, responsible for the scheduling and management of database synchronization tasks. It uses containerization or virtualization technology to encapsulate database synchronization tools and their dependent environments to form a portable deployment unit. Through Nomad's client and server architecture, it realizes dynamic allocation of tasks and unified management of cluster resources.

[0074] Nomad supports multiple types of drivers, through which tasks can be run and scheduled on different platforms. It supports running tasks in Docker containers, running binary files or scripts directly on the host, and running Java applications. It can choose the most appropriate deployment method to implement tasks without limiting the implementation and deployment methods of application services.

[0075] A Nomad cluster consists of a server and one or more client hosts. It runs tasks according to policy scheduling within the entire cluster and exits after the function is completed.

[0076] The task configuration module is used for task definition and configuration. By defining the Nomad task specification, the database synchronization task is abstracted into a configurable service unit. The task configuration includes the synchronization strategy, data source information, target database information, the library table to be synchronized, and the key parameters for the task unique identification. This configuration is recorded in the syncmgr task, and the syncmgr modifies the number of parallel reader / writer task instances according to the task progress. In addition to specific configuration parameters, all tasks also need to configure the driver type and resource restrictions of the task. After the configuration is determined, the Nomad job is started, and then the operation of each task is scheduled.

[0077] The deployment module is used for automated deployment and intelligent scheduling. It uses Nomad's automated deployment function to automatically create and start corresponding containers or virtual machine instances according to task configurations. Through Nomad's scheduling algorithm, it intelligently allocates virtual machine or container resources according to the cluster's resource usage and task requirements to ensure efficient execution of synchronization tasks. Nomad also supports dynamic migration and fault recovery of tasks to ensure continuous availability of services. Nomad supports multiple scheduling strategies, including least used, best location, and random, to help tasks run on the best nodes in the cluster, thereby optimizing resource utilization and performance. It sets the reader to be deployed on a host close to the source database, and sets the writer to be deployed on a host close to the destination database. In the synchronization of library table structures and full data, multiple reader and writer task instances are started to maximize resource usage with concurrent efficiency.

[0078] The service discovery module is used for service discovery and load balancing. It automatically registers and discovers database synchronization service instances through Nomad's service discovery mechanism. When new tasks are added or existing tasks exit, Nomad will automatically update the service list and distribute requests to appropriate service instances according to the load balancing strategy to achieve automatic expansion and failover of services. At different stages of task operation, the Reader / writer configuration is changed to one or more instances, and Nomad will automatically expand or shrink the actual running task instances.

[0079] The monitoring module is used for monitoring and alarming, and monitors the execution status of database synchronization tasks, resource usage, and cluster health in real time; when an abnormality or failure occurs, the system will automatically trigger an alarm notification and take corresponding processing measures to ensure the stable operation of the service.

[0080] Nomad provides an API to obtain the running status of jobs and tasks, and can also send monitoring and alarm information to Prometheus, etc. Syncmgr can obtain data from the monitoring system to generate job status, such as the progress of collection and storage, the amount of data cached by MQ Cluster, the amount of data collected or stored, etc., and provide it to the outside world in the form of API.

[0081] The security module is used for security assurance. It reduces security risks between different tasks by isolating container or virtual machine environments and enhances data isolation between different jobs through network layer policies.

[0082] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A database synchronization method supporting virtual machine and container deployment, characterized in that: The method comprises the following steps: Architecture design, using Nomad as the core scheduler, responsible for scheduling and managing database synchronization tasks; Task definition and configuration: By defining Nomad task specifications, database synchronization tasks are abstracted into configurable service units; Automated deployment and intelligent scheduling: Use Nomad's automated deployment function to automatically create and start corresponding containers or virtual machine instances based on task configurations. Service discovery and load balancing: Automatically register and discover database synchronization service instances through Nomad’s service discovery mechanism; Monitoring and alarming: real-time monitoring of the execution status of database synchronization tasks, resource usage, and cluster health; Security assurance: By isolating container or virtual machine environments, security risks between different tasks are reduced; data isolation between different jobs is enhanced through network layer policies.

2. According to a database synchronization method supporting virtual machine and container deployment according to claim 1, it is characterized in that: The specific operations of architecture design include: Use containerization or virtualization technology to encapsulate database synchronization tools and their dependent environments to form portable deployment units. Through Nomad's client and server architecture, dynamic task allocation and unified management of cluster resources are achieved. Nomad supports multiple types of drivers, through which tasks can be run and scheduled on different platforms. It supports running tasks in Docker containers, running binary files or scripts directly on the host, and running Java applications. It can choose the most appropriate deployment method to implement tasks without limiting the implementation and deployment methods of application services. A Nomad cluster consists of a server and one or more client hosts. It runs tasks according to policy scheduling within the entire cluster and exits after the function is completed.

3. According to a database synchronization method supporting virtual machine and container deployment according to claim 1, it is characterized in that: The specific operations of task definition and configuration include: Task configuration includes synchronization strategy, data source information, target database information, required synchronized library tables, and key parameters for task unique identification. This configuration is recorded in the syncmgr task, which modifies the number of parallel reader / writer task instances based on the task progress. In addition to specific configuration parameters, all tasks also need to configure the driver type and resource restrictions of the task. After the configuration is determined, the Nomad job is started, and then the running of each task is scheduled.

4. According to claim 1, a database synchronization method supporting virtual machine and container deployment is characterized in that: The specific operations of automated deployment and intelligent scheduling include: Through Nomad's scheduling algorithm, virtual machine or container resources are intelligently allocated according to the cluster's resource usage and task requirements to ensure efficient execution of synchronization tasks. Nomad also supports dynamic migration and fault recovery of tasks to ensure continuous availability of services. Nomad supports multiple scheduling strategies, including least used, best location, and random, to help tasks run on the best nodes in the cluster, thereby optimizing resource utilization and performance. The reader is deployed on a host close to the source database, and the writer is deployed on a host close to the destination database. Multiple reader and writer task instances are started during the synchronization of library table structures and full data to maximize resource usage with concurrent efficiency.

5. A database synchronization method supporting virtual machine and container deployment according to claim 1, characterized in that: The specific operations of service discovery and load balancing include: When new tasks are added or existing tasks exit, Nomad will automatically update the service list and distribute requests to appropriate service instances based on the load balancing strategy to achieve automatic expansion and failover of services; at different stages of task operation, change the Reader / writer configuration to one or more instances, and Nomad will automatically expand or shrink the actual running task instances.

6. A database synchronization system supporting virtual machine and container deployment according to the database synchronization method supporting virtual machine and container deployment according to any one of claims 1 to 5, characterized in that: The system consists of an architecture design module, a task configuration module, a deployment module, a service discovery module, a monitoring module and a security module; The architecture design module uses Nomad as the core scheduler, responsible for the scheduling and management of database synchronization tasks; The task configuration module is used for task definition and configuration. By defining the Nomad task specification, the database synchronization task is abstracted into a configurable service unit. The deployment module is used for automated deployment and intelligent scheduling. It uses Nomad's automated deployment function to automatically create and start corresponding containers or virtual machine instances according to task configurations. The service discovery module is used for service discovery and load balancing. It automatically registers and discovers database synchronization service instances through Nomad's service discovery mechanism. The monitoring module is used for monitoring and alarming, and real-time monitoring of the execution status of database synchronization tasks, resource usage, and cluster health status; The security module is used for security assurance. It reduces security risks between different tasks by isolating container or virtual machine environments and enhances data isolation between different jobs through network layer policies.

7. A database synchronization system supporting virtual machine and container deployment according to claim 6, characterized in that: The architecture design module uses containerization or virtualization technology to encapsulate the database synchronization tool and its dependent environment to form a portable deployment unit, and realizes dynamic allocation of tasks and unified management of cluster resources through Nomad's client and server architecture; Nomad supports multiple types of drivers, through which tasks can be run and scheduled on different platforms. It supports running tasks in Docker containers, running binary files or scripts directly on the host, and running Java applications. It can choose the most appropriate deployment method to implement tasks without limiting the implementation and deployment methods of application services. A Nomad cluster consists of a server and one or more client hosts. It runs tasks according to policy scheduling within the entire cluster and exits after the function is completed.

8. A database synchronization system supporting virtual machine and container deployment according to claim 6, characterized in that: The task configuration module includes synchronization strategy, data source information, target database information, required synchronized library tables, and key parameters for task unique identification. This configuration is recorded in the syncmgr task, and the syncmgr modifies the number of parallel reader / writer task instances according to the task progress. In addition to specific configuration parameters, all tasks also need to configure the driver type and resource restrictions of the task. After the configuration is determined, the Nomad job is started, and then the operation of each task is scheduled.

9. A database synchronization system supporting virtual machine and container deployment according to claim 6, characterized in that: The deployment module uses Nomad's scheduling algorithm to intelligently allocate virtual machine or container resources according to the resource usage and task requirements of the cluster to ensure efficient execution of synchronization tasks. Nomad also supports dynamic migration and fault recovery of tasks to ensure continuous availability of services. Nomad supports multiple scheduling strategies, including least used, best location, and random, to help tasks run on the best nodes in the cluster, thereby optimizing resource utilization and performance. The reader is deployed on a host close to the source database, and the writer is deployed on a host close to the destination database. Multiple reader and writer task instances are started during the synchronization of library table structure and full data to maximize resource utilization with concurrent efficiency.

10. A database synchronization system supporting virtual machine and container deployment according to claim 6, characterized in that: In the service discovery module, when a new task is added or an existing task is exited, Nomad will automatically update the service list and distribute the request to the appropriate service instance according to the load balancing strategy to achieve automatic expansion and failover of the service; at different stages of task operation, the Reader / writer configuration is changed to one or more instances, and Nomad will automatically expand or shrink the actual running task instance.

Citation Information

Patent Citations

  • Distributed cluster fault-tolerant recovery method and system used under virtualization platform

    CN115202917A

  • Job flow state control method and device, electronic equipment and storage medium

    CN117785399A

  • Improved method and system for container state synchronization on Kubernetes virtual node

    CN117850971A

  • Method for monitoring configuration synchronization between container clusters

    CN118018559A

  • System and method for dynamic distribution of data and / or services

    CN1520671A

Cited By

  • Data synchronization method, device, equipment, medium and program product

    CN122285185A