DataX distributed data synchronization method and device based on Kubernetes and electronic equipment

Abstract

The invention relates to a DataX distributed data synchronization method and a DataX distributed data synchronization device based on Kubernetes. The method comprises the steps of splitting a data synchronization task into a plurality of subtasks according to a data range, starting a plurality of Pods for parallel execution through Kubernetes Job, and starting a plurality of Task tasks in each Pod through channel parameters of DataX for concurrent processing of subtask data; splitting or merging of the fragments is dynamically triggered based on business indexes such as a line processing rate and system indexes such as a CPU (Central Processing Unit) and a memory utilization rate, fragment state information is recorded through ConfigMap, and the number of the fragments and resource allocation are dynamically adjusted; and combining the DataX execution result indexes of each Pod to ensure the data consistency. Through a two-stage fragmentation parallelization architecture, multi-level parallel processing is realized, the resource utilization rate is maximized, and the data synchronization efficiency is remarkably improved; the dynamic elastic capacity expansion and contraction mechanism can dynamically adjust resource allocation according to service load, optimize resource utilization and avoid waste or performance bottleneck; automatic fragmentation management does not need manual intervention, and the operation efficiency and reliability of the system are improved.

Description

Technical Field

[0001] The present application belongs to the field of big data synchronization technology, and in particular relates to a Kubernetes-based DataX distributed data synchronization method, device, computer-readable storage medium, and electronic device. The present invention is suitable for synchronizing heterogeneous data sources in high-throughput, high-concurrency scenarios. Background Art

[0002] In existing big data synchronization scenarios, data synchronization tools (such as DataX) typically run in standalone mode, which presents the following key issues: 1. Resource bottleneck Memory and CPU limitations: When processing large amounts of data, the standalone version of DataX often fails due to insufficient memory or limited CPU power.

[0003] Insufficient network throughput: In high-concurrency scenarios, the network bandwidth of a single node becomes a performance bottleneck and cannot meet the needs of TB-level data synchronization.

[0004] 2. Lack of Task Sharding Strategy Although existing technologies (such as patent application CN111258742A) schedule DataX containers through Kubernetes, they do not involve task sharding strategies and cannot fully utilize the parallel capabilities of Pods and the concurrent capabilities of DataX's own multiple tasks, resulting in low parallel efficiency.

[0005] (3) Lack of flexible scheduling Although existing technologies (such as patent application CN111258742A) schedule DataX containers through Kubernetes, they lack a dynamic load adjustment mechanism and cannot dynamically adjust the number of shards based on the load, resulting in resource waste or insufficient performance.

[0006] (4) Lack of fault tolerance and recovery mechanisms Traditional solutions do not record synchronization progress, and if a task fails, it needs to be fully retried, which is time-consuming and resource-intensive.

[0007] (V) Insufficient awareness of business indicators Existing technologies scale capacity based on system metrics such as CPU and memory, but are unable to perceive business needs (such as row processing rate and data throughput), resulting in inaccurate scheduling.

[0008] Chinese patent application CN111258742A proposes a "Kubernetes-based DataX scheduling method." While this method implements containerized deployment, it has the following limitations: (1) Static resource allocation: The number of shards and resource quotas are fixed and lack flexibility.

[0009] (2) No two-level parallelism: Pod-level sharding is not utilized, and DataX's task-level concurrency is not combined.

[0010] (3) High demand for manual intervention: Manual monitoring of load and adjustment of sharding are required, and the degree of automation is low. Summary of the Invention

[0011] To address the above problems, this application proposes a new Kubernetes-based DataX distributed data synchronization method and device, aiming to solve the problems of the existing DataX distributed data synchronization method, such as lack of parallel mechanism, lack of dynamic sharding and elastic scaling, and inability to intelligently recover from faults.

[0012] The present invention mainly adopts the following technical strategies: 1. Two-level sharding parallel architecture Pod-level sharding (first-level sharding): Kubernetes Jobs are used to split the large table data of DataX tasks into multiple independent subtasks based on data ranges (such as table partitions or primary key intervals). These subtasks are executed in parallel by different Pods, thus overcoming the resource limitations of a single machine.

[0013] Task-level sharding (second-level sharding): Multiple Tasks are started within the Pod through the DataX channel parameter to concurrently process sub-shard data, fully utilizing the computing resources of a single Pod.

[0014] Innovation: This invention combines the distributed scheduling capabilities of Kubernetes with the fine-grained concurrency capabilities of DataX for the first time, realizing a dual parallelization architecture of "cluster level + process level".

[0015] 2. Dynamic elastic expansion and contraction mechanism Business indicator driven: Based on custom business indicators such as row processing rate (datax_rows_per_second), combined with CPU and memory usage, dynamically trigger the splitting (expansion) or merging (reduction) of shards.

[0016] State-aware sharding adjustment: Sharding status (such as last_processed_id) is recorded in real time through Kubernetes ConfigMap, supporting breakpoint resuming and load balancing, avoiding resource waste or performance bottlenecks caused by static sharding.

[0017] Dynamic adjustment: Monitor the ConfigMap status, automatically trigger expansion and contraction based on indicators, reallocate shard ranges, and adjust the number of Pods.

[0018] 3. Automated Sharding Management Dynamic configuration generation: Automatically calculate shard boundaries based on table partitions or primary key ranges, generate Kubernetes ConfigMap, and inject it into Pods, decoupling configuration from images.

[0019] Dynamic Sharding: Through an elastic scaling mechanism driven by business metrics, the number of DataX shards and resource allocation are dynamically adjusted to ensure efficient and flexible resource utilization.

[0020] 4. Result merging Summarizes the DataX execution metrics (number of success / failure records) of each Pod to ensure data consistency.

[0021] Through the above-mentioned two-level parallelization architecture, dynamic elastic expansion and contraction mechanism, and automated sharding management, the present invention solves the problems of existing data synchronization tools in large-scale data processing scenarios, such as resource bottlenecks, low parallel efficiency, lack of elastic scheduling and fault-tolerant recovery capabilities.

[0022] In order to achieve the above objectives, this application provides the following technical solutions: A first aspect of the present application provides a Kubernetes-based DataX distributed data synchronization method, the method comprising: Split the data synchronization task into multiple subtasks based on the data range, start a Pod for each subtask through Kubernetes Job, and execute the subtasks in parallel; Within each Pod, multiple Tasks are started using the DataX channel parameter to concurrently process subtask data. Dynamically trigger the splitting or merging of shards based on business and system indicators; Record shard status information in real time through Kubernetes ConfigMap; Dynamically adjust the number of shards and resource allocation based on shard status information, including reallocating shard ranges and adjusting the number of Pods; Merge the DataX execution result indicators of each Pod to ensure data consistency.

[0023] Furthermore, in the method of the present application, the data synchronization task is split into multiple subtasks according to the data range, including the following methods: Split by table partition: Automatically read the partition definition of the source table and generate an independent sharding task for each partition; Split by primary key range: Obtain the primary key range through SQL query and divide the range into intervals according to the number of shards.

[0024] Furthermore, in the method of the present application, the business indicators include row processing rate, and the system indicators include CPU and memory usage.

[0025] Furthermore, in the method of the present application, the shard status information includes: the record ID of the shard that was last successfully processed, the shard status, the number of records that the shard has completed processing, the total number of records in the shard, and the data range originally allocated to the shard.

[0026] Furthermore, the method of the present application also includes: automatically calculating the shard boundaries based on the table partition or primary key range, creating an independent Kubernetes ConfigMap for each shard and injecting it into its corresponding Pod, storing DataX task configuration information, and realizing the decoupling of configuration and mirroring.

[0027] Furthermore, in the method of the present application, the dynamically adjusting the number of shards and resource allocation according to the shard status information includes: Capacity expansion: Calculate the number of new shards required based on the amount of unfinished data, reallocate the unfinished data range to the new shards, and start new Pods; Scaling: Merge small shards into large shards, redefine the shard range, and delete redundant Pods.

[0028] Furthermore, in the present application method, The triggering conditions for the capacity expansion process include: (1) The shard processing rate is lower than the preset threshold; (2) The CPU utilization is higher than the preset threshold or the memory usage is higher than the preset threshold; The triggering conditions for the shrinking process include: (1) The average row processing rate of all shards is higher than the preset threshold; (2) The CPU utilization is lower than the preset threshold and the memory usage is lower than the preset threshold.

[0029] Furthermore, in the present application method, (1) The steps of the capacity expansion process include: a) The listener monitors the ConfigMap of the running DataX task, analyzes the status of each shard and the resource usage of the Pod in real time, and triggers capacity expansion when the shard processing rate falls below the threshold or the CPU or memory is overloaded; b) Obtain the status information of each shard from the ConfigMap, calculate the number of new shards required based on the amount of unfinished data, and reallocate the unfinished data range to the new shards; c) Recreate the ConfigMap corresponding to each shard to store the DataX task configuration; d) Use Kubernetes Job to start a new Pod for each shard and mount the corresponding ConfigMap; (2) The steps of the shrinking process include: a) The listener monitors the ConfigMap of the running DataX task, analyzes the status of each shard and the resource usage of the Pod in real time, and triggers scaling down when the processing rate of all shards exceeds the threshold and CPU and memory resources are idle; b) Obtain the status information of each shard from the ConfigMap, merge small shards into large shards, and redefine the shard range; c) Update the new shard range to the ConfigMap storage; d) Delete the extra Pods.

[0030] A second aspect of the present application provides a Kubernetes-based DataX distributed data synchronization device, the device comprising: The first-level task splitting module is used to split the data synchronization task into multiple subtasks according to the data range. A Pod is started for each subtask through a Kubernetes Job to execute the subtasks in parallel. The second-level task splitting module is used to start multiple tasks within each pod using the DataX channel parameter to concurrently process subtask data. Shard adjustment trigger module: used to dynamically trigger the splitting or merging of shards based on business indicators and system indicators; Sharding information collection module: used to record sharding status information in real time through Kubernetes ConfigMap; Shard dynamic adjustment module: used to dynamically adjust the number of shards and resource allocation based on shard status information, including reallocating shard ranges and adjusting the number of Pods; Result merging module: used to merge the DataX execution result indicators of each Pod to ensure data consistency.

[0031] The device implements the steps of the aforementioned Kubernetes-based DataX distributed data synchronization method during runtime.

[0032] A third aspect of the present application provides an electronic device, comprising: a memory and a processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the aforementioned Kubernetes-based DataX distributed data synchronization method.

[0033] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the aforementioned Kubernetes-based DataX distributed data synchronization method are implemented.

[0034] In summary, the Kubernetes-based DataX distributed data synchronization method proposed in this application has the following advantages: (1) Performance improvement: Through the two-level data sharding strategy (Pod-level sharding and Task-level sharding), multi-level parallel processing of data synchronization tasks is achieved, which can maximize resource utilization and significantly improve the efficiency of data synchronization.

[0035] (2) Resource optimization: By combining business indicators (such as row processing rate) with system indicators (such as CPU and memory usage), dynamic sharding and on-demand scaling are achieved, significantly improving resource utilization and avoiding resource waste or performance bottlenecks.

[0036] (3) Fully automatic scheduling: Triggering expansion and contraction and dynamic sharding operations based on business indicators without manual intervention, achieving automated scheduling and improving system operation efficiency and reliability.

[0037] Other features and advantages of this application will be described in detail in the following description, or will be understood through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and technical means clearly indicated in the description, claims, and drawings, and obtained through the implementation of these technical contents. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] To more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings involved in the description of the embodiments. It should be noted that the drawings only illustrate some of the embodiments of the present application. For those skilled in the art, other relevant drawings can be derived from these drawings without engaging in creative work.

[0039] Figure 1 This is the overall implementation flowchart of the DataX distributed data synchronization method based on Kubernetes in this application.

[0040] Figure 2 This is a flowchart for implementing the two-level data sharding strategy in this application method.

[0041] Figure 3 Schematic diagram of data range division for the two-level data sharding strategy in this application method.

[0042] Figure 4 This is a flowchart for implementing the dynamic sharding strategy in this application method.

[0043] Figure 5 This is a structural diagram of the DataX distributed data synchronization device based on Kubernetes in this application.

[0044] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] In this document, the term "including" and any variations thereof (such as "including," "comprising," etc.) are open-ended expressions and should be understood as meaning "including but not limited to," meaning that the listed contents are not exhaustive and may include other contents not explicitly mentioned. The term "based on" should be understood as meaning "based at least in part on," meaning that the basis or condition referred to may not be the only factor and may also involve other relevant factors. The term "one embodiment" should be understood as meaning "at least one embodiment," meaning that the described embodiment is not the only possible implementation method and that other similar embodiments may exist.

[0047] In this application, the terms "a" and "a plurality" are used to modify related elements or features in an illustrative, non-restrictive manner. Unless the context clearly indicates otherwise, "a" should be understood as meaning "at least one," and "a plurality" should be understood as meaning "at least two." Those skilled in the art should interpret these terms appropriately based on the semantics and logical relationships of the context to ensure that they encompass the possibility of "one or more."

[0048] Figure 1 The following is the overall implementation process of the DataX distributed data synchronization method based on Kubernetes provided by this application, including the following steps: S1. Split the data synchronization task into multiple subtasks based on the data range. Use a Kubernetes Job to launch a Pod for each subtask and execute the subtasks in parallel. S2. Within each pod, use the DataX channel parameter to start multiple tasks to concurrently process subtask data. S3. Dynamically trigger shard splitting or merging based on business and system metrics. S4. Record shard status information in real time through Kubernetes ConfigMap; S5. Dynamically adjust the number of shards and resource allocation based on shard status information, including reallocating shard ranges and adjusting the number of pods. S6. Merge the DataX execution result indicators of each pod to ensure data consistency.

[0049] In order to more clearly illustrate the technical solution of the present application, the following will further illustrate it through embodiments of specific scenarios.

[0050] The Kubernetes-based DataX distributed data synchronization method provided in this application includes the following steps: (1) Task sharding steps In order to maximize resource utilization and improve the efficiency of data synchronization, the present invention designs a two-level data sharding strategy to achieve multi-level parallel processing of data synchronization tasks.

[0051] The goal of this solution is to split large table data into multiple shards based on partitions or primary key ranges (intervals). For the first-level shards, multiple pods are generated and executed in parallel using a Kubernetes job. For the second-level shards, multiple tasks are started and executed concurrently in each pod using the DataX channel parameter, and the results are finally merged.

[0052] The design plan is: First-level sharding (Pod level): Split the DataX task into multiple subtasks based on data range (such as table partition / primary key interval). Each subtask is executed in parallel by an independent Kubernetes Pod.

[0053] Second-level sharding (Task level): Within each Pod, DataX starts multiple concurrent Tasks through the channel parameter to further process sharded data in parallel.

[0054] like Figure 2 As shown in the figure, the overall process of the two-level data sharding strategy in this solution includes: S11. Data range division; S12. Configure DataX concurrency parameters. S13. Generate shard configuration; S14. Start the sharded Pod. S15. Dynamic Task Allocation S16. Generate Task configuration.

[0055] The specific implementation steps are as follows: 1. First-level sharding (Pod-level sharding) Goal: Split DataX tasks into multiple subtasks based on data range, with each subtask executed by an independent Kubernetes Pod.

[0056] (1) Data range division The shard boundaries are calculated based on the source table primary key or partitioning rules (such as time range or hash value). The initial number of shards can be set (the number of shards can be adjusted dynamically based on the load later).

[0057] a) Split by table partition: Automatically reads the partition definition of the source table (such as MySQL's PARTITION BY RANGE) and generates an independent sharding task for each partition.

[0058] Example: The orders table is partitioned by date (p202401, p202402), and a sharding task corresponds to each partition.

[0059] b) Split by primary key range: Obtain the primary key range through SQL queries (such as SELECT MIN(id), MAX(id) FROM tables) and divide the range by the number of shards.

[0060] Example: The orders table (100 million rows of data) is divided into four shards by primary key ID: Shard 1: ID 1-25 million; Shard 2: ID 25 million to 50 million; Shard 3: ID 50 million to 75 million; Shard 4: ID 75 million to 100 million.

[0061] (2) Generate shard configuration A separate Kubernetes ConfigMap is created for each shard to store DataX task configuration information. ConfigMap is an API object in Kubernetes for storing non-sensitive configuration data. It stores key-value pairs, decoupling application configuration from container images. It supports injection into pods via environment variables, command-line parameters, or file mounts. In this paper, ConfigMap uses file mounts to inject DataX task configuration information into pods.

[0062] The ConfigMap yaml file example corresponding to shard 1 is as follows:

[0063] (3) Start the sharded Pod Use Kubernetes Job to start a Pod for each shard, mount the corresponding ConfigMap, and execute multiple Pods in parallel.

[0064] The following is an example of the Job YAML file corresponding to shard 1:

[0065] 2. Second-level sharding (Task-level sharding) Goal: Within each Pod, further process sharded data in parallel through DataX's concurrent Task subtasks.

[0066] A DataX Task is the smallest execution unit of a DataX job and is responsible for actual data synchronization. Each Task is dynamically generated by a DataX Job based on the source data's sharding strategy to enable concurrent synchronization.

[0067] (1) Configuring DataX concurrency parameters Set the channel parameter (controls the number of concurrent tasks) in the DataX JSON configuration. For example, when the channel parameter is set to 4, DataX will generate four Task subtasks to execute synchronization tasks concurrently.

[0068] The following is an example of configuring the DataX channel parameters:

[0069] In the present invention, when the channel is set to 4, it means that 4 concurrent tasks will be started in the corresponding Pod to perform the synchronization task.

[0070] (2) Dynamic Task Allocation like Figure 3 As shown in the figure, each Task is dynamically divided into sub-ranges to process sharded data according to the set channel parameters.

[0071] For example, if the channel parameter is 4, shard 1 (ID 1-25 million) will be further split into: Task 1: The ID range is 1-6.25 million; Task 2: The ID range is 6.25 million to 12.5 million; Task 3: The ID range is 12.5 million to 18.75 million; Task 4: The ID range is 18.75 million to 25 million.

[0072] Use DataX's querySql to divide the data range of each task, thereby achieving task-level sharding.

[0073] The configuration template for each Task is:

[0074] Among them, ${start} and ${end} represent the starting value of the data range of each interval respectively.

[0075] For example, the DataX configuration of Task 1 is:

[0076] 3. Merge the results After all sharded Pods have finished running, the program summarizes the DataX running result indicators in each pod, including the number of records successfully read, the number of records successfully written, and the number of failed records, to ensure data consistency.

[0077] In summary, the two-level data sharding strategy of this invention combines Kubernetes Pod-level sharding and DataX Task-level sharding to achieve multi-level parallel processing of data synchronization tasks. The specific effects are as follows: Pod-level sharding: Split a large data table into multiple independent subtasks, which are executed in parallel by different Kubernetes Pods, breaking through the resource limitations of a single machine.

[0078] Task-level sharding: Within each pod, DataX further processes data in parallel through multi-channel concurrent tasks, fully utilizing the computing resources of a single pod.

[0079] (2) Dynamic sharding steps 1. Define the business indicator design of DataX tasks Task-level indicators (business indicators): datax_rows_per_second: The number of rows synchronized per second (collected through DataX logs).

[0080] System-level indicators (K8s native indicators): (1) cpu_usage: CPU utilization of the Pod.

[0081] (2) memory_usage: Pod’s memory usage.

[0082] Expansion trigger conditions (triggered if any of the conditions are met): (1) Business indicator: The shard processing rate (datax_rows_per_second) is lower than the threshold (such as 5000 rows / second).

[0083] (2) System indicators: CPU utilization > 70% or memory utilization > 80%.

[0084] Scaling trigger conditions (must be met simultaneously): (1) Business indicators: The average value of datax_rows_per_second for all shards is > 10,000 rows / second.

[0085] (2) System indicators: CPU utilization <40% and memory utilization <50%.

[0086] 2. Shard state storage design In the present invention, ConfigMap is used to store the progress status information of each shard. The progress status information includes:

[0087] The content of the ConfigMap that stores progress status information is as follows:

[0088] # Global shard status

[0089] # Detailed status of each shard

[0090] 3. Sharding status dynamic update process Add shard status reporting logic to the DataX write plug-in. After the write plug-in successfully writes a batch of data, it reads the latest status information of each shard in the ConfigMap in real time, and then updates the last_processed_id (the record ID of the last successfully processed shard) of each shard in the ConfigMap.

[0091] 4. Dynamically generate new shard configuration This solution uses an elastic scaling mechanism driven by business indicators to dynamically adjust the number of DataX shards and resource allocation, solving the resource waste or performance issues caused by static sharding.

[0092] Objective: Use a listener to monitor the ConfigMap of a running DataX task and trigger elastic scaling when business metrics meet the scaling conditions.

[0093] like Figure 4 As shown in the figure, the specific implementation process includes: (1) Steps to implement capacity expansion: a) The listener monitors the ConfigMap of the running DataX task, analyzes the status of each shard and the resource usage of the Pod in real time, and triggers capacity expansion when the shard processing rate falls below the threshold or the CPU and memory are overloaded; b) Obtain the status information of each shard from the ConfigMap, calculate the number of new shards required based on the amount of unfinished data (based on a rate of 5000 rows / second), and reallocate the unfinished data range to the new shards; c) Recreate the ConfigMap corresponding to each shard to store the DataX task configuration; d) Use Kubernetes Job to start a new Pod for each shard and mount the corresponding ConfigMap.

[0094] During dynamic expansion, the system throughput and overall processing capacity are improved by increasing the number of Pods.

[0095] (2) Steps to implement scaling: a) The listener monitors the ConfigMap of the running DataX task, analyzes the status of each shard and the resource usage of the pod in real time, and triggers scaling down when all shards have high processing rates and idle resources. b) Obtain the status information of each shard from the ConfigMap, merge small shards into large shards, and redefine the shard range; c) Update the new shard range to the ConfigMap storage; d) Delete the extra Pods.

[0096] During dynamic scaling, the main purpose of merging small shards is to optimize resource utilization and improve task execution efficiency. Specifically, it solves the following problems: Resource fragmentation: Too many small shards lead to redundant Pods and increase Kubernetes scheduling overhead.

[0097] Uneven load: Some shards are completed prematurely due to small data volumes, while other shards are still being processed, resulting in idle resources.

[0098] Performance bottleneck: Frequent starting and stopping of small-shard tasks will introduce additional delays (such as container cold start time).

[0099] In summary, the dynamic sharding solution of the present invention solves the limitations of static sharding in the prior art. The specific effects are as follows: (1) Dynamic load balancing When some shards have much larger data volumes than other shards (for example, shard A processes 1 million rows and shard B processes 100,000 rows), large shards are dynamically split or small shards are merged to avoid overloading or idling some Pods.

[0100] (2) Elastic scalability It supports dynamic adjustment of the number of shards and resource allocation based on business indicators (such as row processing rate) to adapt to high throughput and large concurrency requirements.

[0101] Figure 5 The figure shows a Kubernetes-based DataX distributed data synchronization device proposed in this application, including: The first-level task splitting module is used to split the data synchronization task into multiple subtasks according to the data range. A Pod is started for each subtask through a Kubernetes Job to execute the subtasks in parallel. The second-level task splitting module is used to start multiple tasks within each pod using the DataX channel parameter to concurrently process subtask data. Shard adjustment trigger module: used to dynamically trigger the splitting or merging of shards based on business indicators and system indicators; Sharding information collection module: used to record sharding status information in real time through Kubernetes ConfigMap; Shard dynamic adjustment module: used to dynamically adjust the number of shards and resource allocation based on shard status information, including reallocating shard ranges and adjusting the number of Pods; Result merging module: used to merge the DataX execution result indicators of each Pod to ensure data consistency.

[0102] When the above device is running, the steps of the DataX distributed data synchronization method based on Kubernetes disclosed in this application are implemented.

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of the apparatus, methods, and computer program products according to various embodiments of the present application, including architecture, functions, and operations. In these figures, each box may represent a module, a program segment, or a portion of a code, which contains one or more executable instructions for implementing a specified logical function. It should be noted that each box in the block diagram and / or flowchart, and the combination of these boxes, can be implemented using a dedicated hardware-based system to implement the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0104] like Figure 6As shown, an embodiment of the present application further discloses an electronic device, comprising: a processor 310, a communication interface 320, a memory 330 for storing a computer program executable by the processor, and a communication bus 340. The processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the above-mentioned Kubernetes-based DataX distributed data synchronization method.

[0105] It is understood that, in addition to the memory and processor, the electronic device may also include an input device (e.g., a keyboard), an output device (e.g., a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor via an I / O interface (i.e., an input / output interface).

[0106] The operation of the present application can be implemented by writing computer program code using one or more programming languages or a combination thereof. The programming languages include but are not limited to the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc.; A conventional procedural programming language, such as "C" or a similar programming language.

[0107] The execution methods of the program code include but are not limited to: Executes entirely on the user's computer; Partially executed on the user's computer and partially on a remote computer; Executed as a standalone software package; Executes entirely on the remote computer or server.

[0108] In scenarios involving a remote computer, the remote computer can be connected to the user's computer via any type of network, including but not limited to a local area network (LAN) or a wide area network (WAN). Additionally, the remote computer can be connected to an external computer via an Internet service provider, such as the Internet.

[0109] Furthermore, the present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by the processor of an electronic device, the electronic device is enabled to execute the various steps of the Kubernetes-based DataX distributed data synchronization method disclosed in the present application.

[0110] In the context of this application, computer-readable storage media refers to tangible media that can store computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: A removable magnetic storage medium such as a floppy disk.

[0111] (2) Hard disk: includes fixed storage devices such as mechanical hard disks and solid-state hard disks.

[0112] (3) Random Access Memory (RAM): Volatile storage medium used for temporary storage of data and program code.

[0113] (4) Read-only memory (ROM): A non-volatile storage medium used to store fixed programs and data.

[0114] (5) Erasable Programmable Read-Only Memory (EPROM) or Flash Memory: A non-volatile storage medium that supports multiple erasing and programming.

[0115] (6) Fiber optic storage device: storage medium based on fiber optic technology.

[0116] (7) Compact disk read-only memory (CD-ROM): A read-only medium that stores data in the form of an optical disc.

[0117] (8) Optical storage devices: storage media based on optical principles, such as DVDs and Blu-ray discs.

[0118] (9) Magnetic storage devices: storage media based on magnetic principles, such as magnetic tapes and disks.

[0119] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage requirements.

[0120] These computer-readable storage media can be used to store the program code and related data described in this application to support the operation of the program and the persistent storage of data.

[0121] In particular, according to an embodiment of the present application, the process described in the flowchart can be implemented as a computer software program. For example, an embodiment of the present application relates to a computer program product, which includes a computer program carried on a non-transitory computer-readable medium. The computer program includes program code for executing the Kubernetes-based DataX distributed data synchronization method disclosed in this application. When the computer program is executed by a processing device, it can implement the above-mentioned functions defined in the embodiments of the present application.

[0122] Although the above discussion contains several specific implementation details, these details should not be interpreted as limiting the scope of this application. The above description is only a preferred embodiment of the present application and an illustration of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in this application is not limited to the technical solutions formed by the specific combination of the above technical features. At the same time, this application should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concepts.

[0123] Those skilled in the art should also understand that they may modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents, without departing from the spirit and scope of the technical solutions of the embodiments of the present application. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A DataX distributed data synchronization method based on Kubernetes, characterized in that: The method comprises: Split the data synchronization task into multiple subtasks based on the data range, start a Pod for each subtask through Kubernetes Job, and execute the subtasks in parallel; Within each Pod, multiple Tasks are started using the DataX channel parameter to concurrently process subtask data. Dynamically trigger the splitting or merging of shards based on business and system indicators; Record shard status information in real time through Kubernetes ConfigMap; Dynamically adjust the number of shards and resource allocation based on shard status information, including reallocating shard ranges and adjusting the number of Pods; Merge the DataX execution result indicators of each Pod to ensure data consistency.

2. The method according to claim 1, characterized in that The data synchronization task is split into multiple subtasks according to the data range, including the following methods: Split by table partition: Automatically read the partition definition of the source table and generate an independent sharding task for each partition; Split by primary key range: Obtain the primary key range through SQL query and divide the range into intervals according to the number of shards.

3. The method according to claim 1, characterized in that The business indicators include row processing rate, and the system indicators include CPU and memory usage.

4. The method according to claim 1, wherein The shard status information includes: the record ID of the last successfully processed shard, the shard status, the number of records that have been processed by the shard, the total number of records in the shard, and the data range originally allocated to the shard.

5. The method according to claim 1, wherein The method also includes: automatically calculating shard boundaries based on table partitions or primary key ranges, creating an independent Kubernetes ConfigMap for each shard and injecting it into its corresponding Pod, storing DataX task configuration information, and achieving decoupling of configuration and image.

6. The method according to claim 1, characterized in that The dynamically adjusting the number of shards and resource allocation according to the shard status information includes: Capacity expansion: Calculate the number of new shards required based on the amount of unfinished data, reallocate the unfinished data range to the new shards, and start new Pods; Scaling: Merge small shards into large shards, redefine the shard range, and delete redundant Pods.

7. The method according to claim 6, characterized in that The triggering conditions for the capacity expansion process include: (1) The shard processing rate is lower than the preset threshold; (2) The CPU utilization is higher than the preset threshold or the memory usage is higher than the preset threshold; The triggering conditions for the shrinking process include: (1) The average row processing rate of all shards is higher than the preset threshold; (2) The CPU utilization is lower than the preset threshold and the memory usage is lower than the preset threshold.

8. The method according to claim 6, characterized in that (1) The steps of the capacity expansion process include: a) The listener monitors the ConfigMap of the running DataX task, analyzes the status of each shard and the resource usage of the Pod in real time, and triggers capacity expansion when the shard processing rate falls below the threshold or the CPU or memory is overloaded; b) Obtain the status information of each shard from the ConfigMap, calculate the number of new shards required based on the amount of unfinished data, and reallocate the unfinished data range to the new shards; c) Recreate the ConfigMap corresponding to each shard to store the DataX task configuration; d) Use Kubernetes Job to start a new Pod for each shard and mount the corresponding ConfigMap; (2) The steps of the shrinking process include: a) The listener monitors the ConfigMap of the running DataX task, analyzes the status of each shard and the resource usage of the Pod in real time, and triggers scaling down when the processing rate of all shards exceeds the threshold and CPU and memory resources are idle; b) Obtain the status information of each shard from the ConfigMap, merge small shards into large shards, and redefine the shard range; c) Update the new shard range to the ConfigMap storage; d) Delete the extra Pods.

9. A DataX distributed data synchronization device based on Kubernetes, characterized in that: include: The first-level task splitting module is used to split the data synchronization task into multiple subtasks according to the data range. A Pod is started for each subtask through a Kubernetes Job to execute the subtasks in parallel. The second-level task splitting module is used to start multiple tasks within each pod using the DataX channel parameter to concurrently process subtask data. Shard adjustment trigger module: used to dynamically trigger the splitting or merging of shards based on business indicators and system indicators; Sharding information collection module: used to record sharding status information in real time through Kubernetes ConfigMap; Shard dynamic adjustment module: used to dynamically adjust the number of shards and resource allocation based on shard status information, including reallocating shard ranges and adjusting the number of Pods; Result merging module: used to merge the DataX execution result indicators of each Pod to ensure data consistency.

10. An electronic device, characterized in that: include: memory and processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the Kubernetes-based DataX distributed data synchronization method as described in any one of claims 1-8.

Images