Operator-based cross-storage type data replication method and system

Abstract

The application provides an Operator-based cross-storage type data replication method and system, relates to the field of cloud-native data management, defines a data replication custom resource in a K8s cluster, adopts a custom resource+Op controller architecture, the Op controller listens to the data replication custom resource and executes reconciliation logic, creates an adaptive working Pod according to a source and target storage type combination, completes cross-storage type data replication through unified parameter configuration and execution strategy, and writes task progress, state and other information to a state field of the data replication resource in real time. The application realizes unified abstraction of multiple storage types and different combinations, deeply integrates with K8s life cycle, scheduling strategy and RBAC permission control, supports standardized state management, progress observation, cancellation and retry of tasks, does not need to modify the custom resource structure when extending a new storage type, and improves the automation, platformization and operation and maintenance capability of cross-storage data replication.

Description

Technical Field

[0001] This invention relates to the field of cloud-native data management technology, and more specifically, to a cross-storage type data replication method based on K8sOperator. Background Technology

[0002] With the popularization of cloud-native technologies and the widespread application of multi-cluster architectures, enterprise business data is increasingly distributed across various heterogeneous storage types. Typical storage types include Unified File System (UFS, usually mounted via Kubernetes persistent volume PVCs) and Object Storage Service (OSS, using an S3-compatible interface). In actual business operations, it is often necessary to perform data migration or synchronization operations between different storage types, such as cross-node data replication between UFS, cross-bucket data synchronization between OSS, and hybrid storage data migration between OSS and UFS. This is one of the main data management requirements in a cloud-native environment.

[0003] In existing technologies, data replication operations across storage types are mostly implemented by manually executing scripts or using independent migration tools. Specifically, operations personnel manually execute command-line tools such as rclone and rsync in the host or temporary container, or directly use the proprietary migration services provided by cloud vendors. Before performing the replication operation, operations personnel need to configure access methods, authentication credentials, path semantics, and other information for the source and target storage respectively. The entire operation process is highly dependent on manual intervention, and the data replication task is independent of the resource lifecycle and scheduling strategy of the Kubernetes cluster, and is not included in the unified orchestration and management system of the cluster.

[0004] The aforementioned existing technical solutions have obvious technical defects, specifically in the following three aspects:

[0005] 1. There are significant differences in the underlying access protocols, credential management methods, and path semantic rules of different storage types. Existing solutions cannot use the same set of interfaces and models to describe the attribute information of source and target storage, and lack unified storage abstraction capabilities. When it is necessary to extend to new storage types, the corresponding interface logic needs to be developed repeatedly, resulting in high development costs and poor scalability.

[0006] 2. Data replication tasks are not modeled as a standardized resource within the Kubernetes cluster, resulting in a severe disconnect from the Kubernetes orchestration system. This prevents the reuse of native Kubernetes mechanisms such as declarative APIs, schedulers, RBAC access control, and finalizers. The node affinity between tasks and storage cannot be uniformly scheduled. The cleanup of workloads, temporary containers, and other resources after a task ends relies on manual or external scripts, leading to a high risk of resource leakage. Furthermore, the observability of task status is poor, and operations personnel cannot obtain task progress in real time.

[0007] 3. The data replication task lacks standardized status definitions and unified query APIs for key information such as progress, transmission rate, and failure reasons. Operations such as task cancellation, retry, and restart lack standardized execution methods, making it impossible to integrate with the enterprise's cloud-native management platform. This results in insufficient maintainability and platformization capabilities, hindering the realization of automated and platform-based operation and maintenance management.

[0008] A deeper analysis reveals that the root cause of the aforementioned technical defects lies in the fact that existing solutions treat "data replication" as a one-time manual operation independent of Kubernetes cluster orchestration, failing to model the data replication task as a type of custom resource within the cluster. Therefore, they cannot leverage Kubernetes' native capabilities to achieve declarative management, automated scheduling, and full lifecycle control of the task. Summary of the Invention

[0009] Therefore, the purpose of this invention is to propose a cross-storage type data replication method and system based on Kubernetes Operator. It adopts a cloud-native architecture of "custom resource + Operator controller," modeling data replication tasks as Kubernetes custom resources. The Operator controller manages the entire lifecycle of data replication tasks, achieving unified abstraction for multiple storage types, deeply integrating native Kubernetes capabilities, and improving the automation and platformization of cross-storage data replication. It also achieves unified modeling and declarative management of multiple storage types, improving the automation, observability, and scalability of data replication tasks, reducing the operational costs of cross-storage data management in cloud-native environments, and is suitable for data migration and synchronization scenarios between unified file storage, object storage, and other storage types in cloud-native and multi-cluster environments.

[0010] This invention provides an operator-based method for cross-storage type data replication, comprising the following steps:

[0011] (1) Custom resource definition and instantiation: Create a custom resource definition (CRD) for data replication in the Kubernetes cluster, and instantiate the custom resource DataReplication CR based on the CRD. The DataReplication CR contains a specification field (Spec) and a status field (Status). The Spec field is used to uniformly describe the attribute information of the source storage and the target storage, realizing a unified abstraction of different storage types. The Status field is used to record the full lifecycle status information of the data replication task, including task status, replication progress, failure reasons, etc.

[0012] In the Spec field, both source and target contain a storage type field, and the value of type includes UFS (Unified File System) and OSS (Object Storage). When type is UFS, the Kubernetes persistent volume declaration PVC is referenced through the claimRef field to enable UFS volume mounting access. When type is OSS, the access information of OSS is described through the endpoint (S3 endpoint), bucket (bucket name), path (storage path), and credentialSecret (credential key) fields to enable standardized access to S3 compatible interfaces.

[0013] The Status field includes subfields such as UUID (unique task identifier), status (task status), progress (copy progress, percentage), reason (state reason description), and endTimestamp (task end timestamp). The status field has six possible values: Pending, InProgress, Succeeded, Failed, Cancelled, and Deleting. Each status transitions according to a preset state machine to achieve standardized management of task status.

[0014] Specifically, the data replication task is modeled as a custom Kubernetes resource, and the Operator controller implements declarative management of the task. It can reuse Kubernetes' native mechanisms such as declarative API, scheduler, RBAC permission control, and Finalizer, and realizes unified scheduling of task and storage node affinity, automated cleanup of resources, and standardized management of permissions. It is deeply integrated with Kubernetes native capabilities, has a high degree of automation, completely solves the problem of existing solutions being disconnected from the orchestration system, greatly improves the level of automation, and reduces the cost of manual intervention.

[0015] (2) Operator controller deployment and monitoring: Deploy the developed Operator controller to the Kubernetes cluster. Based on the Kubernetes monitoring mechanism, the Operator controller continuously monitors the creation, update, and deletion of custom DataReplication resources, and also monitors the status change events of the worker Pods it creates (such as Running, Completed, Failed) to ensure that the controller can perceive resource status changes in real time and execute subsequent processing logic in a timely manner.

[0016] In this invention, the Operator controller can be implemented based on the controller-runtime framework, or it can be replaced with an event-driven implementation method according to the actual deployment scale, or it can be split into multi-level controllers (the first-level controller is responsible for Pod creation, and the second-level controller is responsible for progress polling and status updates), which has flexible deployment adaptation capabilities.

[0017] (3) Reconcile process triggering and storage type parsing: When the Operator controller receives the creation event of the DataReplication resource, it immediately triggers the reconcile process. The reconcile process first parses the type fields of source and target in the Spec field to determine the type combination of source and target storage. Common combinations include UFS→UFS, OSS→OSS, and OSS→UFS. Then, it determines the corresponding working Pod creation strategy (single Pod or dual Pod strategy) according to the preset mapping relationship.

[0018] The reconciliation process is the key logic of the Operator controller, which includes four branches: deletion branch, cancellation branch, pending execution branch, and in progress branch. Each branch is triggered based on the DataReplication status and annotation information to achieve fine-grained control of tasks.

[0019] (4) Work Pod Creation and Resource Injection: The Operator controller creates the corresponding work Pod in the Kubernetes cluster according to the parsed storage type combination and the corresponding creation strategy. The work Pod is the actual execution unit for data replication and is embedded with a storage runtime container that supports multiple storage backends (such as an image with embedded rclone).

[0020] When creating a worker Pod, the Operator controller performs two key operations:

[0021] Scheduling strategy configuration: Obtain the cellId identifier by parsing the PVC tag, and configure the node affinity strategy for the worker Pod based on the cellId to ensure that the worker Pod with the UFS volume can be scheduled to the node that can access the UFS volume, thus avoiding storage access failure.

[0022] Secure credential injection: Parse the access credentials (access_key_id, secret_access_key) of the source and target storage from sources such as credentialSecret and unified credential service, and inject the credential information into the worker Pod as an environment variable or temporary mount. The credential information is never persisted to the DataReplication custom resource to ensure the security of the credentials.

[0023] The creation strategy for work Pods is designed with differentiated combinations based on storage type, including:

[0024] Dual Pod Strategy (UFS→UFS): Create a source Pod and a target Pod. The source Pod mounts the source PVC and exposes the S3 interface in server mode, converting the UFS volume into an S3-compatible object storage interface. The target Pod mounts the target PVC and reads data from the source Pod's S3 interface in copy mode, completing the UFS to UFS data copy. This strategy is highly compatible with existing storage runtime images.

[0025] Single Pod Strategy (OSS→OSS, OSS→UFS): Create a single worker Pod. For the OSS→OSS combination, the Pod directly accesses the source and target OSS through the S3 compatible interface and completes data replication within the Pod. For the OSS→UFS combination, the Pod mounts the target PVC and accesses the source OSS through the S3 compatible interface, writing data to the UFS volume in local mode. The single Pod strategy reduces network hops and improves replication efficiency.

[0026] (5) Data copying execution and progress write-back: After the working Pod starts, it performs data copying operation according to the injected parameters and preset mode. The storage runtime container is responsible for implementing the underlying storage access and data transmission logic.

[0027] The Operator controller obtains data replication progress through either progress polling or proactive reporting: the default mode is controller polling, where the controller periodically calls the HTTP progress interface (e.g., / progress) of the worker Pods, and the interface returns JSON data containing replication progress and transmission rate; it can also be switched to Pod proactive reporting mode depending on the deployment scale, where worker Pods report real-time progress to the Operator controller or unified task service, reducing the polling pressure on the controller.

[0028] The Operator controller updates the progress subfield of the Status field of the DataReplication resource in real time with information such as replication progress and transmission rate. If an anomaly is detected, the reason for the anomaly is written into the reason subfield, so as to achieve standardized and visual observation of task progress.

[0029] During the execution of a data replication task, operations such as completion, failure, cancellation, and deletion may be triggered. The Operator controller executes corresponding status update and resource cleanup logic based on different operation types, specifically:

[0030] Task completed: The worker Pod status changes to Completed, the controller sets the DataReplication status to Succeeded, records the endTimestamp, and then cleans up the worker Pod and related temporary resources;

[0031] Task failure: When the worker Pod status changes to Failed or a replication anomaly is detected, the controller sets the status to Failed, writes the failure reason (such as storage access failure, network timeout) in the reason field, cleans up the worker Pod and retains logs for operation and maintenance to troubleshoot.

[0032] Task cancellation: When a DataReplication resource carries a preset cancellation annotation (such as osgalaxy.io / action=cancel), the controller triggers a cancellation branch, terminates the replication operation of the worker Pod, sets the status to Cancelled, writes the cancellation reason, and cleans up the worker Pod;

[0033] Task Deletion: When a deletion event for a DataReplication resource is received, the controller triggers a deletion branch. First, the status is set to Deleting to prevent the resource from being deleted quickly. Then, the worker Pods and related database records are cleaned up. Finally, the Kubernetes Finalizer is removed to complete the complete deletion of the DataReplication resource and avoid resource leakage.

[0034] When it is necessary to extend to new storage types (such as CSI temporary volumes, Azure Blob, GCS), only the judgment logic and parameter construction logic of the new storage type need to be added to the worker Pod creation branch of the Operator controller, without modifying the CRD structure of DataReplication. The attribute information of the new storage type can reuse the common fields of the Spec field (such as type, endpoint, credentialSecret), and add provider or protocol fields when necessary to distinguish different protocols, so as to achieve non-intrusive extension of storage types.

[0035] While maintaining full support for various storage types such as UFS and OSS and different combinations, this invention designs a unified resource model and controller logic, so that operations such as the creation, progress monitoring, cancellation and deletion of data replication tasks are all expressed through the same set of resource states. At the same time, it achieves consistency with the creation, scheduling and cleanup process of Kubernetes worker Pods, taking into account compatibility, scalability and maintainability.

[0036] This invention also provides an Operator-based cross-storage type data replication system, deployed in a Kubernetes cluster, for executing the aforementioned Operator-based cross-storage type data replication method. The system includes: a custom resource module, an Operator controller module, a worker Pod execution module, a storage access module, and a credential management module. These modules work collaboratively to achieve full lifecycle management of cross-storage type data replication. The system comprises:

[0037] Custom Resource Module: Used to create custom resource definitions for data replication in a Kubernetes cluster. An instance of the custom resource definition for data replication is a custom resource for data replication. The custom resource for data replication includes a specification field and a status field. The specification field uniformly describes the attribute information of the source storage and the target storage. The status field is used to record the full lifecycle status information of the data replication task.

[0038] The custom resource module, as the system's resource modeling module, is used to create custom resource definitions (CRDs) for data replication and instantiate DataReplication resources according to business needs. The role of this module is to achieve a unified abstraction of different storage types, standardize the description of source and target storage attribute information through the Spec field, and standardize the recording of the full lifecycle status of tasks through the Status field, providing a unified resource model for subsequent automated management and control.

[0039] Operator controller module: used to deploy Operator controllers to Kubernetes clusters. The Operator controller continuously listens for creation, update, and deletion events of custom resources for data replication, as well as status change events of worker Pods.

[0040] The Operator controller module, as a key control module of the system, is crucial for automating data replication. It comprises five sub-units: a listening unit, a reconciliation unit, a Pod management unit, a progress management unit, and a resource cleanup unit.

[0041] The listening unit continuously monitors all event changes of the DataReplication resource and worker Pods, and is the "sensing entry point" of the controller;

[0042] The reconcile unit executes the Reconcile process, completing the logical processing of four branches: deletion, cancellation, pending execution, and in progress. It is the "brain" of the controller.

[0043] The Pod management unit creates and updates working Pods based on storage type combinations, configures node affinity policies, and achieves standardized scheduling of working Pods.

[0044] The progress management unit obtains the replication progress through polling or active reporting, and updates information such as progress, rate, and reasons for abnormalities to the Status field of DataReplication in real time.

[0045] The resource cleanup unit automatically cleans up resources such as working Pods, temporary mounts, and database records when a task is completed, fails, is canceled, or deleted, to prevent resource leaks.

[0046] Specifically, the Operator module can be implemented based on the controller-runtime framework, or replaced with an event-driven implementation, or split into multi-level controllers, including a first-level controller and a second-level controller. The first-level controller is only responsible for creating worker Pods, while the second-level controller is responsible for progress polling and status updates, adapting to different cluster deployment scales.

[0047] Work Pod Execution Module: After the Operator controller receives the creation event of the custom resource for data replication, it executes the reconcile process, first parsing the storage type of the source and target in the Spec field, and then determining the creation strategy of the work Pod based on the combination of storage types;

[0048] The Work Pod execution module, as the actual execution unit of the system, includes one or more Work Pods, each with an embedded storage runtime container. The storage runtime container is the underlying execution engine for data replication, supporting access to and data replication from multiple storage backends, and must implement a standardized progress reporting interface (HTTP interface). This invention uses a container image with an embedded rclone by default, but it can also be replaced with replication engines that conform to the interface convention, such as Restic or a custom S3 / local read / write library, and has good compatibility.

[0049] Storage access module: used by the Operator controller to create corresponding work Pods in the Kubernetes cluster according to the creation strategy. The work Pods are embedded with storage runtimes that support multiple storage backends. The Operator controller injects the access credentials and path information of the source and target storage into the work Pods as environment variables or by mounting. The credentials information is not written to the custom resource CR.

[0050] The storage access module, as the system's storage interface module, is used to achieve standardized access to different types of storage resources, including UFS storage access units and OSS storage access units.

[0051] The UFS storage access unit uses Kubernetes' PVC mounting mechanism to achieve standardized access to UFS volumes and supports various PVC mounting modes (such as read-write and read-only).

[0052] The OSS storage access unit enables standardized access to various OSS systems (such as Ceph and Alibaba Cloud OSS) through an S3 compatible interface, shielding the interface differences between different vendors' OSS systems.

[0053] This storage access module supports non-intrusive expansion of new storage access units (such as AzureBlob access units and GCS access units), requiring only the implementation of the corresponding storage access logic without modifying the main system architecture.

[0054] The credential management module is used to perform data replication operations according to a preset mode after the work Pod starts. The Operator controller polls the progress interface of the work Pod or receives the progress report actively submitted by the work Pod, and updates the replication progress and transmission rate to the Status field of the custom data replication resource in real time. When the data replication task is completed, fails, is canceled, or a deletion operation is triggered, the Operator controller performs the corresponding status update and resource cleanup operations, updates the Status field of the custom data replication resource, and cleans up the work Pod and related temporary resources.

[0055] The credential management module serves as the system's security module, managing all access credentials for source and target storage. It supports parsing and secure injection of credentials from multiple sources, including Kubernetes Secret, Unified Credential Service, Vault, and cloud vendor IAM roles. A key feature of this module is that credentials are not written to disk; they are only injected into worker Pods when they are created, never written to DataReplication custom resources. This effectively ensures the security of credential information and prevents credential leakage.

[0056] Furthermore, the system also includes a scheduling adaptation module, which enables flexible replacement of scheduling strategies. It can replace the node affinity strategy based on cellId with a scheduling strategy based on topology domain and resource reservation, or integrate a custom scheduler to ensure that worker Pods can be reasonably scheduled to nodes that can access the corresponding storage, adapting to the Kubernetes scheduling specifications of different enterprises.

[0057] Furthermore, the working Pod execution module supports two progress reporting methods: controller polling and Pod proactive reporting. These methods can be flexibly switched according to the size of the Kubernetes cluster and the number of tasks: small-scale clusters use the controller polling mode, which has simple logic; large-scale clusters use the Pod proactive reporting mode, which reduces the performance overhead of the controller and improves the overall processing capacity of the system.

[0058] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the operator-based cross-storage type data copying method as described above.

[0059] The present invention also provides a computer device, the computer device including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the operator-based cross-storage type data copying method as described above.

[0060] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0061] The cross-storage type data replication method and system based on Operator provided by this invention achieves unified modeling for multiple storage types such as UFS and OSS by using the Spec field of DataReplication custom resources. It uses the same interface to describe the attribute information of different storage types. When expanding to new storage types, there is no need to modify the CRD structure; only a small amount of judgment logic needs to be added to the Operator controller, achieving non-intrusive expansion. It realizes unified abstraction for multiple storage types, has strong scalability, and significantly reduces the development cost of integrating new storage types. The Status field of DataReplication defines six standardized task states, enabling real-time writing and unified querying of key information such as task progress, transmission rate, and failure reasons. Task cancellation, retry, and deletion operations are all executed in a standardized manner, providing a unified API interface that can be seamlessly integrated with enterprise cloud-native management platforms, achieving platform-based operation and maintenance of data replication tasks and solving the problem of insufficient maintainability in existing solutions. The reconciliation process of the Operator controller achieves fine-grained full lifecycle management of data replication tasks, targeting task completion, ... Corresponding resource cleanup logic is designed for different scenarios such as failure, cancellation, and deletion. Combined with Kubernetes' finalizer mechanism, it ensures the thorough cleanup of worker Pods and temporary resources, effectively avoiding resource leaks and improving the resource utilization of the Kubernetes cluster. A secure credential management mechanism is designed, with credential information injected only when worker Pods are created and not persisted to custom resources, effectively preventing credential leaks. At the same time, all modules of the system support flexible replacement and adaptation. For example, the storage runtime can be replaced with Restic, the controller can be split into multiple levels, and the scheduling strategy can be flexibly replaced. It is highly compatible with existing cloud-native ecosystem components and can be widely adapted to the Kubernetes cluster environments of different enterprises. Two differentiated worker Pod creation strategies are designed for different storage type combinations: single Pod and dual Pod. The dual Pod strategy is used for UFS→UFS to ensure compatibility, while the single Pod strategy is used for OSS→OSS and OSS→UFS to reduce network jumps. While ensuring compatibility, it maximizes the execution efficiency of data replication and can be widely used in cross-storage data migration and synchronization scenarios in cloud-native and multi-cluster environments. Attached Figure Description

[0062] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0063] In the attached diagram:

[0064] Figure 1This is a flowchart illustrating the overall process architecture of the operator-based cross-storage type data replication method according to an embodiment of the present invention.

[0065] Figure 2 This is a flowchart and state machine diagram of the Reconcile process according to an embodiment of the present invention;

[0066] Figure 3 This is a UFS→UFS data flow diagram according to an embodiment of the present invention;

[0067] Figure 4 This is a single Pod data flow diagram of OSS→UFS according to an embodiment of the present invention;

[0068] Figure 5 This is a schematic diagram of the configuration of a computer device according to an embodiment of the present invention. Detailed Implementation

[0069] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of systems and products consistent with some aspects of this disclosure as detailed in the appended claims.

[0070] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0071] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0072] The embodiments of the present invention will be described in further detail below.

[0073] Example

[0074] This invention provides an Operator-based method for cross-storage type data replication, comprising the following steps:

[0075] S1. Create a custom resource definition (CRD) for data replication in the Kubernetes cluster. An instance of the custom resource definition (CRD) for data replication is called a custom resource for data replication (DataReplication CR). The custom resource for data replication includes a specification field and a status field. The specification field uniformly describes the attribute information of the source storage and the target storage. The status field is used to record the full lifecycle status information of the data replication task.

[0076] The Spec field includes the storage type field for both the source and target attributes. When the type is UFS, it also includes the claimRef field, which is used to reference the Kubernetes persistent volume declaration. When the type is OSS, it also includes the endpoint field, bucket name field, path field, and credentialSecret field.

[0077] The Status field includes a UUID unique identifier field, a status task status field, a progress copying progress field, a reason description field, and an endTimestamp end timestamp field. The status task status includes six states: Pending, InProgress, Successed, Failed, Cancelled, and Deleting. Each state transitions according to a preset state machine.

[0078] S2. Deploy the Operator controller to the Kubernetes cluster. The Operator controller continuously listens for the creation, update, and deletion events of custom resources in Data Replication, as well as the status change events of worker Pods.

[0079] S3. After receiving the creation event of the custom resource for data replication, the Operator controller executes the reconcile process, first parsing the storage type of the source and target in the Spec field, and then determining the creation strategy of the worker Pod based on the combination of storage types.

[0080] like Figure 2The diagram shows the Reconcile process and state machine of the Operator controller, including: the Reconcile entry point, the judgment conditions and actions for the four branches of deletion / cancellation / pending execution / in progress; the status transition of DataReplication (Pending→InProgress→Succeeded / Failed / Cancelled / Deleting); and the timing of Pod creation, progress polling, and Pod cleanup.

[0081] The Reconcile process includes four branches: deletion in progress, cancellation, pending execution, and in progress.

[0082] If a custom resource in Data Replication triggers a deletion operation, the deletion branch is entered. First, the status field is set to Deleting, then the worker Pod and related database records are cleaned up, and finally the Kubernetes Finalizer is removed to complete the resource deletion.

[0083] If the custom resource for data replication carries a preset cancellation annotation, enter the cancellation branch, terminate the current data replication operation, set the status field to Cancelled, and write the cancellation reason.

[0084] If the status of the custom resource for data replication is Pending, proceed to the pending execution branch, create a working Pod based on the storage type combination, and wait for the working Pod status to change to Running before setting the status of the Status field to InProgress.

[0085] If the status of the custom resource for data replication is InProgress, enter the in-progress branch, continuously obtain the data replication progress and update the progress in the Status field. If the task is detected as completed, failed, or timed out, perform the corresponding status update and resource cleanup.

[0086] When both the source and target are UFS, a dual-pod creation strategy is adopted, creating a source Pod and a target Pod. The source Pod mounts a source PVC and exposes an S3 interface in server mode, while the target Pod mounts a target PVC and reads data from the source Pod's S3 interface in copy mode to complete the replication. Figure 3The diagram shows the creation and data flow of the source Pod and target Pod during UFS to UFS replication, including: source PVC, target PVC, source Pod (server mode, S3 port), target Pod (copy mode, mounted target PVC); the data flow from the source PVC through the S3 exposed by the source Pod to the target Pod and then to the target PVC; and the annotation of cellId and node affinity.

[0087] When both the source and target are OSS, or the source is OSS and the target is UFS, a single Pod creation strategy is adopted to create a single working Pod. The working Pod directly accesses OSS through the S3 compatible interface, mounts a PVC to access UFS, and completes the data copying operation inside the Pod.

[0088] S4. The Operator controller creates a corresponding working Pod in the Kubernetes cluster according to the creation strategy. The working Pod embeds a storage runtime that supports multiple storage backends. The Operator controller injects the access credentials and path information of the source and target storage into the working Pod as environment variables or by mounting. The credentials information is not written to the custom resource CR.

[0089] Specifically, when creating a worker Pod, the Operator controller obtains the cellId identifier by parsing the tag of the persistent volume declaration (PVC), and configures a node affinity policy for the worker Pod based on the cellId identifier to ensure that the worker Pod mounted with UFS is scheduled to a node that can access the UFS volume.

[0090] The storage runtime is a container image with an embedded rclone, or it can be replaced with a replication engine that supports server mode, exposes S3 interfaces, and provides a copy mode data replication with a progress interface. The progress interface is an HTTP interface that outputs JSON format data containing the replication progress.

[0091] Credentials exist only temporarily within the Pod as environment variables. CR contains no credential information throughout the entire process, ensuring that the credentials are only passed to the Pod and not persisted to disk. Credentials are only visible to the Pod that actually performs the replication. Even if other personnel / components can view the CR, they can only see the Secret name and cannot obtain the actual credentials. Credentials are not written to the CR, nor are they retained due to the persistence of the CR. Credentials disappear immediately after the Pod is destroyed, eliminating the risk of leakage during snapshot recovery.

[0092] Preferably, credential storage is based on K8s Secret, which reuses K8s' native security features such as RBAC access control (which can restrict who can create / access Secret) and etcd encrypted storage, eliminating the need to develop a self-developed security system and making it more reliable.

[0093] The sources of access credentials for the source and target storage include one or more of Kubernetes Secret, Unified Credential Service, Vault, and cloud vendor IAM roles. Before creating a worker Pod, the Operator controller parses the access_key_id (the "username" for accessing OSS) and secret_access_key (the "password" corresponding to the "username" for accessing OSS), and injects the access_key_id and secret_access_key into the worker Pod.

[0094] S5. After the working Pod starts, it performs data replication operation according to the preset mode. The Operator controller polls the progress interface of the working Pod or receives the active progress report of the working Pod, and updates the replication progress and transmission rate to the Status field of the custom resource for data replication (DataReplicationCR) in real time.

[0095] When a data replication task completes, fails, is canceled, or triggers a deletion operation, the Operator controller performs the corresponding status update and resource cleanup operations, updates the Status field of the custom data replication resource DataReplication, and cleans up the working Pod and related temporary resources.

[0096] The storage types include Unified File System (UFS) and Object Storage Service (OSS). The UFS is mounted to the working Pod via a PVC, and the OSS is accessed using an S3-compatible interface. The creation strategy of the working Pod is divided into a single Pod creation strategy and a dual Pod creation strategy based on the combination of the storage types of the source and target.

[0097] When extending to new storage types, only the judgment logic and parameter construction logic of the new storage type are added to the worker Pod creation branch of the Operator controller, without modifying the CRD structure of DataReplication; the new storage types include CSI temporary volumes, AzureBlob, GCS, etc.

[0098] Figure 1This paper illustrates the overall process architecture of the operator-based cross-storage type data replication method according to an embodiment of the present invention, including: Kubernetes cluster, DataReplication CRD and instance, Operator controller, worker Pod (source Pod / target Pod), storage runtime container, UFS (PVC mount) and OSS (S3 endpoint) schematic positions; data flow direction (source → worker Pod → target) and the monitoring and update relationship between the controller, CR and Pod.

[0099] This invention also provides an Operator-based cross-storage type data replication system, deployed in a Kubernetes cluster, for executing the aforementioned Operator-based cross-storage type data replication method, including:

[0100] Custom Resource Module: Used to create custom resource definition (CRD) for data replication and instantiate DataReplication resources. The Spec field of the DataReplication resource uniformly abstracts the attribute information of the source and target storage, and the Status field records the status of the task throughout its entire lifecycle.

[0101] Operator Controller Module: This is the main control module of the system, including a listening unit, a reconciliation unit, a Pod management unit, a progress management unit, and a resource cleanup unit. The listening unit continuously monitors event changes of DataReplication resources and working Pods. The reconciliation unit executes the Reconcile process and completes branch logic processing. The Pod management unit creates / updates working Pods based on storage type and configures node affinity. The progress management unit obtains the data replication progress and updates the Status field of DataReplication. The resource cleanup unit cleans up working Pods and temporary resources after the task is completed.

[0102] Specifically, the Operator module is implemented based on the controller-runtime framework, or can be replaced with an event-driven implementation, or split into a multi-level controller, which includes a first-level controller and a second-level controller. The first-level controller is only responsible for creating the work Pod, and the second-level controller is responsible for progress polling and status updates.

[0103] Work Pod Execution Module: Includes one or more work Pods, each work Pod embedding a storage runtime container. The storage runtime container supports access to multiple storage backends and data replication, and provides a progress reporting interface.

[0104] Preferably, the progress reporting method of the working Pod execution module includes two types: controller polling and Pod active reporting, which can be switched according to the actual deployment scale. The Pod active reporting means that the working Pod reports the replication progress to the Operator controller or unified task service in real time.

[0105] Storage access module: used to connect to different types of storage resources, including UFS storage access unit and OSS storage access unit. The UFS storage access unit accesses UFS volumes through PVC mounting, and the OSS storage access unit accesses OSS through S3 compatible interface, and supports the expansion of new storage access units.

[0106] The credential management module manages access credentials for source and target storage, supports parsing and injection of credentials from multiple sources, ensures secure transmission of credential information, and prevents credential information from being written to custom resources.

[0107] Application examples

[0108] The operator-based cross-storage type data replication method and system of this invention, in a real-world application scenario deployed in a Kubernetes 1.25+ cluster, achieves cross-storage data replication for UFS and OSS storage types and different combinations. The implementation method includes seven steps: custom resource definition and instantiation, operator controller deployment and monitoring, reconciliation process triggering and storage type resolution, worker Pod creation and resource injection, data replication execution and progress write-back, task status processing and resource cleanup, and storage type expansion. The main implementation steps are as follows:

[0109] S1. Custom resource definition and instantiation:

[0110] (1) Create a DataReplication CRD file in the Kubernetes cluster using the kubectlapply command. The CRD file defines the structure of two key fields: Spec and Status. The Spec field contains two sub-objects: source and target, both of which define fields such as type, claimRef, endpoint, bucket, path, and credentialSecret. The Status field defines fields such as uuid, status, progress, reason, and endTimestamp.

[0111] (2) Based on business requirements, create a YAML configuration file for DataReplication and instantiate custom resources; for example, create an OSS→UFS data replication task, where the source type in the Spec field is OSS, the endpoint is configured as https: / / OSS−cn−hangzhou.aliyuncs.com, the bucket as source−bucket, the path as / data, and the credentialSecret as OSS−secret; the target type is UFS, and the claimRef is pvc−target; submit the YAML file to the Kubernetes cluster using the kubectlapply command to complete resource instantiation. Figure 4 The diagram shows the parameters and data flow of a single Pod during OSS to UFS replication, including: source OSS (endpoint, bucket, path), target PVC and mount path, a single working Pod (copy mode, source S3, target local); and the data flow from OSS to the Pod and then to the mount path.

[0112] S2 Operator Controller Deployment and Monitoring:

[0113] (1) Develop an Operator controller based on the controller-runtime framework. The development language is Go. The controller implements operations such as List, Watch, and Update on DataReplication resources, as well as monitoring the status of worker Pods.

[0114] (2) Package the developed Operator controller into a Docker image and push the image to the enterprise's private image repository;

[0115] (3) Deploy the Operator controller to the kube-system namespace of the Kubernetes cluster through the Deployment resource, configure the number of replicas to 1, ensure that the controller runs as a single instance and avoid concurrency conflicts;

[0116] (4) After the controller starts, it continuously listens to all events of the DataReplication resource and the status change events of the working Pod through the Informer mechanism, and the listening results are synchronized to the controller's cache in real time.

[0117] S3, Harmonization Process Triggering and Storage Type Resolution:

[0118] (1) After the controller hears the creation event of the DataReplication resource, it triggers the reconciliation process and reads the complete information of the resource from the cache;

[0119] (2) The reconciliation process parses the Spec field to find that the source type is OSS and the target type is UFS, determines the storage type combination to be OSS→UFS, and selects the single Pod creation strategy according to the preset mapping relationship;

[0120] (3) When the reconciliation process detects that the status of DataReplication is Pending and there are no uncomment or deletion events, it enters the pending execution branch.

[0121] S4. Work Pod Creation and Resource Injection:

[0122] (1) The Pod management unit generates the YAML configuration of the working Pod according to the single Pod creation strategy. The Pod image adopts the storage runtime image of rclone embedded in the enterprise private repository, and the image version is v1.65.0.

[0123] (2) The credential management module parses access_key_id and secret_access_key from OSS-secret and injects them as environment variables into the working Pod;

[0124] (3) Since the target is UFS, the Pod management unit parses the label of pvc−target to obtain cellId=cn−hangzhou−1, and configures the node affinity policy for the working Pod, allowing scheduling only to the node with the label cellId=cn−hangzhou−1;

[0125] (4) The controller creates worker Pods through the Kubernetes API. The namespace of the Pod is consistent with the DataReplication resource. The Pod mounts pvc-target to the / opt / target directory.

[0126] S5, Data Replication Execution and Progress Writeback:

[0127] (1) After the working Pod is scheduled to the node with cellId=cn−hangzhou−1, when the rclone storage is started, rclone accesses the source OSS based on the credential information in the environment variables. The mounted PVC is the target storage. The rclonecopy command is executed to copy the data under the path https: / / OSS−cn−hangzhou.aliyuncs.com / source−bucket / data of the source OSS to the / opt / target directory;

[0128] (2) After the working Pod starts, it exposes the HTTP progress interface / progress on port 8080. The interface returns JSON format data: {"progress":0,"speed":"0B / s","total":1024000000,"copied":0};

[0129] (3) The progress management unit of the controller polls the / progress interface every 5 seconds and updates the progress field value obtained to the progress subfield of the Status field of DataReplication. For example, when the replication is 50% complete, the progress is set to 50.

[0130] (4) If a network timeout occurs during the replication process, rclone will write the exception information to the log of the working Pod. If the controller detects that the progress has not been updated for a long time, it will set the reason field to "Network timeout, unable to access source OSS".

[0131] Task status handling and resource cleanup, specifically including:

[0132] (1) Once the data replication is complete, rclone execution ends, and the status of the worker Pod changes to Completed;

[0133] (2) After the controller listens to the Completed status of the working Pod, it sets the status of DataReplication to Succeeded, records the endTimestamp as the current timestamp (e.g., 1719288000), and sets the progress to 100.

[0134] (3) The resource cleanup unit deletes the working Pod through the Kubernetes API and cleans up the Pod's related logs and temporary mounts;

[0135] (4) If the operation and maintenance personnel need to cancel the task, add the annotation osgalaxy.io / action=cancel to the DataReplication resource through the kubectlannotate command. After the controller listens to the annotation update event, it enters the cancellation branch, terminates the rclone process of the working Pod, sets the status to Cancelled, sets the reason to "user manually cancels", and cleans up the working Pod.

[0136] If an enterprise needs to add support for the Azure Blob storage type, it only needs to add the logic to determine type=azure in the Pod management unit of the Operator controller. When creating a worker Pod, it parses the account_name, account_key, container and other information of AzureBlob from the Spec field, injects them into the environment variables of the worker Pod, and configures the azureblob backend driver in rclone. There is no need to modify the CRD file of DataReplication, which achieves non-intrusive extension.

[0137] The cross-storage type data replication system based on Operator of the present invention includes a custom resource module, an Operator controller module, a worker Pod execution module, a storage access module, a credential management module, and a scheduling adaptation module. The custom resource module defines the structure of DataReplication resources through CRD files and provides YAML configuration templates for operation and maintenance personnel to instantiate resources. This module synchronizes the instantiated resource information to the etcd database of Kubernetes, providing a unified resource model for other modules.

[0138] The Operator controller module is deployed as a Deployment resource and is the main control module of the system. Its listening unit senses the status changes of DataReplication resources and worker Pods in etcd in real time. The reconciliation unit executes the corresponding branch logic according to the status changes. The Pod management unit, progress management unit, and resource cleanup unit complete specific operations according to the instructions of the reconciliation unit. All operation results are synchronized to etcd through the Kubernetes API.

[0139] The worker Pod execution module is dynamically created by the Operator controller. Each worker Pod corresponds to a data replication task, with an embedded rclone storage runtime container that performs the actual data replication operation and reports the progress to the controller via an HTTP interface. After the task is completed, it is automatically deleted by the controller's resource cleanup unit.

[0140] The UFS storage access unit of the storage access module accesses UFS volumes through the Kubernetes PVC mounting mechanism, and the OSS storage access unit accesses OSS through the S3 compatible interface. The two access units encapsulate the underlying logic of storage access and provide a standardized storage access interface for the worker Pod execution module.

[0141] The credential management module interfaces with the enterprise's unified credential service and Kubernetes Secret to achieve unified credential parsing and secure injection, ensuring that credential information is not written to custom resources and is only injected when worker Pods are created, effectively preventing credential leakage;

[0142] The scheduling adaptation module provides scheduling policy support for the Pod management unit. In addition to the node affinity policy based on cellId, it also supports the scheduling policy based on topology domain. It can be flexibly switched according to the enterprise's Kubernetes cluster specifications to ensure reasonable scheduling of worker Pods.

[0143] The interaction process between the modules is as follows: the custom resource module instantiates the DataReplication resource → the listener unit of the Operator controller module detects the resource creation → the reconciliation unit parses the storage type and triggers the branch to be executed → the Pod management unit configures the scheduling policy through the scheduling adaptation module, injects security credentials through the credential management module, and creates a working Pod → the working Pod execution module accesses the source and target storage through the storage access module and performs data replication → the progress management unit obtains the replication progress and updates it to the custom resource → after the task is completed, the resource cleanup unit cleans up the working Pod, completing the entire data replication process.

[0144] The operator-based cross-storage type data replication method and system of the present invention has significant industrial applicability and can be widely applied to enterprise-level data management scenarios under cloud-native, microservice, and multi-cluster technical architectures, including but not limited to: cross-storage data migration of cloud-native platforms in industries such as finance, e-commerce, internet, and manufacturing; cross-site data synchronization in enterprise multi-cluster environments; data interaction between public cloud OSS and private cloud UFS under hybrid cloud architecture; and storage disaster recovery and data backup for cloud-native applications.

[0145] The operator-based cross-storage type data replication system of the present invention can be deployed in various versions of Kubernetes clusters, is highly compatible with existing cloud-native ecosystem components, and can be quickly deployed and expanded through containerization. It significantly improves the automation and platformization level of cross-storage data management for enterprises, reduces operation and maintenance costs, and has broad market application prospects and significant economic value.

[0146] This invention also provides a computer device. Figure 5This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention; see the accompanying drawings. Figure 5 As shown, the computer device includes: an input device 23, an output device 24, a memory 22, and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the operator-based cross-memory type data copying method provided in the above embodiments; wherein the input device 23, the output device 24, the memory 22, and the processor 21 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0147] The memory 22, as a read / write storage medium for a computing device, can be used to store software programs and computer-executable programs, such as the program instructions corresponding to the operator-based cross-storage type data copying method described in this embodiment of the invention. The memory 22 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device. Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include memory remotely located relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0148] Input device 23 can be used to receive input digital or character information, and generate key signal inputs related to user settings and function control of the device; output device 24 may include display devices such as a display screen.

[0149] The processor 21 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 22, thereby realizing the above-mentioned operator-based cross-memory type data copying method.

[0150] The computer equipment provided above can be used to execute the operator-based cross-storage type data copying method provided in the above embodiments, and has corresponding functions and beneficial effects.

[0151] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the operator-based cross-storage type data copying method provided in the above embodiments. The storage medium can be any type of memory device or storage device, including: mounting media such as CD-ROM, floppy disk, or magnetic tape; computer system memory or random access memory such as DRAM, DDRRAM, SRAM, EDORAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory components; the storage medium may also include other types of memory or combinations thereof; furthermore, the storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet); the second computer system can provide program instructions to the first computer for execution. The storage medium includes two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0152] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the operator-based cross-storage type data copying method described in the above embodiments, but can also perform related operations in the operator-based cross-storage type data copying method provided in any embodiment of the present invention.

[0153] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0154] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An Operator-based cross-storage type data replication method, characterized in that, Includes the following steps: S1. Create a custom resource definition for data replication in the Kubernetes cluster. An instance of the custom resource definition for data replication is a custom resource for data replication. The custom resource for data replication includes a specification field (Spec) and a status field (Status). The specification field uniformly describes the attribute information of the source storage and the target storage. The status field is used to record the full lifecycle status information of the data replication task. S2. Deploy the Operator controller to the Kubernetes cluster. The Operator controller continuously listens for the creation, update, and deletion events of custom resources for data replication, as well as the status change events of worker Pods. S3. After receiving the creation event of the custom resource for data replication, the Operator controller executes the Reconcile process, first parsing the storage type of the source and target in the Spec field, and then determining the creation strategy of the worker Pod based on the combination of storage types. S4. The Operator controller creates a corresponding working Pod in the Kubernetes cluster according to the creation strategy. The working Pod embeds a storage runtime that supports multiple storage backends. The Operator controller injects the access credentials and path information of the source and target storage into the working Pod as environment variables or by mounting. The credentials information is not written to the custom resource CR. S5. After the working Pod starts, it performs data replication operation according to the preset mode. The Operator controller polls the progress interface of the working Pod or receives the active progress report of the working Pod, and updates the replication progress and transmission rate to the Status field of the custom data replication resource in real time. When a data replication task completes, fails, is canceled, or triggers a deletion operation, the Operator controller performs corresponding status update and resource cleanup operations, updates the Status field of the custom data replication resource, and cleans up the working Pod and related temporary resources.

2. The operator-based cross-storage type data replication method according to claim 1, characterized in that, The reconcile process in step S3 includes four branch processing steps: deletion branch, cancellation branch, pending execution branch, and in progress branch. If data replication triggers a deletion operation for a custom resource, proceed to the deletion branch, first set the status field to Deleting, then clean up the worker Pod and related database records, and finally remove the Kubernetes Finalizer to complete the resource deletion. If the data copying custom resource carries a preset cancellation annotation, enter the cancellation branch, terminate the current data copying operation, set the status field to Cancelled and write the cancellation reason; If the status of the custom resource being copied is Pending, proceed to the pending execution branch, create a working Pod based on the storage type combination, and wait for the working Pod status to change to Running before setting the status of the Status field to InProgress. If the status of the data copying custom resource is InProgress, enter the in-progress branch, continuously obtain the data copying progress and update the progress in the Status field. If the task is detected as completed, failed or timed out, perform the corresponding status update and resource cleanup.

3. The operator-based cross-storage type data replication method according to claim 1, characterized in that, The attribute information of source and target in the Spec field includes the storage type field. When the type is UFS, it also includes the claimRef field used to reference the Kubernetes persistent volume declaration. When the type is OSS, it also includes the endpoint field, bucket name field, path field, and credentialSecret field.

4. The operator-based cross-storage type data replication method according to claim 1, characterized in that, The Status field includes a UUID unique identifier field, a status task status field, a progress copying progress field, a reason description field, and an endTimestamp end timestamp field. The status task status includes six states: Pending, InProgress, Successed, Failed, Cancelled, and Deleting. Each state completes the transition according to a preset state machine.

5. The operator-based cross-storage type data replication method according to claim 1, characterized in that, The creation strategy for the work Pod is as follows: When both the source and target are UFS, a dual-Pod creation strategy is adopted to create a source Pod and a target Pod. The source Pod mounts the source PVC and exposes the S3 interface in server mode. The target Pod mounts the target PVC and reads data from the S3 interface of the source Pod in copy mode to complete the replication. When both the source and target are OSS, or the source is OSS and the target is UFS, a single Pod creation strategy is adopted to create a single working Pod. The working Pod directly accesses OSS through the S3 compatible interface, mounts a PVC to access UFS, and completes the data copying operation inside the Pod.

6. The operator-based cross-storage type data replication method according to claim 1, characterized in that, When creating a worker Pod, the Operator controller obtains the cellId identifier by parsing the tag of the persistent volume declaration. Based on the cellId identifier, it configures a node affinity policy for the worker Pod to ensure that the worker Pod mounted with UFS is scheduled to a node that can access the UFS volume.

7. The operator-based cross-storage type data replication method according to claim 1, characterized in that, The storage runtime is a container image with embedded rclone, or it can be replaced by a replication engine that supports server mode exposing S3 interfaces and copy mode data replication and provides a progress interface, which is an HTTP interface that outputs JSON format data containing the replication progress.

8. The operator-based cross-storage type data replication method according to claim 1, characterized in that, The sources of access credentials for the source and target storage include one or more of Kubernetes Secret, Unified Credential Service, Vault, and cloud vendor IAM roles. Before creating a worker Pod, the Operator controller parses the access_key_id and secret_access_key to obtain them and injects them into the worker Pod.

9. An Operator-based cross-storage type data replication system, characterized in that, Deployed in a Kubernetes cluster, for executing the operator-based cross-storage type data replication method as described in any one of claims 1-8, comprising: Custom Resource Module: Used to create custom resource definitions for data replication in a Kubernetes cluster. An instance of the custom resource definition for data replication is a custom resource for data replication. The custom resource for data replication includes a specification field and a status field. The specification field uniformly describes the attribute information of the source storage and the target storage. The status field is used to record the full lifecycle status information of the data replication task. Operator controller module: used to deploy Operator controllers to Kubernetes clusters. The Operator controller continuously listens for creation, update, and deletion events of custom resources for data replication, as well as status change events of worker Pods. Work Pod Execution Module: After the Operator controller receives the creation event of the custom resource for data replication, it executes the reconcile process, first parsing the storage type of the source and target in the Spec field, and then determining the creation strategy of the work Pod based on the combination of storage types; Storage access module: used by the Operator controller to create corresponding work Pods in the Kubernetes cluster according to the creation strategy. The work Pods are embedded with storage runtimes that support multiple storage backends. The Operator controller injects the access credentials and path information of the source and target storage into the work Pods as environment variables or by mounting. The credentials information is not written to the custom resource CR. The credential management module is used to perform data replication operations according to a preset mode after the work Pod starts. The Operator controller polls the progress interface of the work Pod or receives the progress report actively submitted by the work Pod, and updates the replication progress and transmission rate to the Status field of the custom data replication resource in real time. When the data replication task is completed, fails, is canceled, or a deletion operation is triggered, the Operator controller performs the corresponding status update and resource cleanup operations, updates the Status field of the custom data replication resource, and cleans up the work Pod and related temporary resources.

10. The operator-based cross-storage type data replication system according to claim 9, characterized in that, The Operator module is implemented based on the controller-runtime framework, or can be replaced with an event-driven implementation, or split into multi-level controllers. The multi-level controllers include a first-level controller and a second-level controller. The first-level controller is only responsible for creating worker Pods, while the second-level controller is responsible for progress polling and status updates.

Images