A k8s cluster-based persistent storage data migration system and method

Abstract

The application provides a K8s cluster-based persistent storage data migration system and method, which comprises a front-end module, a back-end module, and a Job execution module.The front-end module provides a graphical user interface and receives user input migration task configuration parameters through a form.The back-end module responds to a migration task creation request submitted by the front-end module, creates or selects an intermediate storage based on the migration task configuration parameters, dynamically generates and sequentially drives a data backup Job and a data recovery Job through a templating engine.The Job execution module is composed of Pods instantiated in a K8s cluster by the data backup Job and the data recovery Job.The back-end module is configured to call a K8s API Server to obtain a Job resource state in response to a state query request from the front-end module, and feed back the Job resource state as real-time state of the migration task to the front-end module.The front-end module is further used to display a migration progress and a migration log based on the real-time state.The application realizes one-key data migration.

Description

Technical Field

[0001] This invention relates to the field of data migration technology, and specifically to a persistent storage data migration system and method based on a K8s cluster. Background Technology

[0002] In actual operation and maintenance, Kubernetes clusters often contain multiple storage service types (such as NFS, Ceph, GlusterFS, and cloud vendor block storage), and these storage backends differ in performance, availability, and data management strategies. When applications need to migrate data across different storage types, such as for storage performance optimization, cluster upgrades, or cross-cloud migrations, existing technologies require operators to manually perform data backup, migration, and recovery operations, which presents the following technical problems: 1. Complex operation, high professional requirements: Operators must be highly proficient in a series of command-line tools such as kubectl, rsync, tar, and rclone, and have a deep understanding of Kubernetes storage concepts (such as PV, PVC, StorageClass) and workload management mechanisms. Errors in any step can lead to data loss or application service interruption. 2. Cumbersome process, lack of automation: The entire migration process consists of multiple discrete manual steps, failing to form a complete automated workflow. This is not only inefficient and time-consuming, but also makes it difficult to ensure consistency across multiple environments, resulting in poor repeatability. 3. Reliance on low-level details, low level of abstraction: Operators need to directly understand the specific types of source and destination storage services, mount point configurations, access methods, and other low-level details, rather than managing them at a unified abstract level. This increases the complexity of the operation and the probability of errors. 4. Risks of data inconsistency and service interruption: During manual migration, accurately controlling application write operations, ensuring the integrity of data synchronization, and smoothly switching traffic pose significant challenges to operations personnel. Improper timing of operations can easily lead to data inconsistency or unexpected service downtime. Therefore, there is an urgent need to provide a persistent storage data migration system and method based on K8s clusters, which can reduce the technical threshold and operational complexity of data migration between K8s cluster storage services and achieve secure, efficient, and one-click data migration. Summary of the Invention

[0003] In view of this, it is necessary to provide a persistent storage data migration system and method based on Kubernetes clusters to solve the technical problems in the existing technology that rely on manual migration by operators, which is cumbersome and prone to errors, resulting in high migration risks and operational complexity.

[0004] Firstly, to address the aforementioned technical problems, this invention provides a persistent storage data migration system based on a Kubernetes cluster, comprising: The front-end module is configured to provide a graphical user interface for receiving migration task configuration parameters input by the user via a form. The backend module is configured to respond to the migration task creation request submitted by the frontend module, create or select an intermediate storage based on the migration task configuration parameters, and dynamically generate and sequentially drive a data backup job and a data recovery job through a template engine; wherein, the data backup job is used to synchronize the data of the source PVC to the intermediate storage, and the data recovery job is used to synchronize the data of the intermediate storage to a newly created destination PVC; The Job execution module consists of Pods instantiated from the data backup Job and the data recovery Job in the K8s cluster; The backend module is also configured to respond to the status query request from the frontend module by calling the K8s APIServer to obtain the Job resource status, and to feed back the Job resource status as the real-time status of the migration task to the frontend module. The front-end module is also used to display migration progress and migration logs in the graphical user interface based on the real-time status.

[0005] In one possible implementation, the graphical user interface includes a configuration modal, which includes a task basic information configuration sub-frame, a source storage configuration sub-frame, a destination storage configuration sub-frame, and a temporary storage configuration sub-frame. The task basic information configuration sub-frame is used to receive the migration task name, task description information, PVC namespace and source PVC name; The source storage configuration subframe is used to receive the source storage type and source storage capacity; The destination storage configuration subframe is used to receive the destination storage type and destination storage capacity; The temporary storage configuration sub-frame is used to select a temporary storage mode and generate temporary storage parameters based on the temporary storage mode; the temporary storage mode is either a temporary PVC or a remote storage mode. Wherein, when the temporary storage mode is a temporary PVC, the temporary storage parameters include temporary storage service and temporary storage space; when the temporary storage mode is a remote storage mode, the temporary storage parameters include remote path.

[0006] In one possible implementation, the graphical user interface further includes a task status monitoring interface, which includes a task overview area, a task progress area, and a log scrolling display area. The task overview area is used to display the migration task configuration parameters in list form; The task progress area is used to display multiple migration steps of the migration task based on the vertical step bar component, and to determine and highlight the current migration step based on the real-time status. The log scrolling display area is used to display the logs generated during the Job execution process in real time.

[0007] In one possible implementation, the polling strategy of the task status monitoring interface is: When the migration task is in different non-final states, different polling intervals are used; when the migration task is in the final state, polling is stopped.

[0008] In one possible implementation, the front-end module is further configured as follows: When responding to a user's instruction to initiate a migration operation, the status query interface of the backend module is called to obtain the Job resource status. If the Job resource status indicates that there is an associated active migration task, the configuration modal is disabled and the task status monitoring interface is entered. If the Job resource status indicates that there is no associated active migration task, the configuration modal is opened to receive user input.

[0009] In one possible implementation, the backend module includes an interface access layer, a business logic layer, a task execution layer, and a status monitoring layer; The interface access layer is used to receive the migration task creation request and verify whether there is an active migration task in the source PVC specified in the migration task creation request. The business logic layer is used to generate a K8s Job definition based on the temporary storage mode in the migration task configuration parameters when there is no active migration task in the source PVC. The task execution layer is used to create the data backup job and the data recovery job in the K8s cluster based on the K8s Job definition; The status monitoring layer is used to respond to the status query request of the front-end module by calling the K8s API Server to obtain the Job resource status.

[0010] In one possible implementation, the interface access layer is further configured to generate a unique task key based on the PVC namespace and the source PVC name, query the task status table in memory based on the unique task key, and determine whether there is an active migration task for the source PVC based on the task status table and K8s Job resources.

[0011] In one possible implementation, the status monitoring layer is configured to acquire the Job resource status based on a hybrid monitoring mechanism, wherein the hybrid monitoring mechanism is: Determine if the Watch connection is normal. If the Watch connection is normal, use the Watch API to monitor the Job resource status. If the Watch connection is abnormal, use polling mode to actively query the Job resource status at preset intervals.

[0012] In one possible implementation, the business logic layer is configured to listen to the Job resource status reported by the status monitoring layer. When the Job resource status is detected as a data backup Job successfully completed, the business logic layer is instructed to delete the source PVC, create the destination PVC, and create a data recovery Job.

[0013] Secondly, the present invention also provides a control method for a persistent storage data migration system based on a K8s cluster, comprising: The front-end module provides a graphical user interface to receive migration task configuration parameters input by the user. The backend module responds to the migration task creation request submitted by the frontend module, creates or selects an intermediate storage based on the migration task configuration parameters, and dynamically generates and sequentially drives a data backup job and a data recovery job through a template engine; wherein, the data backup job is used to synchronize the data of the source PVC to the intermediate storage, and the data recovery job is used to synchronize the data of the intermediate storage to a newly created destination PVC; The control backend module creates the data backup job and data recovery job in the K8s cluster and instantiates them into Pods by the job execution module; The backend module is controlled to respond to the status query request from the frontend module by calling the K8s API Server to obtain the Job resource status, and the Job resource status is fed back to the frontend module as the real-time status of the migration task; Based on the real-time status, the migration progress and migration log are displayed in the graphical user interface.

[0014] The beneficial effects of this invention are as follows: The persistent storage data migration system based on a Kubernetes cluster provided by this invention firstly configures the front-end module to provide a graphical user interface, receiving user-input migration task configuration parameters via a form. This abstracts complex operations that originally relied on deep command-line knowledge (such as kubectl, rsync) and underlying Kubernetes storage concepts into intuitive form filling and selection. This significantly reduces the technical threshold and complexity of operation and maintenance, enabling users without extensive expert knowledge to safely and correctly execute cross-storage migration tasks. Secondly, after the front-end module generates a migration task creation request, the back-end module is configured to create or select an intermediate storage based on the migration task configuration parameters, and dynamically generates and sequentially drives data backup and data recovery jobs through a template engine. Thirdly, it integrates the discrete and fragmented operation steps of traditional manual mode into a seamless automated workflow. This greatly improves the efficiency and reliability of migration operations, reduces manual intervention, and ensures the consistency and repeatability of the operation process.

[0015] Furthermore, during the migration process, an intermediate storage was created or selected, establishing a process of first performing a complete backup in the intermediate storage, and then synchronizing the data from the original PVC to the destination PVC. Based on the intermediate storage, a reliable data security mechanism can be implemented, avoiding the risk of data loss due to operational errors or accidents, and ensuring data integrity and business continuity. Simultaneously, the connection establishment process for the intermediate storage is automatically handled by the backend module, eliminating the need to concern oneself with specific mount points, access protocols, and other implementation details. This frees operations personnel from the complex details of underlying storage, improving the abstraction level of resource management and the universality of operations.

[0016] Furthermore, the real-time status of migration tasks comes directly from the Job resource status, eliminating the need to maintain an additional state storage engine. This simplifies the system architecture and ensures the accuracy of the status information.

[0017] In summary, this invention transforms the complex, high-risk, and highly specialized underlying implementation and maintenance work of traditional technical solutions into a simple, efficient, secure, and easy-to-manage platform-level service. It enables page-based, wizard-driven data migration between multiple Kubernetes storage services, thereby systematically solving the fundamental pain points of traditional technical solutions, significantly reducing the technical threshold and operational complexity of data migration between Kubernetes cluster storage services, and achieving secure, efficient, and one-click data migration. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram of an embodiment of the persistent storage data migration system based on a K8s cluster provided by the present invention; Figure 2 This is a schematic diagram of an embodiment of the configuration modal frame provided by the present invention; Figure 3 A schematic diagram of an embodiment of the graphical user interface provided by the present invention; Figure 4 A schematic diagram of an embodiment of the backend module provided by the present invention; Figure 5 This is a schematic diagram of an embodiment of the persistent storage data migration method based on a K8s cluster provided by the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this invention illustrate operations implemented according to some embodiments of the invention. It should be understood that the operations in the flowcharts may be implemented out of order, and steps without logical contextual relationships may be reversed or performed simultaneously. Furthermore, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] This invention provides a persistent storage data migration system and method based on a K8s cluster, which will be described below.

[0024] Before introducing specific implementation methods, let's first introduce some technical terms.

[0025] Kubernetes (K8s) is an open-source container orchestration engine from Google used to automate the deployment, scaling, and management of containerized applications. K8s provides a powerful cluster management platform capable of handling service discovery and load balancing, rolling updates, storage orchestration, self-healing, and other functions, making it a core infrastructure for implementing large-scale, highly available microservice architectures.

[0026] A Pod is the smallest unit of computing that can be created and deployed in Kubernetes, representing a single, running collection of processes within a cluster. A Pod contains one or more containers that share network and storage resources. These containers are scheduled and run on the same logical host, share the same IP address, port space, and storage volumes, and have a unified lifecycle.

[0027] A PersistentVolumeClaim (PVC) is a user's request for storage resources. A Pod creates a PVC to request persistent storage of a specific size and access mode. The Kubernetes system finds and binds a PV that meets the PVC's requirements. By mounting the PVC in its configuration, the Pod associates a specified file path within the container with the persistent PV, ensuring that data is not lost due to Pod restarts or deletions.

[0028] A PersistentVolume (PV) is a network storage resource in a Kubernetes cluster that is pre-configured by the administrator or dynamically provisioned through storage classes. PVs are cluster-level resources whose lifecycles are independent of any individual Pod. They abstract the details of the underlying storage (such as NFS, Ceph, cloud storage, etc.) and provide a unified storage interface for the cluster.

[0029] A Job is a workload resource in Kubernetes used to run batch processing tasks. It creates one or more Pods and ensures a specified number of Pods run successfully until the process terminates normally. Unlike long-running Deployments, Jobs stop after their tasks are completed, making them ideal for performing one-off, discrete data migration or processing tasks. In this embodiment of the invention, the Job is used as the core carrier for performing data migration operations. The system dynamically creates one or more Jobs in the backend, and the containers within their Pods specifically perform the data synchronization work from source storage to destination storage.

[0030] Rclone is a powerful open-source command-line tool designed for efficient data management and synchronization between cloud storage and local file systems. It is compatible with over 70 storage systems, covering mainstream object storage (such as Amazon S3, Google Cloud Storage, and Microsoft Azure Blob Storage), file storage, and standard protocol services (such as OpenStack Swift, SFTP, and FTP). Rclone's core features include cross-platform data migration, bidirectional synchronization, integrity verification, and transparent encryption.

[0031] Figure 1 This is a schematic diagram of an embodiment of the persistent storage data migration system based on a K8s cluster provided by the present invention, as shown below. Figure 1 As shown, the persistent storage data migration system 10 based on a K8s cluster includes: Front-end module 100 is configured to provide a graphical user interface for receiving migration task configuration parameters from user input via a form.

[0032] The backend module 200 is configured to respond to the migration task creation request submitted by the frontend module 100, create or select an intermediate storage based on the migration task configuration parameters, and dynamically generate and drive the data backup job and data recovery job in sequence through the template engine; wherein, the data backup job is used to synchronize the data of the source PVC to the intermediate storage, and the data recovery job is used to synchronize the data of the intermediate storage to a newly created destination PVC.

[0033] Specifically: When the intermediate storage is a temporary PVC, it needs to be created; when the intermediate storage is a remote storage configured by Rclone, it is only necessary to select the existing remote storage in the front end.

[0034] The template engine supports advanced features such as conditional statements, loop iterations, and variable substitution. It has good flexibility and scalability, can quickly adapt to new storage types and migration strategies, and greatly improves the maintainability of the system.

[0035] The Job execution module 300 consists of Pods instantiated in the K8s cluster for the data backup Job and the data recovery Job; The backend module 200 is also configured to respond to the status query request from the frontend module 100 by calling the K8s API Server to obtain the Job resource status, and feed back the Job resource status as the real-time status of the migration task to the frontend module 100. The front-end module 100 is also used to display migration progress and migration logs in a graphical user interface based on real-time status.

[0036] Specifically, the front-end module 100 is built using a modern web framework based on React, aiming to provide users with an intuitive, wizard-driven graphical interface to simplify the complex operations of migrating data to K8s persistent storage.

[0037] It should be noted that the form uses the controlled component pattern for form state management, with the state of all form fields being uniformly managed by React. Through complex interactive logic, the system can dynamically adjust form content based on user selections: when a user selects different temporary storage types, the system automatically switches to display the corresponding configuration items; when selecting a target storage class, the system automatically disables items in the option list that are the same as the source storage class to prevent configuration conflicts; when a user selects a pre-configured Rclone configuration from a dropdown list, the form automatically fills in all the detailed information of that configuration, including endpoint addresses, authentication information, and other key parameters, significantly improving user efficiency and configuration accuracy.

[0038] Since resources such as PVCs, Jobs, and Pods cannot have the same name, and it is best to have only one existing migration task corresponding to a PVC, the migration task resource needs to be deleted after the user confirms the completion of the migration task. This is to free up resources and to ensure that the next migration task can be created successfully.

[0039] Therefore, in a preferred embodiment of the present invention: the front-end module 100 also provides the ability to query migration task resources in a graphical user interface and receive user requests for resource cleanup of migration task resources.

[0040] Backend module 200 responds to the resource cleanup request submitted by frontend module 100, calls the K8s API Server to obtain the status of Job, PVC, and Pod resources related to the migration task, and feeds back the Job, PVC, and Pod resources to frontend module 100. After frontend module 100 confirms, it calls the K8s API Server to delete the Job, PVC, and Pod resources.

[0041] Compared with existing technologies, the persistent storage data migration system 10 based on a Kubernetes cluster provided in this embodiment of the invention firstly configures the front-end module 100 to provide a graphical user interface, receiving user-input migration task configuration parameters via a form. This abstracts complex operations that originally relied on deep command-line knowledge (such as kubectl, rsync) and underlying Kubernetes storage concepts into intuitive form filling and selection. This significantly reduces the technical threshold and complexity of operation and maintenance, enabling users without extensive expert knowledge to safely and correctly execute cross-storage migration tasks. Secondly, after the front-end module 100 generates a migration task creation request, the back-end module 200 is configured to create or select an intermediate storage based on the migration task configuration parameters, and dynamically generates and sequentially drives data backup and data recovery jobs through a template engine. Thirdly, it integrates the discrete and fragmented operation steps of traditional manual mode into a seamless automated workflow. This greatly improves the efficiency and reliability of migration operations, reduces manual intervention, and ensures the consistency and repeatability of the operation process.

[0042] Furthermore, during the migration process, an intermediate storage was created or selected, establishing a process of first performing a complete backup in the intermediate storage, and then synchronizing the data from the original PVC to the destination PVC. Based on the intermediate storage, a reliable data security mechanism can be implemented, avoiding the risk of data loss due to operational errors or accidents, and ensuring data integrity and business continuity. Simultaneously, the connection establishment process for the intermediate storage is automatically handled by the backend module, eliminating the need to concern oneself with specific mount points, access protocols, and other implementation details. This frees operations personnel from the complex details of underlying storage, improving the abstraction level of resource management and the universality of operations.

[0043] Furthermore, the real-time status of migration tasks comes directly from the Job resource status, eliminating the need to maintain an additional state storage engine. This simplifies the system architecture and ensures the accuracy of the status information.

[0044] In summary, the embodiments of the present invention transform the complex, high-risk, and highly specialized underlying implementation and maintenance work of traditional technical solutions into a simple, efficient, secure, and easy-to-manage platform-level service. It realizes page-based, wizard-driven data migration between K8s multi-storage services, thereby systematically solving the fundamental pain points of traditional technical solutions, significantly reducing the technical threshold and operational complexity of data migration between K8s cluster storage services, and achieving secure, efficient, and one-click data migration.

[0045] In some embodiments of the present invention, such as Figure 1 and Figure 2As shown, the graphical user interface includes a configuration modal 110, which includes a task basic information configuration sub-frame 111, a source storage configuration sub-frame 112, a destination storage configuration sub-frame 113, and a temporary storage configuration sub-frame 114. The task basic information configuration sub-frame 111 is used to receive the migration task name, task description information, PVC namespace, and source PVC name; The source storage configuration subframe 112 is used to receive the source storage class and source storage capacity.

[0046] Among them, the source storage configuration subframe 112 is in read-only mode and is used for information confirmation.

[0047] The destination storage configuration subframe 113 is used to receive the destination storage class and destination storage capacity.

[0048] It should be noted that, to prevent accidental operations, the destination storage class automatically filters and disables options that are the same as those in the source storage class.

[0049] The temporary storage configuration subframe 114 is used to select the temporary storage mode and generate temporary storage parameters based on the temporary storage mode; the temporary storage mode is either a temporary PVC or a remote storage mode. When the temporary storage mode is a temporary PVC, the temporary storage parameters include temporary storage services and temporary storage space. When the temporary storage mode is a remote storage mode, the temporary storage parameters include the remote path.

[0050] In a specific embodiment of the present invention, the temporary storage service is either a source storage service or a destination storage service. A source storage service refers to using storage-class-nfs, where the source PVC and temporary PVC reside on the same NFS storage backend. A destination storage service refers to using storage-class-ssd, where the source PVC and temporary PVC reside on the same SSD storage backend.

[0051] Specifically, in remote storage mode, you need to select and specify the remote path (such as Bucket name, HDFS storage path) from the pre-configured Rclone configuration list. After selecting the configuration, the graphical user interface will dynamically display detailed connection information (including Endpoint, authentication information, etc.).

[0052] In this embodiment of the invention, Rclone plays a core role in cross-storage data transfer. When a remote storage mode is selected, such as using an object storage bucket as a temporary data transfer station, the pre-configured Rclone access credentials are invoked. By executing Rclone commands, data from the source storage is first uploaded to the designated temporary storage area. After the upload is complete, the data is then downloaded from the transfer station to the destination storage. This mechanism effectively solves the problem of direct high-speed communication between source and destination storage services, ensuring the feasibility and efficiency of data transfer tasks across heterogeneous storage environments.

[0053] To further improve the flexibility and ease of operation of migration task configuration parameters during the configuration process, in some embodiments of the present invention, such as... Figure 2 As shown, the configuration modal 110 also includes a clear button 115 and a submit button 116; The Clear button 115 responds to user clicks, clears the configuration in the form, and restores it to its initial state. The submit button 116 responds to the user's click action, submitting the migration task configuration parameters to the backend module to trigger the execution of the data migration task.

[0054] This embodiment of the invention enables one-click clearing of form configurations and creation and execution of migration tasks by setting a clear button 115 and a submit button 116. This supports quick reconfiguration and ensures the controllability and reliability of migration task submission.

[0055] In some embodiments of the present invention, such as Figure 1 and Figure 3 As shown, the graphical user interface also includes a task status monitoring interface 120, which includes a task overview area 121, a task progress area 122, and a log scrolling display area 123. The task overview area 121 is used to display the migration task configuration parameters in a list format.

[0056] Preferably, to improve the simplicity of the display, the task overview area 121 may only display the key migration task configuration parameters. The keyness of the migration task configuration parameters may be determined based on experience or pre-set rules.

[0057] Task progress area 122 is used to display multiple migration steps of a migration task based on a vertical step bar component, and to determine and highlight the current migration step based on the real-time status.

[0058] Specifically, migration steps may include data backup, data download, etc. If the current migration step is determined to be data backup based on the real-time status, then the data backup step will be highlighted.

[0059] The log scrolling display area 123 is used to display the logs generated during the Job execution process in real time.

[0060] The logs provide users with complete operation tracking and troubleshooting support.

[0061] This embodiment of the invention enables the configuration and monitoring of data migration tasks by configuring the modal box 110 and the task status monitoring interface 120, providing users with a full-process operation experience of task submission and status viewing.

[0062] To improve the granularity of querying migration task progress, in some embodiments of the present invention, the polling strategy of the task status monitoring interface is as follows: When the migration task is in different non-final states, polling is performed at different interval frequencies; when the migration task is in the final state, polling is stopped.

[0063] Specifically, non-final states include, but are not limited to, Running and Pending. Pending means that the task has been submitted but is still preparing for execution and has not yet actually started running. Running means that the preparation work required for the task has been completed, the containers within the Pod have been successfully created, and at least one is executing.

[0064] The final state is either Succeeded (transfer successful) or Failed (transfer failed).

[0065] By controlling the polling frequency, this embodiment of the invention can ensure that users receive timely and accurate status feedback while minimizing system load.

[0066] In terms of polling control, the system implements an on-demand activation mechanism, which binds the polling cycle to the task status monitoring interface. Polling is only started when the task status monitoring interface is active / open, and polling stops immediately if the task status monitoring interface is closed.

[0067] To avoid task conflicts and data consistency issues caused by concurrent operations, in some embodiments of the present invention, the front-end module 100 is further configured as follows: When responding to a user's instruction to initiate a migration operation, the status query interface of the backend module 200 is called to obtain the Job resource status. If the Job resource status indicates that there is an associated active migration task, the configuration modal 110 is disabled and the task status monitoring interface 120 is entered. If the Job resource status indicates that there is no associated active migration task, the configuration modal 110 is opened to receive user input.

[0068] In other words, when an active migration task is detected, the user is automatically guided to the task status monitoring interface 120, effectively preventing duplicate task submissions for the same PVC and ensuring the uniqueness of the migration operation. If the detection result shows no relevant migration task, the user is given access to the configuration modal 110, allowing them to create a new migration task. This mechanism establishes a PVC-level migration task mutex from the source, completely avoiding task conflicts and data consistency issues caused by concurrent operations, further ensuring the reliability of the migration task process.

[0069] To further enhance the reliability and maintainability of the system, a unique delayed cleanup strategy is adopted in some embodiments of the present invention. That is, temporary storage data, job execution records, and related intermediate resources generated during the migration task execution are not automatically deleted immediately after the task is completed, but are maintained in a persistent storage state. Only when the user explicitly confirms the migration result on the task status monitoring interface 120 and manually triggers the cleanup operation will the system call the backend cleanup interface to safely reclaim the relevant resources.

[0070] This design forms a reliable data security protection network. When unexpected failures occur during the migration process and PVC data becomes abnormal, the complete data backup stored in the temporary space can serve as a basis for recovery, effectively preventing the risk of source data loss. At the same time, retaining complete job execution records provides comprehensive data support for troubleshooting and operation auditing, further enhancing the system's reliability and maintainability.

[0071] In some embodiments of the present invention, such as Figure 4 As shown, the backend module 200 includes an interface access layer 210, a business logic layer 220, a task execution layer 230, and a status monitoring layer 240. The interface access layer 210 is used to receive migration task creation requests and verify whether there is an active migration task for the source PVC specified in the migration task creation request.

[0072] Specifically, retrieve the Job and Pod resources corresponding to the source PVC specified in the migration task creation request, and determine whether there are active migration tasks.

[0073] The business logic layer 220 is used to generate a K8s Job definition based on the temporary storage mode in the migration task configuration parameters when there is no active migration task in the source PVC. Task execution layer 230 is used to create data backup jobs and data recovery jobs in the K8s cluster based on K8s Job definitions; The status monitoring layer 240 is used to respond to status query requests from the front-end module by calling the K8s API Server to obtain the Job resource status.

[0074] In this embodiment of the invention, a secondary conflict check is performed in the interface access layer 210 of the backend module 200. This check is performed before the migration task creation request arrives at the backend module 200 and begins processing. This avoids task conflicts occurring within the time window from when the frontend module 100 receives the migration task creation request to when it submits the migration task creation request. This further prevents duplicate task submissions for the same PVC and ensures the uniqueness of the migration operation.

[0075] Specifically, the execution flow of the business logic layer 220 includes: selecting the appropriate processing flow according to the migration type: for temporary PVCs, the system verifies the validity of the source and target storage classes and creates a temporary PVC; for remote storage mode, the system obtains storage authentication information from K8s Secrets and establishes a remote connection.

[0076] The resource status management in this embodiment of the invention is entirely based on the native Kubernetes mechanism and does not rely on an additional persistent storage layer. All task status, execution progress, and historical records are persistently stored through the status fields of Kubernetes Job resources and associated Pod logs, which simplifies the system architecture and ensures the consistency between status information and the actual resource status of the cluster.

[0077] It's important to note that all temporary resources (including Jobs, Pods, and temporary storage resources) are identified and managed using a unified tagging system. During task execution, the system dynamically creates the necessary resources and establishes a correspondence between them and the migration task through resource tags. After the task is completed, the system employs a user-confirmed resource cleanup mechanism. All temporary resources are not automatically cleaned up but are retained in the cluster awaiting explicit user instructions. This design ensures that all intermediate data and execution records are fully preserved before the user confirms the migration results, providing ample assurance for data verification and troubleshooting.

[0078] Since task conflicts have a significant adverse impact on data migration tasks, in order to further ensure that task conflicts do not occur, in some embodiments of the present invention, the interface access layer 210 is also used to generate a unique task key based on the PVC namespace and the source PVC name, and query the task status table in memory based on the unique task key, and determine whether there is an active migration task in the source PVC based on the task status table and K8s Job resources.

[0079] This invention effectively prevents the risks of task conflicts and data inconsistencies by using a dual verification mechanism based on task status tables and Job and Pod resources. It ensures that only one active migration task can be performed on the same PVC at the same time, thus ensuring the accuracy and reliability of the migration task.

[0080] Furthermore, in addition to the task status table, K8s Job resources are also introduced to determine whether there are active migration tasks in the source PVC. This can prevent the technical problem of being unable to determine the existence of active migration tasks due to the loss of the task status table in memory caused by abnormal backend restarts, and further improve the accuracy and reliability of determining whether there are active migration tasks in the source PVC.

[0081] In some embodiments of the present invention, the status monitoring layer 240 is configured to obtain the Job resource status based on a hybrid monitoring mechanism, wherein the hybrid monitoring mechanism is as follows: Determine if the Watch connection is normal. If the Watch connection is normal, use the Watch API to monitor the Job resource status. If the Watch connection is abnormal, use polling mode to actively query the Job resource status at preset intervals.

[0082] This invention innovatively implements a hybrid monitoring strategy combining Watch API and polling queries. Under normal network conditions, by establishing a persistent Watch connection with the Kubernetes API Server, it monitors the status change events of relevant Job resources in real time, obtaining status updates in an event-driven manner to achieve millisecond-level status awareness and significantly reduce system resource consumption. When an abnormal Watch connection or event stream interruption is detected, the system automatically and seamlessly switches to polling degradation mode, actively querying the Job resource status at preset time intervals to ensure reliable tracking of task execution progress under various abnormal conditions.

[0083] In a specific embodiment of the present invention, the business logic layer 220 is configured to listen to the Job resource status reported by the status monitoring layer. When it hears that the Job resource status is a data backup Job successfully completed, it instructs the task execution layer to delete the source PVC, create the destination PVC, and create a data recovery Job.

[0084] This invention implements an intelligent process control mechanism based on the status of Job resources. By monitoring changes in the status of Job resources, it can automatically trigger subsequent operation processes at specific status nodes. For example, after detecting a successful data backup Job, it automatically performs source PVC cleanup and creates a data download Job. This automated process transition based on status events not only improves the execution efficiency of the migration process but also ensures the security of data operations through rigorous status judgment logic.

[0085] To further improve the success rate of migration tasks, this embodiment of the invention also sets up a multi-level error recovery mechanism. Specifically, during task execution, the system continuously monitors the execution status of each step, and immediately triggers the corresponding recovery process upon detecting a failure. For migration task failures, the system attempts to restore the original configuration through the recoverOriginalPVC function; for network transmission errors, the system automatically retryes or downgrades processing based on the error type. The retry strategy adopts an exponential backoff algorithm, ensuring successful task execution while avoiding excessive pressure on the system.

[0086] The backend module 200 proposed in this embodiment of the invention provides a simple, reliable, and easy-to-maintain solution for K8s storage data migration through an innovative stateless architecture and hybrid monitoring mechanism, fully meeting the stringent requirements of enterprise-level applications in terms of reliability, security, and performance.

[0087] This invention implements full lifecycle management of PVC data migration through a front-end and back-end separation architecture. The front-end is built with a visual operation interface based on the React framework, providing users with task configuration and progress monitoring functions; the back-end uses the Gin framework combined with Kubernetes Jobs to implement a stateless task processing architecture.

[0088] In a specific embodiment of the present invention, the specific implementation process of the persistent storage data migration system 10 based on a K8s cluster is as follows: Task creation phase: Users configure migration task parameters through a graphical user interface, and the system calls the corresponding backend interface to create the migration task based on the intermediate storage type (temporary PVC or Rclone remote storage).

[0089] Data backup phase: The system first creates a data backup job, which completely backs up the source PVC data to the intermediate storage space. The backend tracks the backup progress in real time by monitoring the job resource status.

[0090] Storage switchover phase: Once the data backup job is successfully completed, the system automatically deletes the source PVC and creates a new destination PVC according to the configuration.

[0091] Data recovery phase: The system creates a data recovery job, downloads the backup data from intermediate storage to the newly created destination PVC, and completes the data migration.

[0092] Resource retention phase: After the migration task is completed, all intermediate storage data and related job resources are retained in the cluster, waiting for the user to confirm the migration results before being deleted on the front end.

[0093] Task Confirmation Phase: When a user submits a migration task request or initiates a new task on an existing PVC with a migration task, the system automatically brings up the task status monitoring interface, displaying the detailed progress of the current or existing migration tasks. Users can view the task execution status in real time through this interface and confirm the completion of the migration task based on actual needs.

[0094] Resource cleanup phase: After the user confirms the migration is complete, the cleanup operation is triggered through the task status monitoring interface. The system calls the backend interface to delete intermediate storage data and associated Job resources.

[0095] This process ensures the safety and reliability of the data migration process through a phased task execution strategy and a user confirmation mechanism, effectively preventing the risk of data loss due to misoperation, while providing users with complete operational visibility and process control.

[0096] Based on the above specific embodiments, it can be seen that the backend module 200 requires a status query interface, a migration task creation interface for intermediate storage as a temporary PVC, a migration task creation interface for intermediate storage as a remote Rclone configuration storage, and a migration task cleanup interface. The frontend module 100 can implement functions such as status query, migration task creation, and migration task cleanup by calling these interfaces of the backend module 200.

[0097] It should be noted that: to define and configure migration tasks in Kubernetes, a Yaml template design is required for the Job resource, specifically the template engine in frontend module 100. To determine the specific execution logic of the migration task, a container image needs to be created for the Job resource. This image encapsulates all complex business logic and judgments, allowing the Yaml template to simply call the container to complete complex tasks. The ultimate goal of the Job resource image design is to enable a single image to support both temporary PVC and Rclone remote storage migrations, and to handle both data backup and data recovery phases within the migration task. After the Job resource creates the corresponding execution container, it executes the corresponding processing logic based on the migration type and migration stage parameters in the environment variables. Specifically: When the migration type uses a temporary PVC as intermediate storage and the migration phase is data backup, the following logic is executed: First, all migration task parameters are obtained based on environment variables. Then, the configuration information of the source PVC is retrieved based on the PVC name to check if the source PVC is being used by other containers. If it is, the migration task fails, ensuring that data is not tampered with during the migration process. After confirming that the source PVC is only being used by itself, the data upload operation from the source PVC to the temporary PVC begins. Since the Job resource has mounted both the source and temporary PVCs, the migration process can use the rsync tool to preserve all attributes such as permissions, owner, and timestamps during data backup between the two PVC mount points. After the data backup operation is completed, the Job returns success.

[0098] When the migration type uses a temporary PVC as intermediate storage and the migration phase is data download, the following logic is executed: First, all migration task parameters are obtained based on environment variables. Then, the configuration information of the temporary PVC is retrieved based on the PVC name to check if the temporary PVC is being used by other containers. If it is, the migration task fails, ensuring that data is not tampered with during the migration process. After confirming that the temporary PVC is only being used by itself, the data download operation from the temporary PVC to the destination PVC begins. Since the Job resource has mounted both the temporary and destination PVCs, the migration process can use the rsync tool to preserve all attributes such as permissions, owner, and timestamps during data backup between the two PVC mount points. After completing the data backup operation, the Job returns success.

[0099] When the migration type uses a temporary PVC as intermediate storage and the migration phase is data recovery, the following logic is executed: First, all migration task parameters are obtained based on environment variables. Then, the configuration information of the temporary PVC is retrieved based on the PVC name to check if the temporary PVC is being used by other containers. If it is, the migration task fails, ensuring that data is not tampered with during the migration process. After confirming that the temporary PVC is only being used by itself, the data download operation from the temporary PVC to the newly created source PVC begins. Since the Job resource has mounted both the temporary PVC and the newly created source PVC, the migration process can use the rsync tool to preserve all attributes such as permissions, owner, and timestamps during data backup between the two PVC mount points. After the data backup operation is completed, the Job returns success.

[0100] When the migration type is configuring remote storage using Rclone, and the migration phase is data backup, the following logic is executed: First, all migration task parameters are obtained based on environment variables. Then, the configuration information of the source PVC is retrieved based on the PVC name to check if the source PVC is being used by other containers. If it is, the migration task fails, ensuring that data is not tampered with during the migration process. After confirming that the source PVC is only used by itself, the data upload operation from the source PVC to the remote storage begins. Since the Job resource has already mounted the source PVC, the source PVC mount point is compressed and packaged during the migration process to ensure that all attributes such as data file permissions, owner, and timestamp are consistent. Then, the data is copied to the specified path on the remote storage using the rclone command-line tool. After the data backup operation is completed, the Job returns success.

[0101] When the migration type is "Configure remote storage using Rclone" and the migration phase is data download, the following logic is executed: First, the parameters of all migration tasks are obtained based on environment variables. Then, the data is downloaded from the remote storage to the destination PVC. Since the Job resource is already mounted to the destination PVC, the migration process uses the rclone command-line tool to download the backup file from the specified path on the remote storage to the destination PVC mount point and decompresses it. After completing the data backup operation, the Job returns success.

[0102] When the migration type is "Use Rclone to configure remote storage" and the migration phase is data recovery, the following logic is executed: First, the parameters of all migration tasks are obtained based on environment variables. Then, the data is downloaded from the remote storage to the newly created source PVC. Since the Job resource is already mounted to the newly created source PVC, the migration process uses the rclone command-line tool to download the backup files from the specified path on the remote storage to the mount point of the newly created source PVC and decompresses them. After completing the data backup operation, the Job returns success.

[0103] In summary, the persistent storage data migration system based on a K8s cluster provided in this embodiment of the invention is designed for K8s heterogeneous storage environments. It can efficiently and securely achieve data migration between different storage services, simplify operation and maintenance, and improve the reliability and efficiency of data migration, thereby better supporting the persistent storage management needs of enterprise-level containerized applications.

[0104] Correspondingly, this embodiment of the invention also provides a control method for a persistent storage data migration system based on a Kubernetes cluster, applicable to persistent storage data migration systems based on Kubernetes clusters in any of the above embodiments, such as... Figure 5 As shown, the control method for a persistent storage data migration system based on a K8s cluster includes: S501: Receive migration task configuration parameters input by the user based on the graphical user interface provided by the front-end module; S502: The backend module responds to the migration task creation request submitted by the frontend module, creates or selects an intermediate storage based on the migration task configuration parameters, and dynamically generates and drives the data backup job and data recovery job in sequence through the template engine; wherein, the data backup job is used to synchronize the data of the source PVC to the intermediate storage, and the data recovery job is used to synchronize the data of the intermediate storage to a newly created destination PVC. S503, the control backend module creates the data backup job and data recovery job in the K8s cluster and instantiates them into Pods by the job execution module; S504. Control the backend module to respond to the status query request of the frontend module by calling the K8s API Server to obtain the Job resource status, and feed back the Job resource status as the real-time status of the migration task to the frontend module. S505: Display migration progress and migration logs in a graphical user interface based on real-time status.

[0105] It should be noted that the control method of the persistent storage data migration system based on K8s cluster provided in the above embodiments can realize the technical solutions described in the above embodiments of the persistent storage data migration system based on K8s cluster. The principles or specific implementation details of the above steps can be found in the corresponding content of the above embodiments of the persistent storage data migration system based on K8s cluster, and will not be elaborated here.

[0106] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0107] The above provides a detailed description of a persistent storage data migration system and method based on a K8s cluster provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A persistent storage data migration system based on a Kubernetes cluster, characterized in that, include: The front-end module is configured to provide a graphical user interface for receiving migration task configuration parameters input by the user via a form. The backend module is configured to respond to the migration task creation request submitted by the frontend module, create or select an intermediate storage based on the migration task configuration parameters, and dynamically generate and sequentially drive a data backup job and a data recovery job through a template engine; wherein, the data backup job is used to synchronize the data of the source PVC to the intermediate storage, and the data recovery job is used to synchronize the data of the intermediate storage to a newly created destination PVC; The Job execution module consists of Pods instantiated from the data backup Job and the data recovery Job in the K8s cluster; The backend module is also configured to respond to the status query request from the frontend module by calling the K8s API Server to obtain the Job resource status, and to feed back the Job resource status as the real-time status of the migration task to the frontend module. The front-end module is also used to display migration progress and migration logs in the graphical user interface based on the real-time status.

2. The persistent storage data migration system based on a Kubernetes cluster according to claim 1, characterized in that, The graphical user interface includes a configuration modal, which includes a task basic information configuration sub-frame, a source storage configuration sub-frame, a destination storage configuration sub-frame, and a temporary storage configuration sub-frame. The task basic information configuration sub-frame is used to receive the migration task name, task description information, PVC namespace and source PVC name; The source storage configuration subframe is used to receive the source storage type and source storage capacity; The destination storage configuration subframe is used to receive the destination storage type and destination storage capacity; The temporary storage configuration sub-frame is used to select a temporary storage mode and generate temporary storage parameters based on the temporary storage mode; The temporary storage mode is either a temporary PVC or a remote storage mode; Wherein, when the temporary storage mode is a temporary PVC, the temporary storage parameters include temporary storage service and temporary storage space; when the temporary storage mode is a remote storage mode, the temporary storage parameters include remote path.

3. The persistent storage data migration system based on a Kubernetes cluster according to claim 1, characterized in that, The graphical user interface also includes a task status monitoring interface, which includes a task overview area, a task progress area, and a log scrolling display area. The task overview area is used to display the migration task configuration parameters in list form; The task progress area is used to display multiple migration steps of the migration task based on the vertical step bar component, and to determine and highlight the current migration step based on the real-time status. The log scrolling display area is used to display the logs generated during the Job execution process in real time.

4. The persistent storage data migration system based on a Kubernetes cluster according to claim 3, characterized in that, The polling strategy for the task status monitoring interface is as follows: When the migration task is in different non-final states, different polling intervals are used; when the migration task is in the final state, polling is stopped.

5. The persistent storage data migration system based on a Kubernetes cluster according to claim 1, characterized in that, The front-end module is also configured as follows: When responding to a user's instruction to initiate a migration operation, the status query interface of the backend module is called to obtain the Job resource status. If the Job resource status indicates that there is an associated active migration task, the configuration modal is disabled and the task status monitoring interface is entered. If the Job resource status indicates that there is no associated active migration task, the configuration modal is opened to receive user input.

6. The persistent storage data migration system based on a Kubernetes cluster according to claim 2, characterized in that, The backend module includes an interface access layer, a business logic layer, a task execution layer, and a status monitoring layer; The interface access layer is used to receive the migration task creation request and verify whether there is an active migration task in the source PVC specified in the migration task creation request. The business logic layer is used to generate a K8s Job definition based on the temporary storage mode in the migration task configuration parameters when there is no active migration task in the source PVC. The task execution layer is used to create the data backup job and the data recovery job in the K8s cluster based on the K8s Job definition; The status monitoring layer is used to respond to the status query request of the front-end module by calling the K8s API Server to obtain the Job resource status.

7. The persistent storage data migration system based on a Kubernetes cluster according to claim 6, characterized in that, The interface access layer is also used to generate a unique task key based on the PVC namespace and the source PVC name, query the task status table in memory based on the unique task key, and determine whether there is an active migration task for the source PVC based on the task status table and K8s Job resources.

8. The persistent storage data migration system based on a K8s cluster according to claim 6, characterized in that, The status monitoring layer is configured to obtain the Job resource status based on a hybrid monitoring mechanism, which is as follows: Determine if the Watch connection is normal. If the Watch connection is normal, use the Watch API to monitor the Job resource status. If the Watch connection is abnormal, use polling mode to actively query the Job resource status at preset intervals.

9. The persistent storage data migration system based on a K8s cluster according to claim 6, characterized in that, The business logic layer is configured to listen to the Job resource status reported by the status monitoring layer. When it hears that the Job resource status is a data backup Job successfully completed, it instructs the task execution layer to delete the source PVC, create the destination PVC, and create a data recovery Job.

10. A control method for a persistent storage data migration system based on a Kubernetes cluster, characterized in that, include: The front-end module provides a graphical user interface to receive migration task configuration parameters input by the user. The backend module responds to the migration task creation request submitted by the frontend module, creates or selects an intermediate storage based on the migration task configuration parameters, and dynamically generates and sequentially drives a data backup job and a data recovery job through a template engine; wherein, the data backup job is used to synchronize the data of the source PVC to the intermediate storage, and the data recovery job is used to synchronize the data of the intermediate storage to a newly created destination PVC; The control backend module creates the data backup job and data recovery job in the K8s cluster and instantiates them into Pods by the job execution module; The backend module is controlled to respond to the status query request from the frontend module by calling the K8s API Server to obtain the Job resource status, and the Job resource status is fed back to the frontend module as the real-time status of the migration task; Based on the real-time status, the migration progress and migration log are displayed in the graphical user interface.

Images