Abnormality recovery method and device of CSI driver, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a CSI (Channel State Information) driver exception recovery method and device, electronic equipment and a storage medium, and relates to the technical field of storage, and the method comprises the following steps: detecting a connection state between a persistent volume on a computing node and a first iSCSI target on a remote iSCSI storage device through a CSI driver, and if the connection state is detected to be exceptional, sending the persistent volume to the remote iSCSI storage device; if the first iSCSI target is migrated, the current state information of the first iSCSI target is inquired through the CSI driver, if the current state information represents that the first iSCSI target is migrated, connection information corresponding to a migrated second iSCSI target is obtained through the CSI driver, then the computing node is controlled to disconnect a first connection session with the first iSCSI target, and the second iSCSI target is migrated according to the connection information. And establishing a second connection session between the computing node and the second iSCSI target, and finally updating the connection information corresponding to the persistent volume through the CSI driver, thereby realizing automatic sensing of iSCSI target migration and realizing non-sensing connection recovery.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a CSI-driven anomaly recovery method, a CSI-driven anomaly recovery device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In Kubernetes-based containerized environments, container storage interfaces have become the standard way to access external storage. iSCSI, as a common block storage protocol, provides persistent storage for applications in Kubernetes clusters through a CSI (Container Storage Interface Driver). In this architecture, storage resources are provided by remote iSCSI (Internet Small Computer System Interface) targets.

[0003] In actual operation and maintenance, iSCSI targets may migrate due to load balancing, hardware failure, or planned maintenance. After the target migration, the original iSCSI connection will become invalid, causing the corresponding persistent volume to enter an abnormal state. In existing technologies, the main way to recover from such anomalies is through manual intervention or restarting the application that mounted the storage volume. Manual intervention is inefficient and error-prone, while restarting the application will cause service interruption, failing to meet the requirements of high-availability applications. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for abnormal recovery of CSI drivers, in order to solve or partially solve the problem that CSI drivers cannot automatically detect iSCSI target migration and achieve seamless connection recovery for clients in containerized storage scenarios.

[0005] This invention discloses an anomaly recovery method for a CSI driver, applied to a Kubernetes cluster. The Kubernetes cluster has a CSI driver installed, and the compute nodes in the Kubernetes cluster are mounted with remote iSCSI storage devices via the CSI driver. The method includes: The connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device is detected by the CSI driver. If an abnormal connection status is detected, the current status information of the first iSCSI target is queried by the CSI driver. If the current status information indicates that the first iSCSI target has been migrated, then the connection information corresponding to the migrated second iSCSI target is obtained through the CSI driver; The system controls the compute node to disconnect the first connection session with the first iSCSI target, and establishes a second connection session between the compute node and the second iSCSI target according to the connection information. The connection information corresponding to the persistent volume is updated through the CSI driver.

[0006] In some feasible implementations, querying the current status information of the first iSCSI target through the CSI driver includes: The CSI driver detects whether the persistent volume of the compute node is configured with a multipath unit, which is used to provide the compute node with multiple paths for accessing iSCSI targets; If the persistent volume is not configured with the multipath unit, the detection logic is terminated. If the persistent volume is configured with a multipath unit, the current status information of the first iSCSI target is queried through the CSI driver.

[0007] In some feasible implementations, the storage service client manages several of the remote iSCSI storage devices, and querying the current status information of the first iSCSI target includes: Generate a status query request for the first iSCSI target, wherein the status query request includes at least the identifier of the first iSCSI target; Call the application programming interface (API) that communicates with the storage service client, and send the status query request to the storage service client through the API; Receive the current status information corresponding to the identifier returned by the storage service client.

[0008] In some feasible implementations, obtaining the connection information corresponding to the migrated second iSCSI target through the CSI driver includes: Receive response data returned by the storage server through the application programming interface; The connection information of the second iSCSI target is extracted from the response data.

[0009] In some feasible implementations, controlling the compute node to disconnect the first connection session with the first iSCSI target includes: Generate a logout command for the first iSCSI target; In response to the logout command, a logout operation is performed on the first iSCSI target.

[0010] In some feasible implementations, the connection information includes at least a network address, port number, and authentication information. Establishing a second connection session between the computing node and the second iSCSI target according to the connection information includes: Generate connection instructions for the second iSCSI target; In response to the connection command, the network address, the port number, and the authentication information are used to perform a login operation and establish a second connection session between the computing node and the second iSCSI target.

[0011] In some feasible implementations, detecting the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device via the CSI driver includes: The CSI driver periodically detects the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device at preset time intervals.

[0012] This invention also discloses an anomaly recovery device for a CSI driver, applied to a Kubernetes cluster, wherein the Kubernetes cluster has a CSI driver installed, and the compute nodes in the Kubernetes cluster are mounted with remote iSCSI storage devices via the CSI driver; wherein the device includes: The status detection module located in the CSI driver is used to detect the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device. If an abnormal connection status is detected, the current status information of the first iSCSI target is queried through the CSI driver. The information acquisition module located in the CSI driver is used to acquire the connection information corresponding to the migrated second iSCSI target if the current status information indicates that the first iSCSI target has been migrated. The connection processing module located in the CSI driver is used to control the compute node to disconnect the first connection session with the first iSCSI target, and to establish a second connection session between the compute node and the second iSCSI target according to the connection information; The information update module located in the CSI driver is used to update the connection information corresponding to the persistent volume.

[0013] In some feasible implementations, the state detection module is specifically used for: The CSI driver detects whether the persistent volume of the compute node is configured with a multipath unit, which is used to provide the compute node with multiple paths for accessing iSCSI targets; If the persistent volume is not configured with the multipath unit, the detection logic is terminated. If the persistent volume is configured with a multipath unit, the current status information of the first iSCSI target is queried through the CSI driver.

[0014] In some feasible implementations, the storage service client manages several of the aforementioned remote iSCSI storage devices, and the status detection module is specifically used for: Generate a status query request for the first iSCSI target, wherein the status query request includes at least the identifier of the first iSCSI target; Call the application programming interface (API) that communicates with the storage service client, and send the status query request to the storage service client through the API; Receive the current status information corresponding to the identifier returned by the storage service client.

[0015] In some feasible implementations, the information acquisition module is specifically used for: Receive response data returned by the storage server through the application programming interface; The connection information of the second iSCSI target is extracted from the response data.

[0016] In some feasible implementations, the information update module is specifically used for: Generate a logout command for the first iSCSI target; In response to the logout command, a logout operation is performed on the first iSCSI target.

[0017] In some feasible implementations, the connection information includes at least a network address, port number, and authentication information, and the information update module is specifically used for: Generate connection instructions for the second iSCSI target; In response to the connection command, the network address, the port number, and the authentication information are used to perform a login operation and establish a second connection session between the computing node and the second iSCSI target.

[0018] In some feasible implementations, the state detection module is specifically used for: The CSI driver periodically detects the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device at preset time intervals.

[0019] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method described in the embodiments of the present invention.

[0020] This invention also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.

[0021] The embodiments of the present invention have the following advantages: In this embodiment of the invention, the technology is applied to a Kubernetes cluster. The Kubernetes cluster has a CSI driver installed, and the compute nodes in the Kubernetes cluster are mounted with remote iSCSI storage devices via the CSI driver. During the migration of the remote storage target, the CSI driver can detect the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device. If an abnormal connection status is detected, the CSI driver queries the current status information of the first iSCSI target. If the current status information indicates that the first iSCSI target has been migrated, the CSI driver obtains the connection information corresponding to the migrated second iSCSI target. Next, the compute node is controlled to disconnect the first connection session with the first iSCSI target, and a second connection session is established between the compute node and the second iSCSI target according to the connection information. Finally, the connection information corresponding to the persistent volume is updated through the CSI driver. Thus, by integrating a full-link automated recovery mechanism of active detection, status verification and connection reconstruction at the CSI driver layer, it is possible to achieve business-agnostic recovery of storage connections without the upper-layer application Pod's awareness in the iSCSI storage target migration scenario. This significantly reduces manual intervention and business interruption caused by storage infrastructure maintenance or failover, and improves the overall reliability and operation and maintenance automation level of containerized stateful services. Attached Figure Description

[0022] Figure 1 This is a flowchart of the steps of a CSI-driven exception recovery method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the system architecture provided in the embodiments of the present invention; Figure 3 This is a structural block diagram of a CSI-driven anomaly recovery device provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] As an example, in actual operation and maintenance, iSCSI targets may migrate due to load balancing, hardware failure, or planned maintenance. After the target migration, the original iSCSI connection will become invalid, causing the corresponding persistent volume to enter an abnormal state. In existing technologies, the main way to recover from such anomalies is through manual intervention or restarting the application that mounted the storage volume. Manual intervention is inefficient and error-prone, while restarting the application will cause service interruption, failing to meet the requirements of high-availability applications.

[0025] In this invention, the application is to a Kubernetes cluster. The Kubernetes cluster has a CSI driver installed, and the compute nodes in the Kubernetes cluster are mounted with remote iSCSI storage devices via the CSI driver. During the migration of the remote storage target, the CSI driver can detect the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device. If an abnormal connection status is detected, the CSI driver queries the current status information of the first iSCSI target. If the current status information indicates that the first iSCSI target has been migrated, the CSI driver obtains the connection information corresponding to the migrated second iSCSI target. Next, the compute node is controlled to disconnect the first connection session with the first iSCSI target, and a second connection session is established between the compute node and the second iSCSI target according to the connection information. Finally, the connection information corresponding to the persistent volume is updated through the CSI driver. Thus, by integrating a full-link automated recovery mechanism of active detection, status verification and connection reconstruction at the CSI driver layer, it is possible to achieve business-agnostic recovery of storage connections without the upper-layer application Pod's awareness in the iSCSI storage target migration scenario. This significantly reduces manual intervention and business interruption caused by storage infrastructure maintenance or failover, and improves the overall reliability and operation and maintenance automation level of containerized stateful services.

[0026] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical features involved in the embodiments of the present invention are explained and described below: iSCSI (Internet Small Computer System Interface) is a storage technology based on the Internet and the SCSI-3 protocol. The iSCSI protocol transmits SCSI commands through a TCP / IP network, extending the traditional SCSI technology to the Internet environment and enabling networked access and management of storage resources. This protocol allows computer systems to access remote storage devices through IP networks as if they were local hard drives.

[0027] In iSCSI, a Target refers to a device or software that acts as a storage resource provider. Within the iSCSI architecture, a Target typically refers to a device or server that possesses physical storage media (such as hard drives, SSDs, etc.) and is capable of providing storage services over a network (usually an IP network). These Target devices can receive requests from the iSCSI Initiator and perform corresponding storage operations, such as data read / write and volume management.

[0028] CSI (Container Storage Interface) is a standardized interface for interaction between container orchestration systems and external storage systems. CSI is designed to allow third-party storage providers to develop standards-compliant plugins, enabling seamless integration of these storage solutions into Kubernetes clusters.

[0029] Kubernetes (K8s) is an abbreviation of 'ubernete', where '8' is used instead of the eight characters in the name. It is an open-source system for managing containerized applications across multiple hosts in a cloud platform, with the goal of making the deployment of containerized applications simple and efficient.

[0030] In this embodiment of the invention, an application Pod specifically refers to an application Pod that has mounted a storage volume via CSI.

[0031] Reference Figure 1 This document illustrates a flowchart of a CSI driver anomaly recovery method provided in an embodiment of the present invention, applied to a Kubernetes cluster. The Kubernetes cluster has a CSI driver installed, and the compute nodes in the Kubernetes cluster are connected to remote iSCSI storage devices via the CSI driver. Specifically, the method may include the following steps: Step 101: The connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device is detected by the CSI driver. If an abnormal connection status is detected, the current status information of the first iSCSI target is queried by the CSI driver. In this embodiment of the invention, it can be executed in a containerized cloud-native environment, specifically, it can be applied to a Kubernetes cluster, in which a CSI driver corresponding to the storage system can be pre-installed and configured. The Kubernetes cluster contains at least one or more compute nodes, which are physical machines or virtual machines actually running containerized application workloads. Optionally, these compute nodes, through the CSI driver, mount and use remote iSCSI storage devices based on the iSCSI protocol, thereby providing persistent storage capabilities for stateful applications within the cluster.

[0032] Furthermore, remote iSCSI storage devices can be provided by cloud storage service providers, and logically they can appear as a cloud hard drive or block storage volume. Access to this storage device is achieved through one or more iSCSI targets. The iSCSI target is the access endpoint for the storage device on the network; compute nodes access remote devices as if they were local storage by establishing an iSCSI session with the iSCSI target.

[0033] In this invention, the storage service client is a crucial logical entity. It does not refer to the iSCSI target itself that provides data I / O, but rather to the control plane and management interface provided by the storage system. The storage service client manages several remote iSCSI storage devices and their corresponding iSCSI targets. It provides an application programming interface (API) that allows authorized clients (such as the CSI driver of this invention) to query the status and configuration information of storage resources in the form of a RESTful API (Representational State Transfer Application Programming Interface).

[0034] Finally, the CSI driver is endowed with lifecycle management capabilities that go beyond standard interfaces in this invention. Internally, the CSI driver integrates at least a timed monitoring module and a fault recovery module, and relies on a multipath unit (or multipath module) at the compute node operating system level. The multipath unit provides link redundancy and load balancing capabilities by configuring multiple access paths for the same storage device (i.e., connecting to multiple iSCSI targets), which is the technological cornerstone for achieving seamless switching.

[0035] In one example, a Kubernetes cluster has a CSI driver installed, and the compute nodes in the Kubernetes cluster mount remote iSCSI storage devices via the CSI driver. Furthermore, a single storage service client can manage multiple remote iSCSI storage devices, and each remote iSCSI storage device can be configured with multiple iSCSI targets. When an iSCSI target is migrated, the data interaction is essentially still between the same remote iSCSI storage device; the data is not lost, but the access point changes.

[0036] In this embodiment of the invention, when the access point (i.e. iSCSI target) of the same remote iSCSI storage device changes, the connection can be automatically and seamlessly switched through the CSI driver. This enables the upper-layer application Pod to be unaware of the migration of iSCSI storage targets and to restore the storage connection to high availability. This significantly reduces manual intervention and business interruption caused by storage infrastructure maintenance or failover, and improves the overall reliability and automation level of containerized stateful services.

[0037] During this automatic and seamless switching process, the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device can be detected by the CSI driver. If an abnormal connection status is detected, the current status information of the first iSCSI target can be queried through the CSI driver. This information can be used to determine whether the first iSCSI target has been migrated and whether an abnormal recovery process has been triggered, thereby realizing a proactive detection and accurate diagnostic process.

[0038] In some feasible implementations, the CSI driver can periodically detect the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device at preset time intervals. Based on the periodic detection process, the connectivity at the storage protocol level can be more accurately reflected, avoiding misjudgments caused by network policies or intermediate devices.

[0039] When periodically checking the connection status between the persistent volume and the first iSCSI target on the remote iSCSI storage device, a deep health check based on the iSCSI / SCSI protocol layer can be used. Specifically, the CSI driver can query the session status by executing the iscsiadm command on the compute node, or attempt to send a lightweight SCSI command (such as TEST UNIT READY) to the first iSCSI target. If the command times out or there is no response, the connection status is considered abnormal, thus more accurately reflecting the connectivity at the storage protocol layer and avoiding misjudgments caused by network policies or intermediate devices.

[0040] During this periodic detection process, the preset time interval can be a configurable parameter that balances the system overhead of high-frequency detection with the timeliness of fault detection to determine a suitable time interval. For example, the initial value can be set to 30 seconds, and in actual deployment, it can be dynamically adjusted according to the business's recovery time requirements. For core applications such as finance and trading, it can be shortened to 10-15 seconds; for non-core applications, it can be appropriately extended to 1-2 minutes to reduce system load, etc. This invention does not impose any limitations on this.

[0041] Furthermore, the CSI driver needs to perform connection status checks on each persistent volume it manages individually. This is because in complex storage environments, different persistent volumes may be mounted from different remote iSCSI storage devices or connected to different iSCSI targets, and their states may be independent of each other.

[0042] Furthermore, to ensure the reliability and seamlessness of the recovery process, the CSI driver can perform a crucial precondition check before initiating in-depth status queries. This check verifies whether the compute node's persistent volume is configured with a multipath unit. Only when the persistent volume is configured with a multipath unit can it establish connections with multiple iSCSI targets in the remote iSCSI storage device, thus enabling automatic and seamless failover. Otherwise, if the persistent volume is not configured with a corresponding multipath unit, disconnecting the currently unique iSCSI session will inevitably cause the application Pod to encounter an I / O error earlier than expected. Even if the connection can be quickly rebuilt subsequently, it may cause temporary application unavailability or even a crash.

[0043] Specifically, the CSI driver detects whether the persistent volume of the compute node is configured with a multipath unit. The multipath unit is used to establish connections with multiple iSCSI targets in the remote iSCSI storage device. If the persistent volume is not configured with a multipath unit, the detection logic is terminated. If the persistent volume is configured with a multipath unit, the current status information of the first iSCSI target is queried through the CSI driver. Based on the multipath unit detection, the reliability and seamlessness of the recovery process can be effectively ensured.

[0044] Multipathing units are typically implemented using operating system-level software packages (such as Linux DM-MPIO). The cloud storage side needs to provide at least two mirrored iSCSI targets for the same remote iSCSI storage device. These targets are usually distributed across different physical racks or availability zones to achieve high availability. Furthermore, the multipathing module on the compute nodes is configured to simultaneously identify and connect to these two targets.

[0045] In its implementation, the existence of the multipath unit allows I / O requests to be automatically and instantaneously routed to another pre-configured backup path (potentially leading to a second iSCSI target) when one path (to the first iSCSI target) fails. In other words, the recovery action of this invention essentially repairs the failed path in the multipath module, and the entire process is transparent to upper-layer applications.

[0046] For example, the CSI driver can determine whether multipath is ready by checking whether a multipath device file associated with the persistent volume exists on the compute node (e.g., in the / dev / mapper / directory) or by parsing the output of the multipath-ll command. If it is ready, it can trigger an automatic, seamless switchover; otherwise, it will interrupt the corresponding detection logic.

[0047] When it is determined that the persistent volume is configured with a corresponding multipath unit, the CSI driver can generate a status query request for the first iSCSI target. The status query request includes at least the identifier of the first iSCSI target. Then, the application programming interface (API) communicating with the storage service client is called, and the status query request is sent to the storage service client through the API. After receiving the status query request, the storage service client can parse the corresponding identifier, obtain the current status information corresponding to the identifier, and then return the current status information to the compute node so that the compute node can use the CSI driver to check the connection status between the compute node and the first iSCSI target.

[0048] The current status information can be used to determine the cause of the connection anomaly between the first iSCSI target and the compute node. Optionally, connection anomalies can include network jitter or partitioning, timeouts caused by compute node resource constraints (such as memory and CPU), excessive load on the storage target server, slow response, and the iSCSI target actually being migrated or failing. Based on this, the current status information can help determine the root cause of the connection anomaly from the perspective of the storage system. This allows the CSI driver to distinguish the specific failure of "target migration" from other transient problems, and by consulting "authoritative sources," it can effectively avoid incorrect judgments. For example, if the API call itself fails due to network problems, the CSI driver may tend to consider it a temporary network problem rather than target migration, thus avoiding aggressive recovery actions and preventing unnecessary session reconstruction when the target is normal, saving system resources and time.

[0049] Optionally, the current status information can be a structured data set returned by the storage service client, which may include one of the key information types, such as: Active: The target is functioning normally and connections are acceptable. Receiving this status means that the connection failure was likely caused by a network or client-side issue, and the CSI driver should not perform migration recovery.

[0050] Inactive / Unavailable: The target exists but is unavailable due to a failure (such as server crash or service stoppage). This state may trigger a high availability switchover, but it informs the CSI driver that the target has become invalid.

[0051] Migrated: This state is critical for triggering the recovery process of this invention. It explicitly informs the CSI driver that the target has been systematically migrated to a new service endpoint, the old connection should be abandoned, and a connection to the new target should be established.

[0052] Decommissioned: The target has been permanently deleted. Recovery is not possible in this state and may require triggering a higher-level alert or data migration process.

[0053] Unknown: The storage service cannot recognize the target identifier. This may mean the target has been deleted or the identifier is incorrect, and an alert should be issued and the recovery process terminated.

[0054] Step 102: If the current status information indicates that the first iSCSI target has been migrated, then obtain the connection information corresponding to the migrated second iSCSI target through the CSI driver; After the CSI driver receives the corresponding current status information, it can determine whether the first iSCSI target has been migrated based on the information contained in the current status information. If the current status information indicates that the first iSCSI target has been migrated (e.g., the current status information is Migrated), the CSI driver obtains the connection information corresponding to the migrated second iSCSI target. Thus, once it is confirmed from the storage server that the first iSCSI target has been migrated, the CSI driver can officially execute the corresponding recovery process to achieve automatic and seamless switching.

[0055] In some feasible implementations, once it is determined that the first iSCSI target has been migrated, the CSI driver can receive response data returned by the storage server through the application programming interface, and then extract the connection information of the second iSCSI target from the response data so as to establish a session connection with the new iSCSI target based on the connection information.

[0056] Optionally, the connection information can be a complete set of data required to establish an iSCSI connection, which includes at least: network address: the IP address or domain name of the server where the second iSCSI target is located; port number: the TCP port that the iSCSI service listens on, usually 3260; authentication information: including the username and password used for CHAP authentication, which is necessary to establish a secure iSCSI session; iSCSI Qualified Name (IQN): a unique identifier for the second iSCSI target, etc., which are not limited in this invention.

[0057] In addition, the CSI driver can parse API response data. After extracting the connection information, it can perform preliminary format validation, such as checking the validity of the IP address and ensuring that no key fields are missing, in order to prepare for the next step of actual connection operation.

[0058] Step 103: Control the computing node to disconnect the first connection session with the first iSCSI target, and establish a second connection session between the computing node and the second iSCSI target according to the connection information; Once the connection information is determined, the CSI driver can control the compute node to disconnect the first connection session with the first iSCSI target and establish a second connection session between the compute node and the second iSCSI target according to the connection information. By integrating a full-link automated recovery mechanism of active detection, status verification and connection reconstruction at the CSI driver layer, it achieves business-agnostic recovery of storage connections without affecting the upper-layer application Pods in iSCSI storage target migration scenarios. This significantly reduces manual intervention and business interruption caused by storage infrastructure maintenance or failover, and improves the overall reliability and operational automation level of containerized stateful services.

[0059] In some feasible implementations, the CSI driver can first generate a logout command for the first iSCSI target, and then, in response to the logout command, perform a logout operation on the first iSCSI target. Next, since the connection information includes at least the network address, port number, and authentication information, a connection command for the second iSCSI target can be further generated. Then, in response to the connection command, using the network address, port number, and authentication information, a login operation is performed to establish a second connection session between the compute node and the second iSCSI target.

[0060] In some examples, in actual execution, the logout command is specifically manifested as executing a standard iSCSI management command in the operating system shell of the compute node, such as: isciadm-msession-r<session_id> -u or iscsiadm-mnode-T<Target_IQN> -p<Target_IP:Port> -u. The -u parameter represents logout. The CSI driver performs session cleanup by invoking the operating system to execute such commands. Timely logout of expired sessions helps free up system resources (such as TCP (Transmission Control Protocol) connections and memory) and prevents stale sessions from interfering with the correct decision-making of the multipath module.

[0061] Accordingly, the specific process of establishing a connection can be divided into the following steps: The specific steps to establish a connection: This process is usually divided into two sub-steps: ① Discovery: First, the CSI driver needs to instruct the compute node to initiate a discovery request to the second iSCSI target to confirm its availability and obtain detailed information. Example command: iscsiadm -mdiscovery -tst -p <second target IP:port number>.

[0062] ② Login: After successful discovery, execute the login command to establish a formal iSCSI session. This command requires authentication information, example: iscsiadm -mnode -T <second target IQN> -p <second target IP:port number> --login.

[0063] Through the above process, access to the persistent volume remains uninterrupted throughout the entire disconnection and reconstruction process due to the presence of the multipath module. When the first connection session is disconnected, the multipath module detects a path failure, automatically marks I / O as failed on that path, and continues to serve I / O requests through other healthy paths (if they exist). When the second connection session is successfully established, the multipath module identifies it as a new healthy path and gradually distributes the I / O load to that path. For application Pods that mount this persistent volume, they may only perceive a brief I / O delay in extreme cases, without encountering fatal I / O errors, thus achieving seamless recovery for the business.

[0064] Step 104: Update the connection information corresponding to the persistent volume through the CSI driver.

[0065] Once the second connection session is successfully established, the CSI driver needs to update the connection information associated with the persistent volume within its internal state management mechanism. Specifically, this involves replacing the original connection information (such as target IP, IQN, etc.) pointing to the migrated first iSCSI target with the connection information of the newly acquired and successfully connected second iSCSI target. This process ensures consistency between the internal state of the CSI driver and the underlying actual storage connection state, providing a solid foundation for subsequent storage operations. For example, online expansion of storage volumes needs to be based on the correct, currently valid iSCSI target; creating storage snapshots or cloning requires knowing the exact source of the data; and if future anomalies occur, the CSI driver needs to diagnose and recover based on the latest connection information.

[0066] It should be noted that the embodiments of the present invention include, but are not limited to, the examples described above. It is understood that, based on the ideas and guidance of the embodiments of the present invention, those skilled in the art can also make settings according to actual needs, and the present invention does not limit this.

[0067] In this embodiment of the invention, the technology is applied to a Kubernetes cluster. The Kubernetes cluster has a CSI driver installed, and the compute nodes in the Kubernetes cluster are mounted with remote iSCSI storage devices via the CSI driver. During the migration of the remote storage target, the CSI driver can detect the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device. If an abnormal connection status is detected, the CSI driver queries the current status information of the first iSCSI target. If the current status information indicates that the first iSCSI target has been migrated, the CSI driver obtains the connection information corresponding to the migrated second iSCSI target. Next, the compute node is controlled to disconnect the first connection session with the first iSCSI target, and a second connection session is established between the compute node and the second iSCSI target according to the connection information. Finally, the connection information corresponding to the persistent volume is updated through the CSI driver. Thus, by integrating a full-link automated recovery mechanism of active detection, status verification and connection reconstruction at the CSI driver layer, it is possible to achieve business-agnostic recovery of storage connections without the upper-layer application Pod's awareness in the iSCSI storage target migration scenario. This significantly reduces manual intervention and business interruption caused by storage infrastructure maintenance or failover, and improves the overall reliability and operation and maintenance automation level of containerized stateful services.

[0068] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are provided for illustrative purposes: As an example, refer to Figure 2 The diagram below illustrates a system architecture provided in an embodiment of the present invention: A K8s cluster consists of multiple nodes used to run containerized applications.

[0069] CSI driver: Installed in the Kubernetes cluster, it manages the mounting and unmounting of storage resources. The CSI driver can include at least a scheduled monitoring module 1 and a fault recovery module 2, specifically: Scheduled monitoring module 1: Integrated into the CSI driver, used to periodically check the connection status of the storage target; Fault recovery module 2: Integrated into the CSI driver, working in conjunction with the scheduled monitoring module, disconnects the failed session 8 and establishes a new connection 9 when the storage target 6 is detected to have migrated.

[0070] Multipath module 4: PV storage volume 3 must be connected to multiple targets based on multipath technology. There need to be two mirror targets on the cloud storage side. Therefore, if PV storage volume 3 loses connection with one of the targets 6, the storage connection can be automatically and seamlessly restored.

[0071] For example, a CSI-driven exception recovery process may include: 1. Integrate a timed monitoring module for PV storage volume 3, and set the monitoring interval.

[0072] 2. In the periodic check logic, the first step is to determine whether PV3 is implemented based on multipathing. If not, then there are no conditions for repair, and the check logic ends. 3. Provided that the repair conditions are met, further check the health status of the iSCSI connection associated with PV.

[0073] 4. When the timed monitoring module 1 detects an abnormal connection status of the storage Target, CSI actively queries the actual situation of Target 6 through the storage service API 10. If it is confirmed that Target 6 has not been migrated, no repair action is taken, because the multipath module will automatically repair this abnormal situation.

[0074] 5. If the cloud storage service API 10 informs Target 6 that a migration has indeed occurred, CSI obtains the new Target 7 information and triggers the automatic adjustment module.

[0075] 6. Disconnect the iSCSI connection of the migrated Target6.

[0076] 7. The fault recovery module connects to the new storage Target 7 after migration and establishes an iSCSI connection 9.

[0077] 8. Multipath module 4 ensures seamless recovery for customers during the reconnection process.

[0078] 9. After the adjustment is completed, the timed monitoring module 1 updates the connection information and continues monitoring.

[0079] In addition, a corresponding fault tolerance mechanism can be set up. If multiple attempts fail during the reconnection process, an alarm mechanism will be triggered to notify the administrator.

[0080] Through the above process, by integrating a full-link automated recovery mechanism of active detection, status verification and connection reconstruction into the CSI driver layer, it is possible to achieve business-agnostic recovery of storage connections without affecting the upper-layer application Pods in the iSCSI storage target migration scenario. This significantly reduces manual intervention and business interruption caused by storage infrastructure maintenance or failover, and improves the overall reliability and operation and maintenance automation level of containerized stateful services.

[0081] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0082] Reference Figure 3 This diagram illustrates a structural block diagram of a CSI driver-based anomaly recovery device provided in an embodiment of the present invention, applied to a Kubernetes cluster. The Kubernetes cluster has a CSI driver installed, and the compute nodes in the Kubernetes cluster are connected to remote iSCSI storage devices via the CSI driver. Specifically, it may include the following modules: The status detection module 301 located in the CSI driver is used to detect the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device. If an abnormal connection status is detected, the current status information of the first iSCSI target is queried through the CSI driver. The information acquisition module 302 located in the CSI driver is used to acquire the connection information corresponding to the migrated second iSCSI target if the current status information indicates that the first iSCSI target has been migrated. The connection processing module 303 located in the CSI driver is used to control the computing node to disconnect the first connection session with the first iSCSI target, and to establish a second connection session between the computing node and the second iSCSI target according to the connection information; The information update module 304 located in the CSI driver is used to update the connection information corresponding to the persistent volume.

[0083] In some feasible implementations, the state detection module is specifically used for: The CSI driver detects whether the persistent volume of the compute node is configured with a multipath unit, which is used to provide the compute node with multiple paths for accessing iSCSI targets; If the persistent volume is not configured with the multipath unit, the detection logic is terminated. If the persistent volume is configured with a multipath unit, the current status information of the first iSCSI target is queried through the CSI driver.

[0084] In some feasible implementations, the storage service client manages several of the aforementioned remote iSCSI storage devices, and the status detection module is specifically used for: Generate a status query request for the first iSCSI target, wherein the status query request includes at least the identifier of the first iSCSI target; Call the application programming interface (API) that communicates with the storage service client, and send the status query request to the storage service client through the API; Receive the current status information corresponding to the identifier returned by the storage service client.

[0085] In some feasible implementations, the information acquisition module is specifically used for: Receive response data returned by the storage server through the application programming interface; The connection information of the second iSCSI target is extracted from the response data.

[0086] In some feasible implementations, the information update module is specifically used for: Generate a logout command for the first iSCSI target; In response to the logout command, a logout operation is performed on the first iSCSI target.

[0087] In some feasible implementations, the connection information includes at least a network address, port number, and authentication information, and the information update module is specifically used for: Generate connection instructions for the second iSCSI target; In response to the connection command, the network address, the port number, and the authentication information are used to perform a login operation and establish a second connection session between the computing node and the second iSCSI target.

[0088] In some feasible implementations, the state detection module is specifically used for: The CSI driver periodically detects the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device at preset time intervals.

[0089] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0090] In addition, this invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described CSI-driven exception recovery method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0091] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described CSI-driven exception recovery method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0092] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0093] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program code.

[0094] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0098] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0099] The present invention has provided a detailed description of a CSI-driven exception recovery method and a CSI-driven exception recovery device. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A CSI-driven anomaly recovery method, characterized in that, This method is applied to a Kubernetes (K8s) cluster, wherein the K8s cluster has a CSI driver installed, and the compute nodes in the K8s cluster have remote iSCSI storage devices mounted via the CSI driver; wherein the method includes: The connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device is detected by the CSI driver. If an abnormal connection status is detected, the current status information of the first iSCSI target is queried by the CSI driver. If the current status information indicates that the first iSCSI target has been migrated, then the connection information corresponding to the migrated second iSCSI target is obtained through the CSI driver; The system controls the compute node to disconnect the first connection session with the first iSCSI target, and establishes a second connection session between the compute node and the second iSCSI target according to the connection information. The connection information corresponding to the persistent volume is updated through the CSI driver.

2. The method according to claim 1, characterized in that, The step of querying the current status information of the first iSCSI target through the CSI driver includes: The CSI driver detects whether the persistent volume of the compute node is configured with a multipath unit, which is used to provide the compute node with multiple paths for accessing iSCSI targets. If the persistent volume is not configured with the multipath unit, the detection logic is terminated. If the persistent volume is configured with a multipath unit, the current status information of the first iSCSI target is queried through the CSI driver.

3. The method according to claim 1, characterized in that, The storage service client manages several of the aforementioned remote iSCSI storage devices. Querying the current status information of the first iSCSI target includes: Generate a status query request for the first iSCSI target, wherein the status query request includes at least the identifier of the first iSCSI target; Call the application programming interface (API) that communicates with the storage service client, and send the status query request to the storage service client through the API; Receive the current status information corresponding to the identifier returned by the storage service client.

4. The method according to claim 3, characterized in that, The step of obtaining the connection information corresponding to the migrated second iSCSI target through the CSI driver includes: Receive response data returned by the storage server through the application programming interface; The connection information of the second iSCSI target is extracted from the response data.

5. The method according to claim 1, characterized in that, The step of controlling the computing node to disconnect the first connection session with the first iSCSI target includes: Generate a logout command for the first iSCSI target; In response to the logout command, a logout operation is performed on the first iSCSI target.

6. The method according to claim 1, characterized in that, The connection information includes at least a network address, port number, and authentication information. Establishing a second connection session between the computing node and the second iSCSI target according to the connection information includes: Generate connection instructions for the second iSCSI target; In response to the connection command, the network address, the port number, and the authentication information are used to perform a login operation and establish a second connection session between the computing node and the second iSCSI target.

7. The method according to claim 1, characterized in that, The detection of the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device via the CSI driver includes: The CSI driver periodically detects the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device at preset time intervals.

8. A CSI-driven anomaly recovery device, characterized in that, This device is applied to a Kubernetes (K8s) cluster, wherein the K8s cluster has a CSI driver installed, and the compute nodes in the K8s cluster are mounted with remote iSCSI storage devices via the CSI driver; wherein the device includes: The status detection module located in the CSI driver is used to detect the connection status between the persistent volume on the compute node and the first iSCSI target on the remote iSCSI storage device. If an abnormal connection status is detected, the current status information of the first iSCSI target is queried through the CSI driver. The information acquisition module located in the CSI driver is used to acquire the connection information corresponding to the migrated second iSCSI target if the current status information indicates that the first iSCSI target has been migrated. The connection processing module located in the CSI driver is used to control the compute node to disconnect the first connection session with the first iSCSI target, and to establish a second connection session between the compute node and the second iSCSI target according to the connection information; The information update module located in the CSI driver is used to update the connection information corresponding to the persistent volume.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-7.

Images