Distributed storage system upgrading method and device, equipment and medium

Abstract

Embodiments of the invention provide a distributed storage system upgrading method and apparatus, a device and a medium. The method comprises the steps of determining a proxy node in a node cluster of a distributed storage system; then determining a target upgrade node in the remaining nodes, forwarding a read request received by the target upgrade node to a non-upgrade node for processing, and forwarding a write request received by the target upgrade node to an agent node for processing; performing first upgrading processing on the target upgrading node, and performing first data recovery processing on the target upgrading node according to the proxy node; and finally, in response to the completion operation of the first data recovery processing, recovering the read-write function of the target upgrade node, so that the IO access capability is temporarily provided for the target upgrade node in the upgrade process of the target upgrade node through the proxy node storage service, the risk of IO data loss caused by slow start of the storage service is avoided, and the upgrade efficiency of the target upgrade node is improved. And the service continuity in the system upgrading process is ensured.

Description

Technical Field

[0001] The present invention relates to the field of distributed cluster storage technology, and in particular to a distributed storage system upgrade method and device, equipment, and medium. Background Art

[0002] Distributed cluster storage systems often have a very prominent distributed feature, that is, multiple nodes provide unified services in a cluster working mode; generally speaking, the upgrade process of distributed storage systems includes online upgrades and offline upgrades; the principle and process of offline upgrades are relatively simple, but all services running on the distributed storage system need to be suspended during offline upgrades; therefore, the demand and application of online upgrades in distributed storage systems are relatively extensive.

[0003] In the process of online upgrading of distributed storage systems, in order to ensure data reliability, a multi-copy strategy is generally adopted, and at the same time, a strong consistency model is adopted to ensure the consistency of copies, that is, when all copies have completed IO processing, the IO will return success information; in the online upgrade scenario, in order to make the upgraded system take effect, some services need to be restarted, including services for data storage.

[0004] However, restarting the storage service on the upgraded node will cause the IO requests caused by the strong consistency model to wait for the IO return of the upgraded node during the restart period and the time period when the service can be provided again, which may easily lead to the phenomenon of no IO return or return timeout for the upper-layer business. At this time, the duration of the impact on the business depends on the time from the restart of the storage service to the restoration of normal operation. That is, the entire process will cause business interruption and the time is uncontrollable, which will have a serious impact on the continuity of enterprise business. Summary of the invention

[0005] In view of the above problems, a distributed storage system upgrade method, device, equipment, and medium are proposed to overcome the above problems or at least partially solve the above problems, including: A distributed storage system upgrade method, the method comprising: In response to an upgrade requirement of the distributed storage system, determining a proxy node in a node cluster of the distributed storage system; Determine a target upgrade node among the remaining nodes of the node cluster, and determine nodes among the remaining nodes other than the target upgrade node as non-upgrade nodes; forwarding the read request received by the target upgraded node to the non-upgraded node for processing, and forwarding the write request received by the target upgraded node to the proxy node for processing; Performing a first upgrade process on the target upgrade node, and in response to the first upgrade process of the target upgrade node completing an operation, performing a first data recovery process on the target upgrade node according to the proxy node; In response to the completion operation of the first data recovery process, the read and write functions of the target upgraded node are restored.

[0006] Optionally, the method further includes: The step of determining the target upgrade node among the remaining nodes of the node cluster is performed again until the nodes among the remaining nodes complete the first upgrade process and the first data recovery process; A second upgrade process is performed on the proxy node to complete the upgrade requirement of the distributed storage system.

[0007] Optionally, determining a proxy node in a node cluster of the distributed storage system includes: The memory and disk capacity of each node in the node cluster is determined, and the proxy node is determined in the node cluster of the distributed storage system according to the memory and disk capacity of each node.

[0008] Optionally, before forwarding the read request received by the target upgraded node to the non-upgraded node for processing, the method further includes: It is determined whether the target upgrade node is a management node, and if the target upgrade node is a management node, the management function of the target upgrade node is switched to the non-upgrade node.

[0009] Optionally, also include: The data transmission session connected to the target upgraded node is switched to the non-upgraded node for processing.

[0010] Optionally, before determining the target upgrade node among the remaining nodes of the node cluster, the method further includes: In response to the determination operation of the proxy node, the original data in the proxy node is synchronized through the remaining nodes.

[0011] Optionally, the performing a second upgrade process on the proxy node includes: Control the proxy node to replace the storage service upgrade package corresponding to the upgrade requirement, and restart the storage service on the proxy node; The original data in the proxy node is restored through the remaining nodes.

[0012] Optionally, performing a first upgrade process on the target upgrade node includes: The target upgrade node is controlled to replace the storage service upgrade package corresponding to the upgrade requirement, and storage service restart processing is performed on the target upgrade node.

[0013] Optionally, performing a first data recovery process on the target upgrade node according to the proxy node includes: A data connection is established between the target upgrade node and the proxy node, and the input and output request data belonging to the target upgrade node stored in the proxy node is played back to the target upgrade node according to the data connection.

[0014] Optionally, also include: In response to the playback completion operation of the input / output request data, informing the proxy node to stop receiving the input / output request data belonging to the target upgrade node; Determine a first time point corresponding to the playback completion operation and a second time point corresponding to the proxy node stopping receiving input and output request data belonging to the target upgrade node; Determine a first block of data object identifier list corresponding to the input and output request data received by the proxy node between the first time point and the second time point; According to the first block of data object identification list, a second data recovery process is performed on the target upgrade node.

[0015] Optionally, the replaying the input / output request data belonging to the target upgrade node and stored in the proxy node to the target upgrade node according to the data connection includes: Recording the block data object identifiers corresponding to the input / output request data that failed to be replayed, and generating a second block data object identifier list according to the block data object identifiers corresponding to the input / output request data that failed to be replayed; The performing a second data recovery process on the target upgrade node according to the first block data object identification list includes: According to the first block data object identification list and the second block data object identification list, a second data recovery process is performed on the target upgrade node.

[0016] Optionally, determining a proxy node in a node cluster of the distributed storage system according to the memory and disk capacity of each node includes: The node with the largest memory and disk capacity in the node cluster is determined as the proxy node.

[0017] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned distributed storage system upgrade method when executed by the processor.

[0018] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distributed storage system upgrade method as described above is implemented.

[0019] A computer program product includes a computer program, wherein when the computer program is executed by a processor, the distributed storage system upgrade method described above is implemented.

[0020] The embodiments of the present invention have the following advantages: In an embodiment of the present invention, in response to an upgrade requirement of a distributed storage system, a proxy node is determined in a node cluster of the distributed storage system; then a target upgrade node is determined in the remaining nodes of the node cluster, and nodes other than the target upgrade node in the remaining nodes are determined as non-upgrade nodes; thereby, a read request received by the target upgrade node is forwarded to the non-upgrade node for processing, and a write request received by the target upgrade node is forwarded to the proxy node for processing; then a first upgrade process is performed on the target upgrade node, and in response to a completion operation of the first upgrade process of the target upgrade node, a first data recovery process is performed on the target upgrade node according to the proxy node; finally, in response to a completion operation of the first data recovery process, the read and write functions of the target upgrade node are restored, thereby realizing that the proxy node storage service temporarily provides IO access capability to the target upgrade node during the upgrade process of the target upgrade node, thereby reducing the problem of distributed storage performance degradation caused by the restart of the storage service, and the upgrade process will not affect the performance of the distributed storage system, thereby avoiding the risk of IO data loss due to slow startup of the storage service, ensuring business continuity during the system upgrade process, and improving user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0022] Figure 1 is a flowchart of steps of a distributed storage system upgrade method provided by some embodiments of the present invention; Figure 2 is a schematic diagram of a proxy node selection process provided by some embodiments of the present invention; Figure 3 is a node upgrade state flow chart provided by some embodiments of the present invention; Figure 4 It is a schematic diagram of a non-upgraded node IO request processing flow provided by some embodiments of the present invention; Figure 5 It is a schematic diagram of a target upgrade node data playback process provided by some embodiments of the present invention; Figure 6 It is a structural diagram of a distributed storage system upgrading device provided by some embodiments of the present invention. DETAILED DESCRIPTION

[0023] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] In the process of online upgrading of distributed storage systems, in order to ensure data reliability, a multi-copy strategy is generally adopted, and at the same time, a strong consistency model is adopted to ensure the consistency of the copies, that is, when all copies have completed IO processing, the IO will return success information; and in the online upgrade scenario, in order to make the upgraded system effective, some services need to be restarted, including services for data storage. However, the restart of the storage service on the upgraded node will cause the IO requests caused by the strong consistency model to wait for the IO return of the upgraded node during the restart period and the time period when the service can be provided again, which may easily lead to the phenomenon of no IO return or return timeout for the upper-level business. At this time, the duration of the impact on the business depends on the time from the restart of the storage service to the restoration of normal, that is, the entire process will cause business interruption and the time is uncontrollable, which will have a serious impact on the continuity of enterprise business.

[0025] In the embodiment of the present invention, based on the core technical concept of temporarily providing IO access capability to the target upgrade node during the upgrade process of the target upgrade node through the proxy node storage service, the distributed storage system upgrade method in the related art is improved. The present invention will be described in detail below with reference to the accompanying drawings: Reference Figure 1 , shows a flow chart of steps of a distributed storage system upgrade method provided by some embodiments of the present invention, which may specifically include the following steps: Step 101, in response to an upgrade requirement of the distributed storage system, determining a proxy node in a node cluster of the distributed storage system; In the specific implementation, a proxy node can be selected from the cluster nodes during the distributed cluster upgrade preparation phase. The proxy node can provide the following services for the entire cluster nodes during the rolling upgrade: (1) During the upgrade process of the target upgrade node, the IO write request to the target upgrade node will be forwarded to the proxy node, received by the storage service of the proxy node and persisted to the local disk; (2) After the target upgrade node storage service is restarted, the IO request data belonging to the target upgrade node is replayed from the proxy node before joining the cluster; (3) After all other non-proxy nodes have been upgraded, the proxy node replaces the storage service upgrade package and joins the cluster as a new node, synchronizing the original data belonging to the proxy node.

[0026] In some embodiments of the present invention, determining a proxy node in a node cluster of the distributed storage system includes: The memory and disk capacity of each node in the node cluster is determined, and the proxy node is determined in the node cluster of the distributed storage system according to the memory and disk capacity of each node.

[0027] In practical applications, such as Figure 2 As shown in the figure, after the distributed storage cluster enters the online upgrade mode, it can determine the memory and disk capacity of each node in the node cluster and select the node with the largest remaining memory and disk capacity in the cluster (i.e. Figure 2 Node C in the figure is used as the upgrade proxy node, which is responsible for taking over the write IO requests of the target upgrade node during the online upgrade. Since the selected proxy node has sufficient memory and disk storage resources, the risk of IO data loss caused by slow startup of the storage service can be avoided.

[0028] In some embodiments of the present invention, determining a proxy node in a node cluster of the distributed storage system according to the memory and disk capacity of each node includes: The node with the largest memory and disk capacity in the node cluster is determined as the proxy node.

[0029] In actual applications, the memory and disk capacity of each node in the node cluster can be determined and the node with the largest remaining memory and disk capacity in the cluster can be selected as the upgrade proxy node, which is responsible for taking over the write IO requests of the target upgrade node during the online upgrade. Since the selected proxy node has sufficient memory and disk storage resources, the risk of IO data loss due to slow startup of the storage service can be avoided.

[0030] Step 102, determining a target upgrade node among the remaining nodes of the node cluster, and determining nodes other than the target upgrade node among the remaining nodes as non-upgrade nodes; In the specific implementation, after the proxy node is determined, the platform can select a node from the remaining nodes as the target upgrade node and notify the target upgrade node to upgrade. At this time, the nodes other than the target upgrade node in the remaining nodes can be determined as non-upgrade nodes, so as to assist the proxy node in processing the IO requests of the target upgrade node itself during the subsequent upgrade process.

[0031] In some embodiments of the present invention, before determining the target upgrade node among the remaining nodes of the node cluster, the method further includes: In response to the determination operation of the proxy node, the original data in the proxy node is synchronized through the remaining nodes.

[0032] In the specific implementation, after the proxy node is selected, the non-proxy nodes can be upgraded in sequence. Since the target upgrade node will undergo different state changes during the entire upgrade process, different upgrade states can be defined separately, and the changes in the upgrade state can be synchronized to all nodes in the cluster through events, so that the upgrade node storage service can control the upgrade process of the target upgrade node through the flow of the upgrade state machine.

[0033] Specifically, Figure 3 As shown, the target upgrade node in the present invention may include the following states during the entire upgrade process: UPGRADE_INIT (upgrade initial state), UPGRADE_PREPARE (upgrade preparation state), UPGRADE_REPLAY_START (data playback state), UPGRADE_REPLAY_FINISH (data playback completion state), UPGRADE_SUCCESS (upgrade success state), UPGRADE_FAILED (upgrade failure state). For example, in this embodiment, when the target upgrade node is in the upgrade initial state, the platform will check whether the cluster node state is normal, and replace the storage service upgrade package, set the cluster state to the online upgrade state, thereby selecting a node with the largest remaining memory and disk capacity in the cluster as the proxy node, triggering the corresponding task to notify the cluster of the relevant information of the proxy node, so that other nodes in the cluster can synchronize data according to the new hash algorithm after receiving the proxy node information, and synchronize the original data of the proxy node to the non-proxy nodes in the cluster, so that in the subsequent process, after completing the upgrade of other nodes, the upgrade process of the proxy node itself can be directly carried out.

[0034] Step 103: forwarding the read request received by the target upgraded node to the non-upgraded node for processing, and forwarding the write request received by the target upgraded node to the proxy node for processing; In the specific implementation, after the proxy node and the target upgrade node are determined, the state of the target upgrade node can be adjusted to the upgrade preparation state. This state will switch the iscsi session (iscsi session, i.e., data transmission session) connected to the target upgrade node to other non-upgrade nodes, and the new session will no longer be connected to the target upgrade node to ensure that the upper-layer business is not affected during the restart of the target upgrade node storage service. At the same time, if the target upgrade node also assumes other management roles in the storage cluster, such as the target upgrade node is the management node connected to the storage service or the main management node of the cluster, these management functions need to be switched to other non-upgrade nodes. After the target upgrade node passes through this state, all preparations can be considered completed.

[0035] Furthermore, the state of the target upgraded node can be adjusted to the data playback state. After other non-upgraded nodes receive the event notification that the target upgraded node has entered the data playback state, such as Figure 4 As shown, the write IO request belonging to the target upgraded node will be forwarded to the proxy node (that is, the upgrade status of the replica node is judged to be yes), and the proxy node storage service will complete the persistent storage of the write IO request data of the target upgraded node. The read IO request will be forwarded to other non-upgraded nodes for completion. It should be noted that when the non-upgraded node processes the read request, it will directly process the write request when the corresponding replica exists locally on the node. If the corresponding replica does not exist locally on the node, it will determine whether other nodes with the replica are in the upgraded state. If they are not in the upgraded state, a remote read request will be initiated. If they are in the upgraded state, other nodes with the replica will be found to ensure the continuity and stability of the overall upgrade process.

[0036] Specifically, when the corresponding process in this embodiment is actually executed, the target upgrade node may notify the cluster target upgrade node that the upgrade state is UPGRADE_PREPARE (upgrade preparation state) after checking the state of the proxy node. If the target upgrade node is the iscsi session connection management node or the cluster's primary management node, the management function is switched to another node; Switch the iscsi session connection on the target upgrade node. When the session management node receives the new connection and re-assigns the connection node, it can filter out the target upgrade node according to the upgrade status. Update the upgrade status of the target upgrade node to UPGRADE_REPLAY_START (data replay status) and broadcast it to all nodes in the cluster through events: After the non-upgraded session node receives the IO request, if it is a read IO request belonging to the target upgraded node, it is forwarded to the non-upgraded node for completion; if it is a write IO request belonging to the target upgraded node, it is forwarded to the proxy node for completion. In this state, the proxy node starts to receive the write IO data of the target upgraded node.

[0037] In some embodiments of the present invention, before forwarding the read request received by the target upgraded node to the non-upgraded node for processing, the process further includes: It is determined whether the target upgrade node is a management node, and if the target upgrade node is a management node, the management function of the target upgrade node is switched to the non-upgrade node.

[0038] In the specific implementation, after the proxy node and the target upgrade node are determined, the state of the target upgrade node can be adjusted to the upgrade preparation state. This state will switch the iscsi session (iscsi session, i.e., data transmission session) connected to the target upgrade node to other non-upgrade nodes, and the new session will no longer be connected to the target upgrade node to ensure that the upper-layer business is not affected during the restart of the target upgrade node storage service. At the same time, if the target upgrade node also assumes other management roles in the storage cluster, such as the target upgrade node is the management node connected to the storage service or the main management node of the cluster, these management functions need to be switched to other non-upgrade nodes. After the target upgrade node passes through this state, all preparations can be considered completed.

[0039] In some embodiments of the present invention, it also includes: The data transmission session connected to the target upgraded node is switched to the non-upgraded node for processing.

[0040] In actual applications, after the proxy node and the target upgrade node are determined, the state of the target upgrade node can be adjusted to the upgrade preparation state. This state will switch the iscsi session (iscsi session, i.e., data transmission session) connected to the target upgrade node to other non-upgraded nodes, and the new session will no longer be connected to the target upgrade node to ensure that the upper-layer business is not affected during the restart of the target upgrade node storage service.

[0041] Step 104, performing a first upgrade process on the target upgrade node, and in response to the first upgrade process of the target upgrade node completing an operation, performing a first data recovery process on the target upgrade node according to the proxy node; In the specific implementation, after the above steps are completed, the first upgrade process can be performed on the target upgrade node, and the target upgrade node storage service will automatically exit and be restarted by the monitoring service. After the storage service is restarted, it will recognize that the cluster is in online upgrade mode, and the target upgrade node status will be updated to data playback status. At this time, the first data recovery process can be performed on the target upgrade node according to the proxy node, that is, at this time, the target upgrade node will establish a connection with the proxy node, and replay the IO request data belonging to the target upgrade node from the proxy node.

[0042] In some embodiments of the present invention, performing a first upgrade process on the target upgrade node includes: The target upgrade node is controlled to replace the storage service upgrade package corresponding to the upgrade requirement, and storage service restart processing is performed on the target upgrade node.

[0043] In actual applications, the target upgrade node can be controlled to replace the storage service upgrade package corresponding to the upgrade requirement. The target upgrade node storage service will automatically exit and be restarted by the monitoring service. After the storage service is restarted, it will recognize that the cluster is in online upgrade mode, and the target upgrade node status will be updated to data playback status. At this time, the first data recovery process can be performed on the target upgrade node according to the proxy node, that is, the target upgrade node will establish a connection with the proxy node, and replay the IO request data belonging to the target upgrade node from the proxy node.

[0044] In some embodiments of the present invention, performing a first data recovery process on the target upgrade node according to the proxy node includes: A data connection is established between the target upgrade node and the proxy node, and the input and output request data belonging to the target upgrade node stored in the proxy node is played back to the target upgrade node according to the data connection.

[0045] In the specific implementation, Figure 5As shown in the figure, after establishing the data connection between the target upgrade node and the agent node, the replay request can be sent, the replay data head in the agent node can be received and parsed, and the IO request data can be obtained in sequence according to the head. The request is submitted to the io poller (io process thread, the thread of the storage service processing io) in sequence, so that the io poller processes the submitted io write request to ensure that it is completed in sequence. When the replay request fails to write, the oid corresponding to the IO request data can be saved to the recovery list (the second block of data object identification list) for data recovery in the subsequent process; after the data replay is completed, the target upgrade node can be controlled to join the cluster as the NODE_UPGRADE_JOIN (updated node) type, and after waiting for all the requests for the aforementioned data replay to be completed, the agent-node is notified to stop receiving IO, and the oid list (the first block of data object identification list) corresponding to the IO request received from the completion of the data replay (the first time point) to the notification of stopping IO (the second time point) is obtained from the agent-node; thus, the last obtained oid list and the oid list of the failed replay are submitted to the recovery task together to perform the second data recovery process on the target upgrade node to ensure the integrity of the node data; After all the above steps are completed, it can be considered that the first data recovery process on the target upgrade node is completed, and it is determined that the target upgrade node has been completely upgraded.

[0046] In actual applications, after the data playback is completed, a request can be sent to the cluster's primary management node to add the target upgrade node to the cluster: After receiving the join request, the master management node can check the version number and submit the join cluster task event notification of the target upgrade node if the requirements are met; The target upgrade node can process the joining task of the target upgrade node, wait for all the playback IO request tasks to be completed in sequence, and record the data oid identifier corresponding to the request that failed to be played back; Send a stop receiving IO message to the proxy node, and obtain the oid list corresponding to the IO requests received from the proxy node between the completion of data playback and the notification of stopping IO; The oid list of failed replays and the oid list last obtained from the proxy node are added as upgrade nodes to the task list of cluster data recovery. At this time, the upgrade node status is updated to the data playback completion status. When the recovery task (second upgrade process) is completed, it can be considered that the target upgrade node has been fully upgraded.

[0047] In some embodiments of the present invention, it also includes: In response to the playback completion operation of the input / output request data, informing the proxy node to stop receiving the input / output request data belonging to the target upgrade node; Determine a first time point corresponding to the playback completion operation and a second time point corresponding to the proxy node stopping receiving input and output request data belonging to the target upgrade node; Determine a first block of data object identifier list corresponding to the input and output request data received by the proxy node between the first time point and the second time point; According to the first block of data object identification list, a second data recovery process is performed on the target upgrade node.

[0048] In actual applications, after establishing a data connection between the target upgrade node and the agent node, a playback request can be sent to receive and parse the playback data head in the agent node, so as to obtain IO request data in sequence according to the head, and the requests are submitted to the io poller in sequence so that the io poller processes the submitted io write requests to ensure that they are completed in sequence. After the data playback is completed, the target upgrade node can be controlled to join the cluster as NODE_UPGRADE_JOIN (updated node) type, and after waiting for all the requests for the aforementioned data playback to be completed, the agent-node is notified to stop receiving IO, and the oid list (first block of data object identifier list) corresponding to the IO requests received from the completion of data playback (first time point) to the notification of stopping IO (second time point) is obtained from the agent-node; thereby, the last obtained oid list is submitted to the recovery task to perform the second data recovery processing on the target upgrade node.

[0049] In some embodiments of the present invention, the step of replaying the input / output request data belonging to the target upgraded node and stored in the proxy node to the target upgraded node according to the data connection includes: Recording the block data object identifiers corresponding to the input / output request data that failed to be replayed, and generating a second block data object identifier list according to the block data object identifiers corresponding to the input / output request data that failed to be replayed; The performing a second data recovery process on the target upgrade node according to the first block data object identification list includes: According to the first block data object identification list and the second block data object identification list, a second data recovery process is performed on the target upgrade node.

[0050] In the specific implementation, after establishing the data connection between the target upgrade node and the agent node, a replay request can be sent, and the replay data head in the agent node can be received and parsed, so as to obtain the IO request data in sequence according to the head, and the request is submitted to the io poller in sequence so that the io poller processes the submitted io write request to ensure that it is completed in sequence. When the replay request fails to write, the oid corresponding to the IO request data can be saved to the recovery list (the second block of data object identifier list) for data recovery in the subsequent process; after the data replay is completed, the target upgrade node can be controlled to join the cluster as the NODE_UPGRADE_JOIN (updated node) type, and after waiting for all the requests for the aforementioned data replay to be completed, the agent-node is notified to stop receiving IO, and the oid list (the first block of data object identifier list) corresponding to the IO request received from the completion of the data replay (the first time point) to the notification of stopping IO (the second time point) is obtained from the agent-node; thus, the last obtained oid list and the oid list of failed replay are submitted to the recovery task together to perform the second data recovery process on the target upgrade node to ensure the integrity of the node data; After all the above steps are completed, it can be considered that the first data recovery process on the target upgrade node is completed, and it is determined that the target upgrade node has been completely upgraded.

[0051] In actual applications, after the data playback is completed, a request can be sent to the cluster's primary management node to add the target upgrade node to the cluster: After receiving the join request, the master management node can check the version number and submit the join cluster task event notification of the target upgrade node if the requirements are met; The target upgrade node can process the joining task of the target upgrade node, wait for all the playback IO request tasks to be completed in sequence, and record the data oid identifier corresponding to the request that failed to be played back; Send a stop receiving IO message to the proxy node, and obtain the oid list corresponding to the IO requests received from the proxy node between the completion of data playback and the notification of stopping IO; The oid list of failed replays and the oid list last obtained from the proxy node are added as upgrade nodes to the task list of cluster data recovery. At this time, the upgrade node status is updated to the data playback completion status. When the recovery task (second upgrade process) is completed, it can be considered that the target upgrade node has been fully upgraded.

[0052] Step 105: In response to the completion operation of the first data recovery process, the read and write functions of the target upgraded node are restored.

[0053] In actual applications, after the above steps are completed, the upgrade status of the target upgrade node can be adjusted to the data playback completion status. In this status, most of the IO request data received by the proxy node has been played back in sequence, and a very small part of the data needs to be restored through the recovery process. After entering this status, the non-upgraded node will switch the IO requests belonging to the target upgrade node from the proxy node to the target upgrade node; that is, for the non-upgraded node, when the upgrade node status is changed to the data playback completion status, the IO read request belonging to the upgrade node will be directly forwarded to the upgrade node, and the write request will also be switched from forwarding to the proxy node to the upgrade node, and the read and write functions of the target upgrade node have been restored.

[0054] Furthermore, the upgrade status of the target upgrade node can be adjusted to a successful upgrade status, that is, the last small amount of data recovery is completed, and all data of the target upgrade node is normal. After the proxy node receives the event notification of the successful upgrade of the target upgrade node, it can clear local related resources and start preparing for the upgrade of the next node.

[0055] As for the upgrade failure state, when an exception occurs in the target upgrade node during the upgrade process, such as a target upgrade node exception or other non-upgrade node exception events, the target upgrade node state can be adjusted to an upgrade failure state, waiting for manual intervention to repair the environment before upgrading again.

[0056] In some embodiments of the present invention, the method further comprises: The step of determining the target upgrade node among the remaining nodes of the node cluster is performed again until the nodes among the remaining nodes complete the first upgrade process and the first data recovery process; A second upgrade process is performed on the proxy node to complete the upgrade requirement of the distributed storage system.

[0057] In the specific implementation, after completing the upgrade of the target upgrade node, the nodes in the node cluster except the proxy node can be upgraded in turn according to the same process until all nodes except the proxy node have completed the upgrade, and then the second upgrade process can be performed on the proxy node to meet the actual upgrade requirements of the distributed storage system.

[0058] Specifically, the storage service upgrade package corresponding to the replacement upgrade requirement of the same process control proxy node can be installed, and the storage service can be restarted for the proxy node; subsequently, since the original data of the proxy node has been synchronized to the non-proxy node in the cluster when the proxy node was determined in the aforementioned steps, the original data in the proxy node can be restored through the non-proxy node at this time; that is, after the proxy node replaces the storage service upgrade package, it can join the cluster as a new node and synchronize the original data belonging to the proxy node from other nodes in the cluster; after the proxy node upgrade is completed, the cluster status can be updated to normal status, and each node can be controlled to clear the related resources created during the upgrade.

[0059] Through the distributed storage system upgrade method provided by the present invention, a node can be selected from the storage cluster as an upgrade proxy node, and the proxy node takes over the IO write request that should be forwarded to the node during the restart service of the upgrade node. At the same time, by defining various upgrade states of the upgrade node, and synchronizing the changes in the upgrade state to all nodes in the cluster through events, the resource switching, IO forwarding switching and other processes during the node upgrade process are accurately controlled, which effectively solves the many problems faced by online upgrades in distributed cluster storage, such as IO interruptions during the upgrade period affecting the business, or restarting other services to receive IO at the upgrade node, which occupies a large amount of resources and causes the upgrade to fail. The proxy node storage service temporarily provides IO access capabilities for the upgrade node, thereby reducing the problem of distributed storage performance degradation caused by the restart of the storage service, and the performance will not be affected during the upgrade process. Since the selected proxy node has sufficient memory and disk storage resources, the risk of IO data loss due to slow startup of the storage service can be avoided.

[0060] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0061] Reference Figure 6 , shows a schematic diagram of the structure of a distributed storage system upgrade device provided by some embodiments of the present invention, which may specifically include the following modules: The proxy node determination module 601 is used to determine a proxy node in a node cluster of the distributed storage system in response to an upgrade requirement of the distributed storage system; A target upgrade node determination module 602 is used to determine a target upgrade node among the remaining nodes of the node cluster, and determine nodes other than the target upgrade node among the remaining nodes as non-upgrade nodes; The request forwarding module 603 is used to forward the read request received by the target upgraded node to the non-upgraded node for processing, and forward the write request received by the target upgraded node to the proxy node for processing; A first upgrade processing module 604 is used to perform a first upgrade process on the target upgrade node, and in response to the first upgrade process of the target upgrade node completing the operation, perform a first data recovery process on the target upgrade node according to the proxy node; The read / write function recovery module 605 is used to recover the read / write function of the target upgraded node in response to the completion operation of the first data recovery process.

[0062] In some embodiments of the present invention, the device further comprises: A re-execution module, used for re-executing the step of determining the target upgrade node among the remaining nodes of the node cluster until the nodes among the remaining nodes have completed the first upgrade process and the first data recovery process; The second upgrade processing module is used to perform a second upgrade process on the proxy node to complete the upgrade requirements of the distributed storage system.

[0063] In some embodiments of the present invention, the proxy node determination module 601 includes: The proxy node determination submodule is used to determine the memory and disk capacity of each node in the node cluster, and determine the proxy node in the node cluster of the distributed storage system according to the memory and disk capacity of each node.

[0064] In some embodiments of the present invention, the device further comprises: The management function switching module is used to determine whether the target upgrade node is a management node, and if the target upgrade node is a management node, switch the management function of the target upgrade node to the non-upgrade node.

[0065] In some embodiments of the present invention, the device further comprises: The data transmission session switching module is used to switch the data transmission session connected to the target upgraded node to the non-upgraded node for processing.

[0066] In some embodiments of the present invention, the device further comprises: The proxy node data synchronization module is used to synchronize the original data in the proxy node through the remaining nodes in response to the determination operation of the proxy node.

[0067] In some embodiments of the present invention, the second upgrade processing module includes: A first upgrade package replacement submodule is used to control the proxy node to replace the storage service upgrade package corresponding to the upgrade requirement, and to restart the storage service for the proxy node; The original data recovery submodule is used to recover the original data in the proxy node through the remaining nodes.

[0068] In some embodiments of the present invention, the first upgrade processing module 604 includes: The second upgrade package replacement submodule is used to control the target upgrade node to replace the storage service upgrade package corresponding to the upgrade requirement, and to restart the storage service for the target upgrade node.

[0069] In some embodiments of the present invention, the first upgrade processing module 604 includes: The data connection establishment submodule is used to establish a data connection between the target upgrade node and the proxy node, and replay the input and output request data belonging to the target upgrade node stored in the proxy node to the target upgrade node according to the data connection.

[0070] In some embodiments of the present invention, the first upgrade processing module 604 further includes: a stop receiving submodule, configured to notify the proxy node to stop receiving the input and output request data belonging to the target upgrade node in response to the playback completion operation of the input and output request data; A time point determination submodule, used to determine a first time point corresponding to the playback completion operation and a second time point corresponding to the proxy node stopping receiving input and output request data belonging to the target upgrade node; A first block data object identifier list determination submodule, used to determine a first block data object identifier list corresponding to the input and output request data received by the proxy node between the first time point and the second time point; The second data recovery processing submodule is used to perform a second data recovery process on the target upgrade node according to the first block data object identification list.

[0071] In some embodiments of the present invention, the data connection establishment submodule includes: a block data object identifier recording unit, configured to record the block data object identifier corresponding to the input / output request data that fails to be replayed, and generate a second block data object identifier list according to the block data object identifier corresponding to the input / output request data that fails to be replayed; The second data recovery processing submodule includes: The second data recovery processing unit is used to perform a second data recovery process on the target upgrade node according to the first block data object identification list and the second block data object identification list.

[0072] In some embodiments of the present invention, the proxy node determination submodule includes: The proxy node determination unit is used to determine the node with the largest memory and disk capacity in the node cluster as the proxy node.

[0073] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0074] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above distributed storage system upgrade method when executed by a processor.

[0075] Some embodiments of the present invention further provide an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned distributed storage system upgrade method when executed by the processor.

[0076] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the distributed storage system upgrade method as described above is implemented.

[0077] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0078] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0079] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0082] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0083] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0084] The above is a detailed introduction to the provided distributed storage system upgrade method and device, equipment, and medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A distributed storage system upgrade method, characterized in that: The method comprises: In response to an upgrade requirement of the distributed storage system, determining a proxy node in a node cluster of the distributed storage system; Determine a target upgrade node among the remaining nodes of the node cluster, and determine nodes other than the target upgrade node among the remaining nodes as non-upgrade nodes; forwarding the read request received by the target upgraded node to the non-upgraded node for processing, and forwarding the write request received by the target upgraded node to the proxy node for processing; Performing a first upgrade process on the target upgrade node, and in response to the first upgrade process of the target upgrade node completing an operation, performing a first data recovery process on the target upgrade node according to the proxy node; In response to the completion operation of the first data recovery process, the read and write functions of the target upgraded node are restored.

2. The method according to claim 1, characterized in that The method further comprises: The step of determining the target upgrade node among the remaining nodes of the node cluster is performed again until the nodes among the remaining nodes complete the first upgrade process and the first data recovery process; A second upgrade process is performed on the proxy node to complete the upgrade requirement of the distributed storage system.

3. The method according to claim 1, characterized in that The determining of a proxy node in a node cluster of the distributed storage system includes: The memory and disk capacity of each node in the node cluster is determined, and the proxy node is determined in the node cluster of the distributed storage system according to the memory and disk capacity of each node.

4. The method according to claim 1, characterized in that: Before forwarding the read request received by the target upgraded node to the non-upgraded node for processing, the method further includes: It is determined whether the target upgrade node is a management node, and if the target upgrade node is a management node, the management function of the target upgrade node is switched to the non-upgrade node.

5. The method according to claim 4, characterized in that Also includes: The data transmission session connected to the target upgraded node is switched to the non-upgraded node for processing.

6. The method according to claim 2, characterized in that Before determining the target upgrade node among the remaining nodes in the node cluster, the method further includes: In response to the determination operation of the proxy node, the original data in the proxy node is synchronized through the remaining nodes.

7. The method according to claim 6, characterized in that The performing a second upgrade process on the proxy node includes: Control the proxy node to replace the storage service upgrade package corresponding to the upgrade requirement, and restart the storage service on the proxy node; The original data in the proxy node is restored through the remaining nodes.

8. The method according to claim 1, characterized in that The performing a first upgrade process on the target upgrade node includes: The target upgrade node is controlled to replace the storage service upgrade package corresponding to the upgrade requirement, and storage service restart processing is performed on the target upgrade node.

9. The method according to claim 1, characterized in that: The performing a first data recovery process on the target upgrade node according to the proxy node includes: A data connection is established between the target upgrade node and the proxy node, and the input and output request data belonging to the target upgrade node stored in the proxy node is played back to the target upgrade node according to the data connection.

10. The method according to claim 9, characterized in that Also includes: In response to the playback completion operation of the input / output request data, informing the proxy node to stop receiving the input / output request data belonging to the target upgrade node; Determine a first time point corresponding to the playback completion operation and a second time point corresponding to the proxy node stopping receiving input and output request data belonging to the target upgrade node; Determine a first block of data object identifier list corresponding to the input and output request data received by the proxy node between the first time point and the second time point; According to the first block of data object identification list, a second data recovery process is performed on the target upgrade node.

11. The method according to claim 10, characterized in that The step of replaying the input / output request data belonging to the target upgrade node stored in the proxy node to the target upgrade node according to the data connection includes: Recording the block data object identifiers corresponding to the input / output request data that failed to be replayed, and generating a second block data object identifier list according to the block data object identifiers corresponding to the input / output request data that failed to be replayed; The performing a second data recovery process on the target upgrade node according to the first block data object identification list includes: According to the first block data object identification list and the second block data object identification list, a second data recovery process is performed on the target upgrade node.

12. The method according to claim 3, characterized in that Determining a proxy node in a node cluster of the distributed storage system according to the memory and disk capacity of each node includes: The node with the largest memory and disk capacity in the node cluster is determined as the proxy node.

13. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the distributed storage system upgrading method according to any one of claims 1 to 12 is implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distributed storage system upgrading method according to any one of claims 1 to 12 is implemented.

15. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the distributed storage system upgrading method according to any one of claims 1 to 12.

Images