Link adaptation fault tolerance method, device and server for multi-controller cluster storage

Abstract

This invention relates to the field of storage multi-controller clusters, and discloses a link adaptive fault-tolerant method, apparatus, and server for storage multi-controller clusters. The method includes: obtaining a first priority path set and selecting a path from the first priority path set for communication between storage multi-controller clusters; when the communication performance of any path in the first priority path set does not match a preset communication performance, migrating the path that does not match the preset communication performance from the first priority path set to a second priority path set; when the path in the first priority path set is empty, selecting a path from the second priority path set for communication between storage multi-controller clusters. This invention removes paths that may cause large communication delays or timeouts between clusters from the first priority path set, ensuring cluster communication speed and preventing cluster lease expiration due to cluster communication timeouts; simultaneously, when there is no path in the first priority path set, selecting a path from the second priority path set for communication ensures continued service operation and avoids abnormal service interruptions.

Description

Technical Field

[0001] This invention relates to the field of storage multi-controller cluster technology, specifically to a link adaptive fault-tolerant method, apparatus, and server for storage multi-controller clusters. Background Technology

[0002] With the continuous development of IT technology, data centers in industries such as finance have increasingly higher requirements for the reliability of storage systems, often requiring 99.9999% reliability. This places strict requirements on the reliability of storage systems, leading to the emergence of multi-controller clusters in software testing activities.

[0003] Currently, storage multi-controller cluster links mainly exist in two states: normal and faulty. However, due to the complexity of their environment, storage multi-controller cluster links may also exist in states between the two. For example, when clusters communicate via fiber optic links, errors or delays may occur due to cable folds, damage, or low power of optical modules. There may also be intermittent link interruptions caused by problems with optical modules or switches. These issues can cause inter-cluster communication I / O (Input / Output) to repeatedly retry or be blocked on unhealthy links, resulting in I / O timeouts. If the timeout period reaches the cluster lease expiration time, the cluster lease will expire, leading to abnormal cluster services.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a link adaptive fault-tolerant method, apparatus and server for storage multi-controller clusters, to solve the problem that existing storage multi-controller clusters, due to the addition of optical fiber and other connections, increase the fault domain, causing cluster communication I / O to repeatedly retry or be blocked on unhealthy links, resulting in I / O timeouts, which may eventually lead to cluster lease expiration.

[0006] In a first aspect, the present invention provides a link adaptive fault-tolerant method for storage multi-controller clusters. The method includes: obtaining a first priority path set and selecting a path from the first priority path set for communication between storage multi-controller clusters; wherein each path includes at least one link; when the communication performance of any path in the first priority path set does not match a preset communication performance, the path that does not match the preset communication performance is migrated from the first priority path set to a second priority path set; when the paths in the first priority path set are empty, a path is selected from the second priority path set for communication between storage multi-controller clusters. Through the above process, paths corresponding to links that may cause large communication delays or timeouts between clusters can be removed from the first priority path set, maximizing the cluster communication rate and preventing cluster lease expiration due to communication timeout failures; simultaneously, when the paths in the first priority path set are empty, a path is selected from the second priority path set for communication between storage multi-controller clusters to ensure that services can continue to operate and to prevent abnormal service interruptions between clusters.

[0007] In one alternative implementation, after selecting a path from the first priority path set for communication between storage multi-controller clusters, the method further includes:

[0008] When any path in the first priority path set fails, the failed path is migrated from the first priority path set to the failed path set; wherein, path failure includes physical connection loss of any link in the path, and / or node failure of any link.

[0009] Test routines are sent to each fault path in the fault path set to detect whether the fault path has restored normal communication;

[0010] When a faulty path resumes normal communication, it will be migrated from the faulty path set to the first priority path set.

[0011] In one alternative implementation, paths that do not conform to preset communication performance include:

[0012] A path whose communication performance does not match the preset communication performance due to instability, delay, or verification error of any link in the path.

[0013] In one optional implementation, when the communication performance of any path in the first priority path set does not match the preset communication performance, the step of migrating the path that does not match the preset communication performance from the first priority path set to the second priority path set includes:

[0014] When the number of outages of any link in the first priority path set exceeds a preset threshold within a preset time, the path is migrated from the first priority path set to the second priority path set.

[0015] When any link corresponding to any path in the first priority path set has a first preset percentage of input / output data greater than the delay time under the routine test program, the path is migrated from the first priority path set to the second priority path set.

[0016] Calculate the newly added cyclic redundancy check error on the link corresponding to any path in the first priority path set. When the increase of the newly added cyclic redundancy check error is greater than the second preset percentage compared with the previous calculation node, the path is migrated from the first priority path set to the second priority path set.

[0017] In an optional implementation, after migrating paths that do not conform to preset communication performance from a first priority path set to a second priority path set, the method further includes:

[0018] Deploy test routines to each degraded path in the second priority path set;

[0019] When the communication performance of the downgraded path matches the preset communication performance within a preset time, the downgraded path will be migrated from the second priority path set to the first priority path set.

[0020] In an optional implementation, after migrating paths that do not conform to preset communication performance from a first priority path set to a second priority path set, the method further includes:

[0021] When the number of outages of any link in the second priority path set exceeds a preset threshold within a preset time, the path is migrated from the second priority path set to the fault path set.

[0022] In one alternative implementation, the method further includes:

[0023] When the paths in the first priority path set and the second priority path set are empty, a cluster communication failure alarm is reported.

[0024] Secondly, this invention provides a link adaptive fault-tolerant device for a storage multi-controller cluster. This fault-tolerant device mainly includes: a first path selection module, a path migration module, and a second path selection module. The first path selection module is used to obtain a first priority path set and select paths from the first priority path set for communication between storage multi-controller clusters. Each path includes at least one link. The path migration module is used to migrate paths with performance inconsistent with preset communication performance from the first priority path set to a second priority path set when the communication performance of any path in the first priority path set does not match the preset communication performance. The second path selection module is used to select paths from the second priority path set for communication between storage multi-controller clusters when the paths in the first priority path set are empty. Through the above process, paths corresponding to links that may cause large communication delays or timeouts between clusters can be removed from the first priority path set, maximizing the cluster communication rate and preventing cluster lease expiration due to communication timeout failures. Simultaneously, when the paths in the first priority path set are empty, paths are selected from the second priority path set for communication between storage multi-controller clusters to ensure continued service operation and prevent abnormal service interruptions between clusters.

[0025] Thirdly, the present invention provides a server, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the link adaptive fault-tolerant method of the storage multi-controller cluster described in the first aspect or any corresponding embodiment.

[0026] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the link adaptive fault-tolerant method for a storage multi-controller cluster as described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0027] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of an application environment according to an embodiment of the present invention;

[0029] Figure 2 This is a flowchart illustrating the link adaptive fault tolerance method for a storage multi-controller cluster according to an embodiment of the present invention.

[0030] Figure 3This is a flowchart illustrating another link adaptive fault-tolerant method for a storage multi-controller cluster according to an embodiment of the present invention.

[0031] Figure 4 This is a flowchart illustrating another link adaptive fault-tolerant method for a storage multi-controller cluster according to an embodiment of the present invention.

[0032] Figure 5 This is a data flow diagram of a link adaptive fault-tolerant method for another storage multi-controller cluster according to an embodiment of the present invention;

[0033] Figure 6 This is a structural block diagram of the link adaptive fault-tolerant device for a storage multi-controller cluster according to an embodiment of the present invention;

[0034] Figure 7 This is a schematic diagram of the hardware structure of the server according to an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided by an embodiment of the present invention. The schematic diagram includes a first storage multi-controller server 100 and a second storage multi-controller server 200. The first storage multi-controller server 100 and the second storage multi-controller server 200 are connected through a fiber optic switch 300 to form a storage multi-controller cluster. The communication path between the first storage multi-controller server 100 and the second storage multi-controller server 200 forms a cluster link for communication between the storage multi-controller clusters, that is, each path includes at least one link.

[0037] Cluster software is deployed on each storage multi-controller server to form a storage multi-controller cluster system. The storage multi-controller cluster system uses the cluster software to allocate all cluster links to a first-priority path set and selects paths within this set for communication. When any path in the first-priority path set fails, the failed path is migrated from the first-priority path set to a second-priority path set, and migrated back to the first-priority path set when normal communication is detected. When the communication performance of any path in the first-priority path set does not match the preset communication performance, the path is migrated from the first-priority path set to a second-priority path set, and degraded paths in this second-priority path set can still be migrated back to the first-priority path set or the second-priority path set based on their communication performance. When the first-priority path set is empty, a path is selected from the second-priority path set for communication between the storage multi-controller clusters. When both the first-priority and second-priority path sets are empty, a cluster communication failure alarm is reported.

[0038] The two storage multi-controller servers mentioned here are just examples. Storage multi-controller clusters include dual-controller, quad-controller, six-controller, eight-controller, sixteen-controller, and even more. The cluster link is the communication path between storage multi-controller servers. For example, one dual-controller storage server is one control cabinet. When two dual-controller storage servers form a storage multi-controller cluster, they need to use the Fiber Channel (FC) protocol or other protocols to connect and communicate. The cluster link here can be the communication link between two dual-controller storage servers.

[0039] According to an embodiment of the present invention, a link adaptive fault tolerance method for a storage multi-controller cluster is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] This embodiment provides a link adaptive fault tolerance method for a storage multi-controller cluster, which can be used in the aforementioned storage multi-controller server. Figure 2 This is a flowchart of a link adaptive fault-tolerant method for a storage multi-controller cluster according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0041] Step S201: Obtain the first priority path set, and select a path from the first priority path set for communication between the storage multi-controller clusters.

[0042] In this embodiment, communication connections are first established between the various storage multi-controller servers to obtain a storage multi-controller cluster. The cluster links formed by the communication paths between the storage multi-controller clusters are allocated to an optimal path set, and a path is selected from the optimal path set for communication to ensure the efficiency and quality of communication between the storage multi-controller clusters. Each path includes at least one link.

[0043] In one optional implementation, two storage multi-controller servers are operating normally and connected to a fiber optic switch via fiber optic cables, with configured ports (zones) for communication. The two storage multi-controller servers establish a storage multi-controller cluster system using cluster software, and the path health status monitoring program for the storage multi-controller cluster system is started. Simultaneously, all cluster links formed by communication paths between the storage multi-controller clusters are assigned to a first-priority path set, and a path is selected from this first-priority path set for cluster communication. That is, based on the input / output data status of each physical link between the storage multi-controller servers, the optimal path is dynamically selected to improve the efficiency, reliability, and stability of input / output data transmission. The path health status program is actually a submodule within the cluster software; it can be understood as an independent process running on the main configuration node of the storage multi-controller cluster to monitor the status of the storage multi-controller cluster links.

[0044] Optionally, when selecting a path from the first priority path set for communication between storage multi-controller clusters, a routing algorithm can be used to select a path from the first priority path set for communication between storage multi-controller clusters. This routing algorithm includes, but is not limited to, round-robin scheduling algorithm, minimum input / output data queue depth scheduling algorithm, and minimum input / output data task volume scheduling algorithm.

[0045] When the number and size of input / output data queues on each physical link are roughly equal, a round-robin scheduling algorithm can be used for path selection, i.e., input / output data is sent in a round-robin fashion on each link. When the number of input / output data queues on each physical link differs significantly between some links, a minimum input / output data queue depth scheduling algorithm can be used for path selection, i.e., the algorithm selects the path with the relatively smaller number of input / output data queues to send input / output data for transmission. When the number of bytes in the input / output data queue on a certain physical link is large, a minimum input / output data workload scheduling algorithm can be used, i.e., the current input / output data is inserted into the queue with the relatively high data transmission pressure.

[0046] For example, two storage multi-controller servers are connected via a fiber optic switch. The cluster software allocates all cluster links to the first priority path set (OptPathSet), forming OptPathSet as [path1, path2, ... pathN]. In the initial state, the storage multi-controller cluster system sends input / output data only to the paths selected in OptPathSet according to the routing algorithm. If the routing algorithm is a round-robin scheduling algorithm, then the input / output data of the cluster communication is sent to all paths in OptPathSet in turn.

[0047] Step S202: When the communication performance of any path in the first priority path set does not match the preset communication performance, the path that does not match the preset communication performance is migrated from the first priority path set to the second priority path set.

[0048] In this embodiment, after input / output data is routed and sent from the first priority path set, the communication performance of each path is monitored by the path health status monitoring program. When the communication performance of any path in the first priority path set does not match the preset communication performance, the path that does not match the preset communication performance is migrated from the first priority path set to the second priority path set, so that it no longer participates in the communication between the storage multi-controller clusters. This ensures that the communication link between the storage multi-controller clusters is always in a healthy online state, avoiding communication timeouts or failures of the storage multi-controller clusters due to unstable links, high latency, bit errors, etc.

[0049] Optionally, the paths that do not conform to the preset communication performance include: paths whose communication performance does not conform to the preset communication performance due to instability, delay, or verification errors of any link in the path. Link instability can include intermittent link interruptions, such as data interruptions caused by fiber optic switch problems; link delay can include link delay data caused by intermittent link interruptions and / or link error data; link verification errors can include data errors on the link caused by fiber optic cable folding, damage, or low optical module power.

[0050] In one optional implementation, if any link corresponding to path3 experiences bit errors due to fiber optic cable folding, damage, or low optical module power, or if the link repeatedly disconnects due to fiber optic switch issues, then path3 is removed from OptPathSet and placed in the second priority path set (DegradePathSet). At this time, OptPathSet is [path1, path4…pathN], and DegradePathSet is [path3]. If other links also experience disconnections or bit errors, OptPathSet and DegradePathSet will be further updated. When there are available paths in OptPathSet, only paths are selected from OptPathSet to send input / output data for communication between storage multi-controller clusters. This ensures that input / output data between clusters will not experience performance degradation or input / output blocking due to fault retries.

[0051] Step S203: When the path in the first priority path set is empty, select a path from the second priority path set for communication between the storage multi-controller clusters.

[0052] In this embodiment, when the path in the first priority path set is empty, a path is selected from the second priority path set for communication between the storage multi-controller clusters, so as to temporarily maintain the simple operation of the storage multi-controller cluster system, rather than directly causing the storage multi-controller cluster to crash.

[0053] In one alternative implementation, if all paths in OptPathSet fail or degrade, resulting in no paths available in OptPathSet, the storage multi-controller cluster system will select a path from DegradePathSet to send input / output data for communication between storage multi-controller clusters, thereby ensuring that services can continue to run. Although there is a certain delay or timeout, communication between storage multi-controller clusters is still possible.

[0054] The link adaptive fault-tolerant method for storage multi-controller clusters provided in this embodiment obtains a first priority path set and selects a path from the first priority path set for communication to ensure the efficiency and quality of communication between storage multi-controller clusters. When the communication performance of any path in the first priority path set does not match the preset communication performance, the path that does not match the preset communication performance is migrated from the first priority path set to the second priority path set, so that it no longer participates in the input / output data communication between storage multi-controller clusters. This ensures that the communication link between storage multi-controller clusters is always in a healthy online state, avoiding communication timeouts or failures caused by link instability, high latency, bit errors, etc. When the path in the first priority path set is empty, a path is selected from the second priority path set for communication between storage multi-controller clusters to temporarily maintain the simple operation of the storage multi-controller cluster system, rather than directly causing the storage multi-controller cluster to crash.

[0055] This embodiment provides a link adaptive fault tolerance method for a storage multi-controller cluster, which can be used in the aforementioned storage multi-controller server. Figure 3 This is a flowchart of a link adaptive fault-tolerant method for a storage multi-controller cluster according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0056] Step S301: Obtain the first priority path set, and select a path from the first priority path set for communication between the storage multi-controller clusters.

[0057] Please see details Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0058] Step S302: When the communication performance of any path in the first priority path set does not match the preset communication performance, the path that does not match the preset communication performance is migrated from the first priority path set to the second priority path set.

[0059] Specifically, step S302 includes:

[0060] Step S3021: When the number of interruptions of the link corresponding to any path in the first priority path set within a preset time exceeds a preset threshold, the path is migrated from the first priority path set to the second priority path set.

[0061] In this embodiment, when the number of intermittent disconnections of any link corresponding to a path in the first priority path set exceeds a preset threshold within a preset time, that path is migrated from the first priority path set to the second priority path set, preventing it from participating in input / output data communication between storage multi-controller clusters. This ensures that the communication links between storage multi-controller clusters are always in a healthy online state, avoiding communication timeouts or failures caused by link instability. The specific values ​​of the preset time, the number of intermittent disconnections within the preset time, and the preset threshold can be adjusted according to actual needs.

[0062] In one optional implementation, if path 3, corresponding to a link, experiences repeated intermittent disconnections due to a fiber optic switch issue (more than 3 disconnections within 30 minutes), indicating link instability, then path 3 is removed from the OptPathSet and placed in the second priority path set (DegradePathSet). At this point, OptPathSet is [path1, path4…pathN], and DegradePathSet is [path3]. If other links also experience intermittent disconnections or errors, OptPathSet and DegradePathSet will be further updated. When a link goes offline, the driver returns a linkdown error code to the storage system. The storage system then marks the cluster link as faulty and removes it from the preferred path set. After the link recovers, it reverts from faulty to preferred. This repeated occurrence demonstrates link instability; although it may appear usable intermittently, it is unstable overall. This is called link instability degradation.

[0063] Step S3022: When any link in the first priority path set has a first preset percentage of input / output data greater than the delay time under the routine test program, the path is migrated from the first priority path set to the second priority path set.

[0064] In this embodiment, when any path in the first priority path set has a first preset percentage of input / output data exceeding the delay time under the routine test program, that path is migrated from the first priority path set to the second priority path set. This ensures that the communication links between the storage multi-controller clusters are always in a healthy online state, avoiding communication timeouts or failures caused by high link latency. The specific values ​​of the first preset percentage and the delay time can be adjusted according to actual needs.

[0065] In one optional implementation, if the path corresponding to the link is path3, the link response latency is checked by the routine test program. If 50% of the 100 input / output data exceed 500ms, it is considered to be a high latency link and should be degraded to high latency. Path3 is removed from OptPathSet and placed in the second priority path set (DegradePathSet). At this time, OptPathSet is [path1, path4...pathN] and DegradePathSet is [path3]. If other links also experience intermittent interruptions or bit errors, OptPathSet and DegradePathSet will be updated further.

[0066] Step S3023: Calculate the newly added cyclic redundancy check error on the link corresponding to any path in the first priority path set. When the increase of the newly added cyclic redundancy check error is greater than the second preset percentage compared with the previous calculation node, the path is migrated from the first priority path set to the second priority path set.

[0067] In this embodiment, by periodically, irregularly, or through computational requests, the newly added cyclic redundancy check (CRC) error on the link corresponding to any path in the first priority path set is calculated. When the increase in the newly added CRC error exceeds a second preset percentage compared to the previous calculation node, the path is migrated from the first priority path set to the second priority path set. This ensures that the communication link between the storage multi-controller clusters is always in a healthy online state, avoiding communication timeouts or failures caused by link errors. The value of the second preset percentage can be adjusted according to actual needs.

[0068] In one optional implementation, if the path corresponding to the link, path3, periodically or irregularly counts the new cyclic redundancy check errors (mostly caused by bit errors) that occur due to fiber optic cable folding, damage, or low optical module power, and the increase exceeds 80% compared to the previous cycle, then the link is set to bit error degradation, and path3 is removed from OptPathSet and placed in the second priority path set (DegradePathSet). At this time, OptPathSet is [path1, path4…pathN], and DegradePathSet is [path3]. If other links also experience intermittent outages or bit errors, OptPathSet and DegradePathSet will be further updated.

[0069] Step S303: Send the test routine to each degraded path in the second priority path set.

[0070] In one optional implementation, the path health status monitoring program periodically, irregularly, or upon request, distributes routine test programs to paths in the second priority path set. Routine test programs can be distributed to each degraded path in the second priority path set every 5 seconds. These routine test programs are management I / O operations proactively distributed by the cluster software to routinely distribute I / O across all cluster links and check the corresponding I / O data.

[0071] Step S304: When the communication performance of the downgraded path matches the preset communication performance within a preset time, the downgraded path is migrated from the second priority path set to the first priority path set.

[0072] In one optional implementation, when the degraded path does not experience any further interruptions or bit errors within 30 minutes and there is no delay, the degraded path is migrated from the second priority path set to the first priority path set.

[0073] It should be noted that after a path is downgraded, the driver will return a linkdown error code to the storage multi-controller cluster system. The storage multi-controller cluster system will remove the path from the preferred path set. After the link is restored, it will become a preferred path again. If this happens repeatedly, it proves that the link is unstable. Although it may seem to work intermittently, it is unstable overall. Therefore, this downgrade is a link instability downgrade.

[0074] Step S305: When the number of interruptions of any link in the second priority path set within a preset time exceeds a preset threshold, the path corresponding to that link is migrated from the second priority path set to the fault path set.

[0075] In one optional implementation, when the number of interruptions on a degraded path exceeds three within 30 minutes, the degraded path is migrated from the second priority path set to the fault path set. The specific values ​​of the preset time, the number of interruptions within the preset time, and the preset threshold can be adjusted according to actual needs.

[0076] Step S306: When the path in the first priority path set is empty, select a path from the second priority path set for communication between the storage multi-controller clusters.

[0077] Please see details Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0078] Step S307: When the paths in the first priority path set and the second priority path set are empty, report a cluster communication failure alarm.

[0079] The link adaptive fault-tolerance method for storage multi-controller clusters provided in this embodiment obtains a first priority path set and selects a path from the first priority path set for communication to ensure the efficiency and quality of communication between storage multi-controller clusters. When the communication performance of any path in the first priority path set does not match the preset communication performance, the path that does not match the preset communication performance is migrated from the first priority path set to the second priority path set, so that it no longer participates in the input / output data communication between storage multi-controller clusters. This ensures that the communication link between storage multi-controller clusters is always in a healthy online state, avoiding communication timeouts or failures caused by link errors. When the path in the first priority path set is empty, a path is selected from the second priority path set for communication between storage multi-controller clusters to temporarily maintain the storage multi-controller clusters. The system operates smoothly without directly causing the storage multi-controller cluster to crash. Test programs are distributed to each degraded path in the second-priority path set. When the communication performance of a degraded path matches the preset communication performance within a preset time, it is migrated from the second-priority path set to the first-priority path set, thus preventing normally communicating paths from remaining in the second-priority path set. Degraded paths in the second-priority path set that cannot restore normal communication within a preset time are migrated from the second-priority path set to the fault path set to ensure the reliability of the second-priority path set and remind users to promptly troubleshoot faulty paths. When paths in both the first and second-priority path sets are empty, a cluster communication fault alarm is reported to remind users to promptly troubleshoot faulty paths.

[0080] This embodiment provides a link adaptive fault tolerance method for a storage multi-controller cluster, which can be used in the aforementioned storage multi-controller server. Figure 4 This is a flowchart of a link adaptive fault-tolerant method for a storage multi-controller cluster according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps:

[0081] Step S401: Obtain the first priority path set, and select a path from the first priority path set for communication between the storage multi-controller clusters.

[0082] Please see details Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0083] Step S402: When any path in the first priority path set fails, the failed path is migrated from the first priority path set to the failed path set.

[0084] Path failures include physical disconnection of any link in the path and / or node failure of any link.

[0085] In one optional implementation, if the physical connection of any link in path2 is broken, and / or any link node fails, then path2 is removed from OptPathSet and placed into the fault path set (FaultPathSet). At this time, OptPathSet is [path1, path3…pathN], and FaultPathSet is [path2]. If other links also become broken, OptPathSet and FaultPathSet will be further updated. All path failures will follow the international standard protocol T10.org. For example, if a link is disconnected, the driver will return a linkdown error code to the storage multi-controller cluster system, and the storage multi-controller cluster system will set the path to fault according to predetermined logic.

[0086] Step S403: Send routine test programs to each fault path in the fault path set to detect whether the fault path has restored normal communication.

[0087] In one optional implementation, the path health status monitoring program sends routine test programs to the paths in the fault path set every 5 seconds. The frequency of sending routine test programs to each fault path in the fault path set can be adjusted according to actual needs, and can also be non-periodic, or sent based on routine test requests.

[0088] Step S404: When the faulty path resumes normal communication, the faulty path is migrated from the faulty path set to the first priority path set.

[0089] In this embodiment, when the faulty path resumes normal communication, the faulty path is migrated from the faulty path set to the first priority path set to enable normal communication.

[0090] Step S405: When the communication performance of any path in the first priority path set does not match the preset communication performance, the path that does not match the preset communication performance is migrated from the first priority path set to the second priority path set.

[0091] Please see details Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0092] Step S406: When the path in the first priority path set is empty, select a path from the second priority path set for communication between the storage multi-controller clusters.

[0093] Please see details Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0094] The link adaptive fault-tolerant method for storage multi-controller clusters provided in this embodiment obtains a first priority path set and selects a path from the first priority path set for communication to ensure the efficiency and quality of communication between storage multi-controller clusters. When the communication performance of any path in the first priority path set does not match the preset communication performance, the path that does not match the preset communication performance is migrated from the first priority path set to the second priority path set, so that it no longer participates in the input / output data communication between storage multi-controller clusters, thereby ensuring that the communication link between storage multi-controller clusters is always in a healthy online state and avoiding communication timeouts or failures caused by link errors. When the path in the first priority path set is empty, a path is selected from the second priority path set for communication between storage multi-controller clusters to temporarily maintain the simple operation of the storage multi-controller cluster system, rather than directly causing the storage multi-controller cluster to crash. By issuing routine test programs to each faulty path in the faulty path set, when the faulty path recovers normal communication, the faulty path is migrated from the faulty path set to the first priority path set, thereby avoiding the path with normal communication remaining in the faulty path set.

[0095] As one or more specific application embodiments of the present invention, such as Figure 5 As shown, it includes:

[0096] First, the two storage multi-controller servers are running normally and connected to a fiber optic switch via fiber optic cable, with configured ports (zones) for communication. The two storage multi-controller servers then establish a storage multi-controller cluster system using cluster software, and the path health monitoring program for the storage multi-controller cluster system is started.

[0097] Secondly, the cluster software allocates all cluster links to the first priority path set (OptPathSet), forming OptPathSet[path1,path2…pathN]. Initially, the multi-controller cluster system distributes cluster input / output data only to paths selected within (OptPathSet) based on the routing algorithm. If the routing algorithm is round-robin, the cluster communication distributes cluster input / output data to all paths in the (OptPathSet) path set in turn. Each path includes at least one link.

[0098] Third, during the operation of the storage multi-controller cluster system, if a link in path 2 is broken, path 2 will be removed from OptPathSet and placed into the FaultPathSet. At this time, OptPathSet will be [path1, path3…pathN], and FaultPathSet will be [path2]. If links are also broken in other paths, OptPathSet and FaultPathSet will be updated further. When there are available paths in OptPathSet, cluster input / output data will only be sent from the path set in OptPathSet for inter-cluster business communication. This ensures that cluster input / output data will not suffer from performance degradation or input / output blocking due to fault retries. At this time, the path health monitoring program will send a routine test program to the path in FaultPathSet every 5 seconds. If the path is restored, the link will be removed from FaultPathSet and put back into OptPathSet.

[0099] Fourth, if path 3 experiences bit errors due to fiber optic cable folding, damage, or low optical module power, or if the link repeatedly disconnects due to switch problems, with more than 3 disconnections within 30 minutes, then path 3 will be removed from OptPathSet and placed in the second priority path set (DegradePathSet). At this time, OptPathSet will be [path1, path4…pathN], FaultPathSet will be [path2], and DegradePathSet will be [path3]. If other links also experience intermittent outages or bit errors, OptPathSet and DegradePathSet will be further updated. When an available path exists in OptPathSet, only paths from the OptPathSet will be selected to send input / output data for inter-cluster communication. This ensures that cluster input / output data will not experience performance degradation or input / output blocking due to fault retries. At this time, the path health monitoring program will send a routine test program to DegradePathSet every 5 seconds. If the path does not experience intermittent outages or bit errors within 30 minutes, the path will be removed from DegradePathSet and put back into OptPathSet. If a degraded path in DegradePathSet becomes disconnected, the degraded path will be transferred from DegradePathSet to FaultPathSet.

[0100] Fifth, if all paths in OptPathSet fail or degrade, resulting in no paths available in OptPathSet, the cluster system will select a path from DegradePathSet to send cluster input / output data for inter-cluster communication, thereby ensuring that services can continue to run. Although there is some latency or timeout, inter-cluster communication is still possible.

[0101] Sixth, if both OptPathSet and DegradePathSet are empty, the cluster will report a cluster communication failure alarm, indicating that there are no available paths between the entire cluster and that services between the clusters will be abnormally interrupted.

[0102] The link adaptive fault-tolerance method for storage multi-controller clusters provided in this embodiment obtains a first priority path set and selects a path from the first priority path set for communication to ensure the efficiency and quality of communication between storage multi-controller clusters. When the communication performance of any path in the first priority path set does not match the preset communication performance, the path that does not match the preset communication performance is migrated from the first priority path set to the second priority path set, so that it no longer participates in the input / output data communication between storage multi-controller clusters. This ensures that the communication link between storage multi-controller clusters is always in a healthy online state, avoiding communication timeouts or failures caused by link errors. When the path in the first priority path set is empty, a path is selected from the second priority path set for communication between storage multi-controller clusters to temporarily maintain the storage multi-controller clusters. The system operates smoothly without directly causing the storage multi-controller cluster to crash. Test programs are distributed to each degraded path in the second-priority path set. When the communication performance of a degraded path matches the preset communication performance within a preset time, it is migrated from the second-priority path set to the first-priority path set, thus preventing normally communicating paths from remaining in the second-priority path set. Degraded paths in the second-priority path set that cannot restore normal communication within a preset time are migrated from the second-priority path set to the fault path set to ensure the reliability of the second-priority path set and remind users to promptly troubleshoot faulty paths. When paths in both the first and second-priority path sets are empty, a cluster communication fault alarm is reported to remind users to promptly troubleshoot faulty paths.

[0103] This embodiment also provides a link-adaptive fault-tolerant device for a storage multi-controller cluster. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0104] This embodiment provides a link adaptive fault-tolerant device for a storage multi-controller cluster, such as... Figure 6 As shown, it includes:

[0105] The first path selection module 601 is used to obtain a first priority path set and select a path from the first priority path set for communication between the storage multi-controller clusters.

[0106] Each path includes at least one link.

[0107] The path migration module 602 is used to migrate the path that does not match the preset communication performance from the first priority path set to the second priority path set when the communication performance of any path in the first priority path set does not match the preset communication performance.

[0108] Among them, paths that do not conform to the preset communication performance include:

[0109] A path whose communication performance does not match the preset communication performance due to instability, delay, or verification error of any link in the path.

[0110] In some alternative implementations, the path migration module 602 includes:

[0111] The first migration unit is used to migrate the path from the first priority path set to the second priority path set when the number of interruptions of the link corresponding to any path in the first priority path set within a preset time exceeds a preset threshold.

[0112] The first migration unit is further configured to migrate the faulty path from the first priority path set to the faulty path set when any path in the first priority path set fails. Path failures include physical disconnection of any link in the path and / or node failure of any link.

[0113] The second migration unit is used to migrate the path from the first priority path set to the second priority path set when the link corresponding to any path in the first priority path set has a first preset percentage of input / output data greater than the delay time under the routine test program.

[0114] The second migration unit is used to calculate the newly added cyclic redundancy check error on the link corresponding to any path in the first priority path set. When the increase of the newly added cyclic redundancy check error is greater than the second preset percentage compared with the previous calculation node, the path is migrated from the first priority path set to the second priority path set.

[0115] The second path selection module 603 is used to select a path from the second priority path set for communication between storage multi-controller clusters when the path in the first priority path set is empty.

[0116] In some alternative implementations, the second path selection module 603 includes:

[0117] The routine test distribution unit is used to distribute routine test programs to each degraded path in the second priority path set.

[0118] The routine test distribution unit is also used to distribute routine test programs to each fault path in the fault path set in order to detect whether the fault path has restored normal communication.

[0119] The fourth migration unit is used to migrate the degraded path from the second priority path set to the first priority path set when the communication performance of the degraded path matches the preset communication performance within a preset time.

[0120] The fourth migration unit is also used to migrate the faulty path from the faulty path set to the first priority path set when the faulty path resumes normal communication.

[0121] The fifth migration unit migrates the path corresponding to any path in the second priority path set from the second priority path set to the fault path set when the number of intermittent interruptions within a preset time exceeds a preset threshold.

[0122] In some optional implementations, the device further includes a fault alarm module for reporting a cluster communication fault alarm when the paths in the first priority path set and the second priority path set are empty.

[0123] In this embodiment, the link adaptive fault-tolerant device of the storage multi-controller cluster is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0124] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0125] This invention also provides a server having the above-described features. Figure 6 The storage multi-controller cluster shown is a link adaptive fault-tolerant device.

[0126] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a server provided in an optional embodiment of the present invention, such as... Figure 7As shown, the server includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the server, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple servers can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 7 Take a processor 10 as an example.

[0127] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0128] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0129] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the server based on the display of a mini-program landing page, etc. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories can be connected to the server via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0130] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0131] The server also includes a communication interface 30 for communicating with other devices or communication networks.

[0132] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0133] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A link-adaptive fault-tolerant method for a storage multi-controller cluster, characterized in that, The method includes: Obtain a first priority path set, and select a path from the first priority path set for communication between storage multi-controller clusters; wherein each path includes at least one link; When the communication performance of any path in the first priority path set does not match the preset communication performance, the path that does not match the preset communication performance will be migrated from the first priority path set to the second priority path set. When the path in the first priority path set is empty, a path is selected from the second priority path set for communication between the storage multi-controller clusters. When the communication performance of any path in the first priority path set does not match the preset communication performance, the step of migrating the path that does not match the preset communication performance from the first priority path set to the second priority path set includes: When the number of outages of any link in the first priority path set exceeds a preset threshold within a preset time, the path is migrated from the first priority path set to the second priority path set. When any link corresponding to any path in the first priority path set has a first preset percentage of input / output data greater than the delay time under the routine test program, the path is migrated from the first priority path set to the second priority path set. Calculate the newly added cyclic redundancy check error on the link corresponding to any path in the first priority path set. When the increase of the newly added cyclic redundancy check error is greater than the second preset percentage compared with the previous calculation node, the path is migrated from the first priority path set to the second priority path set.

2. The method according to claim 1, characterized in that, After selecting a path from the first priority path set for communication between the storage multi-controller clusters, the method further includes: When any path in the first priority path set fails, the failed path is migrated from the first priority path set to the failed path set; wherein, path failure includes physical connection disconnection of any link in the path, and / or node failure of any link; A routine test program is sent to each fault path in the fault path set to detect whether the fault path has restored normal communication. When the faulty path resumes normal communication, the faulty path is migrated from the faulty path set to the first priority path set.

3. The method according to claim 2, characterized in that, The paths that do not conform to the preset communication performance include: A path whose communication performance does not match the preset communication performance due to instability, delay, or verification error of any link in the path.

4. The method according to claim 1, characterized in that, After migrating the path that does not meet the preset communication performance from the first priority path set to the second priority path set, the method further includes: Send example test programs to each degraded path in the second priority path set; When the communication performance of the downgraded path matches the preset communication performance within a preset time, the downgraded path is migrated from the second priority path set to the first priority path set.

5. The method according to claim 4, characterized in that, After migrating paths that do not meet preset communication performance from the first priority path set to the second priority path set, the method further includes: When the number of outages of any link in the second priority path set exceeds a preset threshold within a preset time, the path corresponding to that link is migrated from the second priority path set to the fault path set.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: When the paths in the first priority path set and the second priority path set are empty, a cluster communication failure alarm is reported.

7. A link-adaptive fault-tolerant device for a storage multi-controller cluster, characterized in that, The device includes: The first path selection module is used to obtain a first priority path set and select a path from the first priority path set for communication between storage multi-controller clusters; wherein each path includes at least one link. A path migration module is configured to migrate any path in the first priority path set to a second priority path set when the communication performance of any path in the first priority path set does not match the preset communication performance. This includes: When the number of interruptions of any link corresponding to any path in the first priority path set exceeds a preset threshold within a preset time, the path is migrated from the first priority path set to the second priority path set; when the link corresponding to any path in the first priority path set has a first preset percentage of input / output data exceeding the delay time under the routine test program, the path is migrated from the first priority path set to the second priority path set; calculate the newly added cyclic redundancy check error on the link corresponding to any path in the first priority path set, and when the increase of the newly added cyclic redundancy check error is greater than a second preset percentage compared to the previous calculation node, the path is migrated from the first priority path set to the second priority path set; The second path selection module is used to select a path from the second priority path set for communication between storage multi-controller clusters when the path in the first priority path set is empty.

8. A server, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 6.

Images