Centralized storage system fault processing method and electronic device

Abstract

The application discloses a centralized storage system fault processing method and electronic equipment, and relates to the technical field of centralized storage systems. Hardware resources of the centralized storage system are used to divide multiple processors in multiple nodes in the system into multiple resource groups, an independently running software instance is created according to each resource group, the software instance is distributed to each node, and a target instance for processing a request and data is determined. When the target instance fails, instance switching is performed by a software controller, and the request and data are processed by a backup instance. Thus, when a failure occurs, a node does not need to exit a cluster, and node recombination is not needed, the problem of service interruption is avoided, and the performance of the centralized storage system is improved.

Description

Technical Field

[0001] This application relates to the field of centralized storage system technology, and in particular to a centralized storage system fault handling method and electronic device. Background Technology

[0002] Centralized storage systems are widely used in large-scale data processing scenarios. In practical applications, storage systems need to meet the requirements of high performance and data reliability. Any node failure or performance fluctuation can lead to business interruption and service quality degradation. Traditional centralized storage systems rely on a single control node to manage global data, and the failure of this single control node can easily paralyze the entire system.

[0003] Currently, centralized storage systems employ a distributed redundancy architecture to improve system reliability. However, in these technologies, when a node fails due to a software error, the cluster needs to restore redundancy through node reorganization. During the recovery process, surviving nodes must suspend operations to prevent requests from being sent to the failed node, thus causing service interruptions when a node fails. Summary of the Invention

[0004] This application provides a centralized storage system fault handling method and electronic device to at least solve the problem of service interruption when node failure occurs in related technologies.

[0005] This application provides a method for handling faults in a centralized storage system, including:

[0006] Obtain hardware resource information for the centralized storage system.

[0007] Based on the hardware resource information of the centralized storage system, multiple processors in multiple nodes within the centralized storage system are grouped to generate processor grouping information.

[0008] Multiple software instances are created based on the processor's grouping information.

[0009] Multiple software instances are allocated to multiple nodes within a centralized storage system, instance allocation information is generated, and target instances are determined based on the instance allocation information. The target instances are used to process requests and data from the centralized storage system.

[0010] In response to a failure of the target instance, the software controller performs an instance switching operation based on pending requests and pending data, and processes the pending requests and pending data based on the switched standby instance.

[0011] This application also provides a centralized storage system fault handling device, including:

[0012] The first acquisition module is used to acquire hardware resource information of the centralized storage system.

[0013] The grouping module is used to group multiple processors in multiple nodes within a centralized storage system based on the hardware resource information of the centralized storage system, and generate grouping information for the processors.

[0014] The first creation module is used to create multiple software instances based on the processor's grouping information.

[0015] The generation module is used to allocate multiple software instances to multiple nodes within a centralized storage system, generate instance allocation information, and determine target instances based on the instance allocation information. The target instances are used to process requests and data from the centralized storage system.

[0016] The switching module is used to respond to the failure of the target instance. It performs instance switching operations through the software controller based on the pending requests and pending data, and processes the pending requests and pending data based on the switched standby instance.

[0017] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described centralized storage system fault handling methods when executing the computer program.

[0018] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described centralized storage system fault handling methods.

[0019] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described centralized storage system fault handling methods.

[0020] By utilizing the hardware resources of a centralized storage system, multiple processors in multiple nodes are divided into multiple resource groups. Independently running software instances are created for each resource group, and these software instances are allocated to each node. The target instance for processing requests and data is determined. When the target instance fails, the software controller performs instance switching, and the backup instance processes the requests and data. This ensures that nodes do not need to leave the cluster or undergo reorganization when a failure occurs, thus avoiding business interruption and improving the performance of the centralized storage system. Attached Figure Description

[0021] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram illustrating an application scenario of the centralized storage system fault handling method provided in this application embodiment;

[0023] Figure 2 A flowchart illustrating the centralized storage system fault handling method provided in this application embodiment. Figure 1 ;

[0024] Figure 3 A flowchart illustrating the centralized storage system fault handling method provided in this application embodiment. Figure 2 ;

[0025] Figure 4 A flowchart illustrating the centralized storage system fault handling method provided in this application embodiment. Figure 3 ;

[0026] Figure 5 A flowchart illustrating the centralized storage system fault handling method provided in this application embodiment. Figure 4 ;

[0027] Figure 6 A flowchart illustrating the centralized storage system fault handling method provided in this application embodiment. Figure 5 ;

[0028] Figure 7 A schematic diagram of the structure of a centralized storage system fault handling device provided in an embodiment of this application;

[0029] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0031] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0032] First, let me explain the terms used in this application:

[0033] Example: Creating and running multiple independent, isolated environments at the software or logic level on a hardware platform.

[0034] Cluster: A collection of interconnected, independent storage nodes that work together to form a unified computing resource.

[0035] To address the issue of service interruption due to node failure in related technologies, this application proposes the following technical concept: Considering the acquisition of hardware resources in a centralized storage system, multiple processors in multiple nodes within the system are divided into multiple resource groups. Independently running software instances are created for each resource group and allocated to various nodes. Considering that the target instance for processing requests and data is determined based on the instance allocation information, when the target instance fails, instance switching is performed through the software controller, and requests and data are processed by the backup instance. This ensures that nodes do not need to leave the cluster or undergo node reorganization when a failure occurs, thus avoiding the problem of service interruption.

[0036] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] The specific application environment architecture or specific hardware architecture on which the execution of the centralized storage system fault handling method depends is described here.

[0038] refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario for the centralized storage system fault handling method provided in an embodiment of this application. For example... Figure 1 As shown, this application scenario includes a centralized storage system 101, which includes multiple nodes.

[0039] Specifically, the hardware resource information of the centralized storage system 101 is obtained. Based on the hardware resource information, multiple processors in multiple nodes are grouped to generate processor grouping information. Multiple software instances are created based on the processor grouping information, and these software instances are allocated to multiple nodes to generate instance allocation information. The target instance is then determined based on the instance allocation information. When the target instance fails, an instance switchover operation is performed through the software controller to switch to a standby instance, which then handles pending requests and data.

[0040] Figure 2 A flowchart illustrating the centralized storage system fault handling method provided in this application embodiment. Figure 1 ,like Figure 2 As shown, embodiments of this application provide a method for handling faults in a centralized storage system. The method is described in detail below:

[0041] S201: Obtain hardware resource information of the centralized storage system.

[0042] Specifically, the hardware configuration information of all nodes in the system is scanned and collected through the system management tools embedded in the centralized storage system.

[0043] In this embodiment, hardware resource information includes, but is not limited to: the model of the central processing unit (CPU) of each node, the number of physical cores, the number of logical cores, the clock speed, and the size of the L3 cache.

[0044] S202: Based on the hardware resource information of the centralized storage system, group multiple processors in multiple nodes within the centralized storage system to generate processor grouping information.

[0045] Specifically, the hardware resources of multiple nodes are evaluated for performance based on hardware resource information, multiple processors within multiple nodes are filtered based on the data processing capability evaluation values ​​of multiple nodes, multiple processors are grouped based on the filtered processor information, and multiple processor groups are associated with group identifiers to generate processor grouping information.

[0046] For example, hardware isolation groups are formed based on non-uniform memory access topology, individual processor nodes are identified, and groups are formed with non-uniform memory access as the boundary to ensure that each group of processors, local memory, and high-speed serial computer expansion bus channels are independent.

[0047] S203: Create multiple software instances based on the processor's grouping information.

[0048] Specifically, the instance type of the software instance is determined based on the processor's grouping information. The instance template is obtained by calling the preset instance template library based on the instance type. The configuration parameters of the software instance are determined based on the processor's grouping information. The configuration parameters of the software instance are written into the instance template to create multiple software instances.

[0049] S204: Assign multiple software instances to multiple nodes within a centralized storage system, generate instance allocation information, and determine the target instance based on the instance allocation information, wherein the target instance is used to process requests and data from the centralized storage system.

[0050] Specifically, based on the node load balancing strategy, multiple software instances are scheduled and deployed to the physical nodes where their bound processor groups are located, forming an instance allocation information table.

[0051] Specifically, from the software instances, one or a group of active instances are designated as target instances, responsible for receiving and processing input / output requests and data read / write operations from the upper-layer application.

[0052] S205: In response to a failure of the target instance, the software controller performs an instance switching operation based on the pending requests and pending data, and processes the pending requests and pending data based on the switched standby instance.

[0053] Specifically, in response to a failure of the target instance, the software controller obtains a set of backup instances, determines a list of backup instances based on the correlation between multiple backup instances in the set and the target instance, filters the backup instances based on their pre-set business priority tags, obtains the backup instances to be switched, sends the requests and data to be processed to the backup instances to be switched, and processes the requests and data according to the switched backup instances.

[0054] As can be seen from the above embodiments, by using the hardware resources of the centralized storage system, multiple processors in multiple nodes within the system are divided into multiple resource groups. Independently running software instances are created according to each resource group, and the software instances are allocated to each node. The target instance for processing requests and data is determined. When the target instance fails, the software controller performs instance switching, and the backup instance processes the requests and data. This ensures that when a failure occurs, the node does not need to leave the cluster or undergo node reorganization, thus avoiding business interruption and improving the performance of the centralized storage system.

[0055] In one embodiment of this application, step S205 includes:

[0056] S205a: In response to a failure of the target instance, obtain a set of backup instances through the software controller.

[0057] Specifically, after receiving the instance switching instruction, the software controller queries the system instance registry for all software instances that are currently in a "healthy" state and have not been marked as "target instances" to form a set of backup instances.

[0058] S205b: Determine the list of standby instances based on the correlation between multiple standby instances and the target instance in the standby instance collection.

[0059] Specifically, the correlation between each backup instance and the faulty target instance is calculated, and instances with high correlation are selected first.

[0060] In this embodiment, the correlation includes, but is not limited to, whether they are located within the same processor group or whether they are deployed on the same physical node.

[0061] S205c: Filter multiple standby instances in the standby instance list according to the pre-set business priority tags of the standby instances to obtain the standby instance to be switched.

[0062] Specifically, the software controller selects the first "low priority" standby instance in the list as the standby instance to be switched over, based on the list order.

[0063] In this embodiment, priority labels include, but are not limited to, high priority, medium priority, and low priority.

[0064] S205d: Sends pending requests and pending data to the standby instance to be switched over via the software controller, and processes the pending requests and pending data according to the standby instance to be switched over.

[0065] Specifically, a data link is created between the target instance and the standby instance to be switched over. The requests and data to be processed are transmitted to the standby instance to be switched over through the data link. Data access permissions of the target instance are obtained and synchronized to the standby instance to be switched over. The requests and data to be processed are processed according to the standby instance to be switched over.

[0066] As can be seen from the above embodiments, by filtering the correlation between multiple backup instances and the target instance in the backup instance collection, the backup instances are initially screened to obtain a backup instance list. A second screening is performed based on the business priority tags in the backup instance list to determine the backup instance to be switched. The software controller sends the requests and data to be processed to the backup instance to be switched, and the requests and data are processed by the high-priority backup instance, avoiding the preemption of low-priority instances and improving data processing efficiency.

[0067] Figure 3 A flowchart illustrating the centralized storage system fault handling method provided in this application embodiment. Figure 2 ,like Figure 3 As shown, step S205d includes:

[0068] S301: Create a data link between the target instance and the standby instance to be switched over, based on the configuration information of the target instance and the standby instance to be switched over. The data link is used to transmit pending requests and pending data.

[0069] In this embodiment, the data link is a dedicated, encrypted network transmission channel established through remote direct memory access technology, used to transmit requests and data blocks to be processed.

[0070] Specifically, after an instance joins the cluster, the controller selects a primary instance based on priority and health algorithms, configures the primary instance as the instance that provides services, participates in the cluster's data processing flow, maintains metadata, and synchronizes the primary instance's data through the local image.

[0071] S302: Transmit the pending requests and data to the standby instance to be switched over via the data link, and verify the data integrity.

[0072] Specifically, requests and data that are waiting to be processed or are being processed in the target instance's memory queue at the time of the failure are transferred in batches or streamed to the standby instance to be switched over via an established data link. After the transfer is complete, a checksum is calculated and compared to verify the integrity of the data during the transfer process.

[0073] S303: If the data integrity verification passes, obtain the data access permissions of the target instance and synchronize the data access permissions of the target instance to the standby instance to be switched over, so as to complete the switchover between the target instance and the standby instance.

[0074] Specifically, after data integrity verification is successful, the software controller, through the storage management interface, removes the mount and access permissions of the failed target instance to the backend file system. It then remounts the file system and grants access permissions to the standby instance to be switched over.

[0075] S304: Process pending requests and pending data based on the switched standby instance.

[0076] Specifically, after permission synchronization is complete, the software controller notifies the upstream load balancer to direct subsequent new requests to the standby instance. Simultaneously, the standby instance begins processing the pending requests and data taken over from the target instance, thus completing the service takeover.

[0077] As can be seen from the above embodiments, by obtaining the configuration information of the target instance and the standby instance, a data link is created between the target instance and the standby instance. The requests and data to be processed are transmitted to the standby instance through the data link. The integrity of the transmitted data is verified, and the data access permissions of the target instance are synchronized to the standby instance. The requests and data are processed by the standby instance, achieving seamless switching without leaving the cluster and avoiding business interruption.

[0078] In one embodiment of this application, step S202 includes:

[0079] S202a: Based on the hardware resource information of the centralized storage system, the performance of the hardware resources of multiple nodes is evaluated, and the data processing capability evaluation value of multiple nodes is generated.

[0080] Specifically, a hardware performance evaluation system is created. The weight coefficients of multiple evaluation indicators are determined by the analytic hierarchy process. The original values ​​of multiple evaluation indicators are extracted based on the hardware resource information of the centralized storage system. The original values ​​are numerically mapped. The data processing capability evaluation value of multiple nodes is generated by weighting the weight coefficients of multiple evaluation indicators and the values ​​of multiple standardized indicators.

[0081] S202b: Based on the data processing capability evaluation values ​​of multiple nodes, multiple processors within multiple nodes are filtered to obtain the filtered processor information.

[0082] Specifically, a performance threshold is set, and processors from nodes whose data processing capability evaluation value is higher than the threshold are selected to participate in subsequent grouping, while processor nodes marked as "test" or "reserved" are removed.

[0083] In this embodiment, the filtered processor information includes, but is not limited to, processor ID, node ID, and corresponding evaluation value.

[0084] S202c: Group multiple processors based on the filtered processor information to generate multiple processor groups.

[0085] Specifically, based on the screened processor information, the core parameters of multiple processors are extracted. The core parameters of multiple processors are used as feature vectors by a clustering algorithm to cluster the screened processors, verify the inter-group collaborative performance of the initial processor group, and generate multiple processor groups.

[0086] S202d: Associates multiple processor groups with group identifiers to generate processor group information.

[0087] Specifically, a unique group identifier is assigned to each processor group, and this unique group identifier is associated with and bound to the IDs of all processors within the group. The group information is then recorded in a mapping table.

[0088] As can be seen from the above embodiments, by collecting hardware resource information, evaluating the performance of hardware resources, generating data processing capability evaluation values ​​for multiple nodes, filtering the capabilities of multiple processors based on the data processing capability evaluation values ​​of multiple nodes, grouping the filtered processors, generating processor groups, associating processor groups with group identifiers, generating group information for multiple processors, avoiding resource waste through performance evaluation, and ensuring that the performance of each group meets the running requirements of the software instance.

[0089] Figure 4 A flowchart illustrating the centralized storage system fault handling method provided in this application embodiment. Figure 3 ,like Figure 4 As shown, before step S205, the following steps are also included:

[0090] S401: Obtain the running status data of the target instance.

[0091] Specifically, the runtime status data of the target instance is periodically collected by a monitoring agent deployed on the node where the target instance is located.

[0092] In this embodiment, the running status data includes, but is not limited to, the CPU utilization, memory usage, I / O request processing latency, error log output frequency, and process liveness heartbeat signal of the instance process.

[0093] S402: Compare the runtime status data of the target instance with multiple data thresholds in the monitoring strategy to generate comparison information.

[0094] Specifically, the preset monitoring strategy defines health thresholds for various status data, compares the collected real-time data with the health thresholds, and generates comparison information.

[0095] S403: Determine whether the target instance has malfunctioned based on the comparison information.

[0096] Specifically, the generated comparison information is logically compared with the fault judgment rules to determine whether the target instance has failed.

[0097] S404: If the target instance fails, an instance switching instruction is created, which is used to trigger the switching process between the target instance and the standby instance.

[0098] Specifically, when a target instance fails, the monitoring service module of the centralized storage system automatically generates a structured instance switching instruction, which is transmitted to the software controller via the message bus.

[0099] In this embodiment, the content recorded in the instance switching instruction includes, but is not limited to, the identifier of the target instance, the fault timestamp, and the command code that triggers the switching operation.

[0100] As can be seen from the above embodiments, by adding an active monitoring and judgment mechanism before instance switching, the operating status data of the target instance is obtained in real time and compared with the preset monitoring strategy, thereby accurately determining whether the target instance has failed. When a failure is determined to have occurred, an instance switching command is automatically created to trigger the subsequent switching process, realizing intelligent and rapid fault identification and automated switching triggering, improving the accuracy and timeliness of fault response, and reducing the risk of system delay or interruption caused by manual intervention and misjudgment.

[0101] Figure 5 A flowchart illustrating the centralized storage system fault handling method provided in this application embodiment. Figure 4 ,like Figure 5 As shown, step S202c includes:

[0102] S501: Extracts core parameters of multiple processors based on the filtered processor information.

[0103] In this embodiment, the core parameters of the processor include, but are not limited to, the number of physical cores, the number of logical cores, the clock speed, and the cache size.

[0104] S502: The core parameters of multiple processors are used as feature vectors by a clustering algorithm to cluster the selected processors and obtain the initial processor group.

[0105] Specifically, a clustering algorithm is used to take the feature vector of the processor as input, preset the number of groups, and divide processors with similar performance into the same cluster, with each cluster forming an initial processor group.

[0106] S503: Verify the inter-group coordination performance of the initial processor group and obtain inter-group latency information.

[0107] Specifically, network latency tests and memory access latency tests are performed between processors in different groups, and the average and maximum latency of inter-group communication are measured as inter-group latency information.

[0108] S504: If the inter-group delay information meets the preset communication delay threshold, the initial processor group is determined to be multiple processor groups.

[0109] Specifically, a preset communication delay threshold is set. If the communication delay between all groups is below the threshold, the initial grouping is confirmed as the final multiple processor groups.

[0110] As can be seen from the above embodiments, by extracting the core parameters of the processor and using a clustering algorithm for intelligent grouping, initial processor groups are generated. The inter-group collaboration performance is further verified and compared with a preset communication latency threshold to ensure that the final processor groups not only have similar hardware characteristics but also high intra-group collaboration efficiency and controllable communication latency, thereby improving the data processing efficiency and stability of the entire storage system.

[0111] Figure 6 A flowchart illustrating the centralized storage system fault handling method provided in this application embodiment. Figure 5 ,like Figure 6 As shown, step S202a includes:

[0112] S601: Create a hardware performance evaluation system, which includes multiple evaluation indicators.

[0113] In this embodiment, the evaluation dimensions of the hardware performance evaluation system include, but are not limited to, CPU computing power, memory bandwidth, and I / O capability.

[0114] Among them, the sub-indicators under the CPU computing power dimension include, but are not limited to, the number of cores, the single-core clock speed, and the cache size; the sub-indicators under the memory bandwidth dimension include, but are not limited to, memory capacity and memory frequency; and the sub-indicators under the I / O capability dimension include, but are not limited to, the storage controller throughput and the network interface card bandwidth.

[0115] S602: Determine the weight coefficients of multiple evaluation indicators using the analytic hierarchy process (AHP).

[0116] Specifically, the weight of each indicator in the evaluation system is calculated by comparing each other pairwise using the analytic hierarchy process (AHP).

[0117] S603: Extract the raw values ​​of multiple evaluation indicators based on the hardware resource information of the centralized storage system, and perform numerical mapping on the multiple raw values ​​to obtain multiple standardized indicator values.

[0118] Specifically, the raw values ​​of each indicator are extracted from the acquired hardware resource information, and the max-min normalization method is used to map all raw values ​​to the range of 0 to 1 to obtain standardized indicator values.

[0119] S604: Based on the weight coefficients of multiple evaluation indicators and the values ​​of multiple standardized indicators, a weighted calculation is performed to generate an evaluation value for the data processing capabilities of multiple nodes.

[0120] Specifically, the data processing capability evaluation value of a node is obtained by multiplying each standardized index value of the node by its corresponding weight coefficient and then summing the results.

[0121] As can be seen from the above embodiments, by establishing a structured hardware performance evaluation system and using the analytic hierarchy process to determine the weight of each evaluation index, the standardization and weighted comprehensive evaluation of the original indicators of hardware resources can be achieved, making the basis for processor grouping more accurate and comprehensive, avoiding the one-sidedness of single index evaluation, thereby ensuring that resource grouping is more in line with actual business load requirements and optimizing the overall utilization efficiency of system resources.

[0122] In one embodiment of this application, step S203 includes:

[0123] S203a: Determine the instance type of the software instance based on the processor's grouping information.

[0124] For example, if all processors in a group have a high core count and large cache, the software instance type corresponding to that group is determined as a "compute-intensive instance". If the network and storage I / O capabilities of the node where the processor is located are strong, the instance type can be determined as a "data throughput-intensive instance".

[0125] S203b: Call the preset instance template library according to the instance type of the software instance to obtain the instance template.

[0126] In this embodiment, the predefined configuration file template types in the instance template library include, but are not limited to, compute-intensive, data throughput-intensive, and low-latency types.

[0127] S203c: Determine the configuration parameters of the software instance based on the processor's grouping information, and write the configuration parameters of the software instance into the instance template to create multiple software instances.

[0128] In this embodiment, the configuration parameters include, but are not limited to, the list of processor cores bound to the instance, the allocated memory size, and the network port number.

[0129] Specifically, the configuration parameters are filled into the corresponding instance template obtained to generate the configuration file of the software instance. The system container management engine starts and initializes the software instance on the node according to the configuration file.

[0130] As can be seen from the above embodiments, by determining the instance type based on processor grouping information, retrieving the corresponding template from a preset instance template library, and then writing the specific configuration parameters into the template, multiple software instances matching the hardware group can be created quickly and in batches. This method streamlines and templates the instance creation process, improves deployment efficiency, ensures the consistency and manageability of instance configurations, and reduces the error rate of manual configuration.

[0131] In one embodiment of this application, after step S205, the method further includes:

[0132] S206: Collect performance time-series data through monitoring probes.

[0133] Specifically, after the instance switch is completed, performance time-series data is continuously collected at fixed time intervals through monitoring probes deployed at various layers of the system.

[0134] In this embodiment, performance timing data includes, but is not limited to, the CPU utilization, memory utilization, disk I / O throughput, and network bandwidth utilization of the standby instance after the switchover.

[0135] S207: Associate and parse the performance time-series data with the fault records of the target instance to generate fault association data.

[0136] Specifically, the performance time-series data collected within a time window before and after the fault occurred is correlated with the recorded fault events to analyze whether there were specific abnormal patterns in the performance time-series data before the fault occurred.

[0137] S208: Create fault model nodes based on fault association data, and update the fault model nodes and fault association data to the fault knowledge graph of the centralized storage system. The fault knowledge graph of the centralized storage system is used to query the switching strategy of software instances.

[0138] Specifically, based on the fault association data, the triple relationship of "fault phenomenon, cause and handling action" is extracted, and the triple is added as a new fault model node to the fault knowledge graph of the centralized storage system.

[0139] As demonstrated in the above embodiments, by collecting system performance time-series data through monitoring probes and performing correlation analysis with fault records, fault-related data is generated, fault model nodes are created, and the data is updated to the fault knowledge graph of the centralized storage system. This method enables the continuous accumulation and structured storage of fault handling experience, constructs a systematic fault knowledge base, and provides data support and an intelligent decision-making foundation for rapid diagnosis of similar faults, predictive maintenance, and the formulation of better switching strategies, thereby enhancing the system's self-learning and self-healing capabilities.

[0140] In one embodiment of this application, step S302 is followed by:

[0141] S701: Takes snapshots of data synchronization transactions through a distributed transaction manager, records data change operations at snapshot points, and generates a data change log.

[0142] Specifically, during the transmission of pending requests and pending data via the data link, the distributed transaction manager in the centralized storage system is activated to take snapshots of the pending request queue and cached data in the target instance's memory. The snapshot point is selected as the precise timestamp of the start of data transmission. The distributed transaction manager continuously tracks and records the new data change operations that occur on the target instance from the snapshot point until the transmission is completed, and records the change operations in sequence in the form of transaction logs to generate a data change log.

[0143] In this embodiment, the data change log includes, but is not limited to, the transaction ID, operation type, operation object identifier, change content, and incrementing logical timestamp for each change operation.

[0144] S702: Transmit snapshots and data change logs to the standby instance to be switched over via a data link, and reconstruct the data state based on the data change logs using the standby instance.

[0145] Specifically, when transmitting pending requests and pending data, the generated snapshot files and data change log files are transmitted as additional data streams to the standby instance to be switched over via a parallel channel.

[0146] Specifically, the standby instance's data reconstruction engine sequentially reads the data change logs and replays each data change operation according to the logical timestamp order in the logs, advancing the data state from the snapshot point to the latest state before the failure occurred.

[0147] In this embodiment, the process of rebuilding the data state is in an isolated state. During the rebuilding process, the standby instance is in a "data synchronization" state, suspending its response to external service requests until the log replay is completed and the state consistency is verified, at which point it is converted to a serviceable state.

[0148] As can be seen from the above embodiments, by integrating a distributed transaction manager, snapshots are created at key transaction points of data synchronization and subsequent change operations are recorded to generate a complete data change log. The snapshots and logs are transmitted to a standby instance using a data link, and the standby instance replays the operations according to the log sequence to reconstruct the data state. This achieves accurate capture, efficient transmission, and rapid reconstruction of data state changes. The log-driven incremental synchronization mechanism replaces the traditional full data copy, significantly reducing bandwidth consumption and time latency during the synchronization process.

[0149] Figure 7 This is a schematic diagram of the structure of a centralized storage system fault handling device provided in an embodiment of this application. Figure 7 As shown, embodiments of this application also provide a centralized storage system fault handling device 70, including: a first acquisition module 701, a grouping module 702, a first creation module 703, a generation module 704, and a switching module 705.

[0150] The first acquisition module 701 is used to acquire hardware resource information of the centralized storage system.

[0151] The grouping module 702 is used to group multiple processors in multiple nodes within a centralized storage system according to the hardware resource information of the centralized storage system, and generate grouping information for the processors.

[0152] The first creation module 703 is used to create multiple software instances based on the processor's grouping information.

[0153] The generation module 704 is used to allocate multiple software instances to multiple nodes in the centralized storage system, generate instance allocation information, and determine the target instance based on the instance allocation information, wherein the target instance is used to process requests and data of the centralized storage system.

[0154] The switching module 705 is used to respond to the failure of the target instance by performing an instance switching operation through the software controller based on the pending requests and pending data, and processing the pending requests and pending data based on the switched standby instance.

[0155] In one embodiment of this application, the switching module 705 includes:

[0156] The acquisition unit is used to acquire a set of backup instances in response to a failure of the target instance through the software controller.

[0157] The first determining unit is used to determine a list of standby instances based on the correlation between multiple standby instances and the target instance in the standby instance collection.

[0158] The first filtering unit is used to filter multiple standby instances in the standby instance list according to the pre-set business priority tags of the standby instances, and obtain the standby instances to be switched.

[0159] The sending unit is used to send the pending requests and pending data to the standby instance to be switched over via the software controller, and to process the pending requests and pending data according to the standby instance to be switched over.

[0160] In one embodiment of this application, the sending unit includes:

[0161] The first creation subunit is used to create a data link between the target instance and the standby instance to be switched over, based on the configuration information of the target instance and the standby instance to be switched over. The data link is used to transmit requests to be processed and data to be processed.

[0162] The transmission subunit is used to transmit the requests and data to be processed to the standby instance to be switched over via a data link, and to verify the integrity of the data.

[0163] The acquisition sub-unit is used to acquire the data access permissions of the target instance if the data integrity verification passes, and to synchronize the data access permissions of the target instance to the standby instance to be switched over, so as to complete the switchover between the target instance and the standby instance.

[0164] The processing subunit is used to process pending requests and pending data based on the switched standby instance.

[0165] In one embodiment of this application, the grouping module 702 includes:

[0166] The evaluation unit is used to evaluate the performance of the hardware resources of multiple nodes based on the hardware resource information of the centralized storage system, and generate evaluation values ​​of the data processing capabilities of multiple nodes.

[0167] The second filtering unit is used to filter multiple processors within multiple nodes based on the data processing capability evaluation values ​​of multiple nodes, and obtain the filtered processor information.

[0168] The grouping unit is used to group multiple processors according to the filtered processor information to generate multiple processor groups.

[0169] The association unit is used to associate multiple processor groups with group identifiers to generate processor group information.

[0170] In one embodiment of this application, the centralized storage system fault handling apparatus 70 further includes:

[0171] The second acquisition module is used to acquire the running status data of the target instance.

[0172] The comparison module is used to compare the running status data of the target instance with multiple data thresholds in the monitoring strategy and generate comparison information.

[0173] The judgment module is used to determine whether the target instance has failed based on the comparison information.

[0174] The second creation module is used to create an instance switching instruction if the target instance fails. The instance switching instruction is used to trigger the switching process between the target instance and the standby instance.

[0175] In one embodiment of this application, the grouping unit includes:

[0176] The first extraction subunit is used to extract the core parameters of multiple processors based on the filtered processor information.

[0177] The clustering subunit is used to cluster the selected processors by using the core parameters of multiple processors as feature vectors through a clustering algorithm, thus obtaining the initial processor group.

[0178] The verification subunit is used to verify the inter-group coordination performance of the initial processor group and obtain inter-group latency information.

[0179] The first determining subunit is used to determine the initial processor group into multiple processor groups if the inter-group delay information meets a preset communication delay threshold.

[0180] In one embodiment of this application, the evaluation unit includes:

[0181] The second creation subunit is used to create a hardware performance evaluation system, which includes multiple evaluation indicators.

[0182] The second determining subunit is used to determine the weight coefficients of multiple evaluation indicators through the analytic hierarchy process.

[0183] The second extraction subunit is used to extract the original values ​​of multiple evaluation indicators based on the hardware resource information of the centralized storage system, and to perform numerical mapping on the multiple original values ​​to obtain multiple standardized indicator values.

[0184] The weighted calculation subunit is used to perform weighted calculations based on the weight coefficients of multiple evaluation indicators and the values ​​of multiple standardized indicators to generate data processing capability evaluation values ​​for multiple nodes.

[0185] In one embodiment of this application, the first creation module 703 includes:

[0186] The second determining unit is used to determine the instance type of the software instance based on the processor's grouping information.

[0187] The calling unit is used to call the preset instance template library according to the instance type of the software instance in order to obtain the instance template.

[0188] The write unit is used to determine the configuration parameters of the software instance based on the processor's grouping information, and write the configuration parameters of the software instance into the instance template to create multiple software instances.

[0189] In one embodiment of this application, the centralized storage system fault handling apparatus 70 further includes:

[0190] The acquisition module is used to collect performance time-series data through monitoring probes.

[0191] The association parsing module is used to associate and parse performance time-series data with the fault records of the target instance to generate fault association data.

[0192] The third creation module is used to create fault model nodes based on fault association data, and update the fault model nodes and fault association data to the fault knowledge graph of the centralized storage system. The fault knowledge graph of the centralized storage system is used to query the switching strategy of software instances.

[0193] For a description of the features in the embodiment corresponding to the centralized storage system fault handling device, please refer to the relevant description of the embodiment corresponding to the centralized storage system fault handling method, which will not be repeated here.

[0194] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes at least one processor 801 and a memory 802. Optionally, the electronic device 80 further includes a communication component 803. The processor 801, memory 802, and communication component 803 are connected via a bus.

[0195] In a specific implementation, at least one processor 801 executes computer execution instructions stored in memory 802, causing at least one processor 801 to execute the above-described embodiment of the centralized storage system fault handling method.

[0196] The specific implementation process of processor 801 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0197] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0198] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0199] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0200] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the centralized storage system fault handling method when it is run.

[0201] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0202] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the centralized storage system fault handling method.

[0203] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the centralized storage system fault handling method.

[0204] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0205] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0206] The above provides a detailed description of a centralized storage system fault handling method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for handling faults in a centralized storage system, characterized in that, include: Obtain hardware resource information from the centralized storage system; Based on the hardware resource information of the centralized storage system, multiple processors in multiple nodes within the centralized storage system are grouped to generate processor grouping information; Multiple software instances are created based on the grouping information of the processor; The multiple software instances are allocated to multiple nodes within the centralized storage system, instance allocation information is generated, and a target instance is determined based on the instance allocation information, wherein the target instance is used to process requests and data from the centralized storage system; In response to a failure of the target instance, the software controller performs an instance switching operation based on the pending requests and pending data, and processes the pending requests and pending data based on the switched standby instance. The step of creating multiple software instances based on the processor's grouping information includes: The instance type of the software instance is determined based on the grouping information of the processor; According to the instance type of the software instance, a preset instance template library is invoked to obtain an instance template; The configuration parameters of the software instance are determined based on the grouping information of the processor, and the configuration parameters of the software instance are written into the instance template to create multiple software instances.

2. The centralized storage system fault handling method according to claim 1, characterized in that, The operation in response to a failure of the target instance includes, via a software controller, performing an instance switching operation based on pending requests and pending data, and processing the pending requests and pending data using the switched-off standby instance, including: In response to a failure of the target instance, a set of backup instances is obtained through the software controller; A list of backup instances is determined based on the correlation between multiple backup instances in the backup instance set and the target instance; The backup instances in the backup instance list are filtered according to the pre-set business priority tags of the backup instances to obtain the backup instances to be switched. The software controller sends the pending requests and data to the standby instance to be switched over, and processes the pending requests and data according to the standby instance to be switched over.

3. The centralized storage system fault handling method according to claim 2, characterized in that, The step of sending the pending request and pending data to the standby instance to be switched over via the software controller, and processing the pending request and pending data according to the standby instance to be switched over, includes: A data link is created between the target instance and the standby instance to be switched over, based on the configuration information of the target instance and the standby instance to be switched over, wherein the data link is used to transmit requests to be processed and data to be processed. The pending requests and pending data are transmitted to the standby instance to be switched over via the data link, and the data integrity is verified. If the data integrity verification passes, the data access permissions of the target instance are obtained, and the data access permissions of the target instance are synchronized to the standby instance to be switched, so as to complete the switch between the target instance and the standby instance; The pending requests and pending data are processed according to the switched standby instance.

4. The centralized storage system fault handling method according to claim 1, characterized in that, The step of grouping multiple processors among multiple nodes in the centralized storage system according to the hardware resource information of the centralized storage system, and generating processor grouping information, includes: Based on the hardware resource information of the centralized storage system, the performance of the hardware resources of multiple nodes is evaluated, and the data processing capability evaluation value of multiple nodes is generated. Based on the data processing capability evaluation values ​​of the multiple nodes, multiple processors within the multiple nodes are filtered to obtain the filtered processor information; Based on the filtered processor information, multiple processors are grouped to generate multiple processor groups; The multiple processor groups are associated with group identifiers to generate processor group information.

5. The centralized storage system fault handling method according to claim 1, characterized in that, The operation in response to a failure of the target instance, before the software controller performs an instance switching operation based on the pending request and pending data, further includes: Obtain the runtime status data of the target instance; The running status data of the target instance is compared with multiple data thresholds in the monitoring strategy to generate comparison information; Based on the comparison information, determine whether the target instance has malfunctioned; If the target instance fails, an instance switching instruction is created, wherein the instance switching instruction is used to trigger the switching process between the target instance and the standby instance.

6. The centralized storage system fault handling method according to claim 4, characterized in that, The step of grouping multiple processors according to the filtered processor information to generate multiple processor groups includes: Extract core parameters of multiple processors based on the filtered processor information; The core parameters of the multiple processors are used as feature vectors by a clustering algorithm to cluster the screened processors and obtain the initial processor grouping. Verify the inter-group coordination performance of the initial processor group to obtain inter-group latency information; If the inter-group delay information meets the preset communication delay threshold, then the initial processor group is determined to be multiple processor groups.

7. The centralized storage system fault handling method according to claim 4, characterized in that, The step of evaluating the hardware resources of multiple nodes based on the hardware resource information of the centralized storage system to generate data processing capability evaluation values ​​for multiple nodes includes: Create a hardware performance evaluation system, which includes multiple evaluation indicators; The weight coefficients of the multiple evaluation indicators were determined using the analytic hierarchy process (AHP). Based on the hardware resource information of the centralized storage system, the original values ​​of multiple evaluation indicators are extracted, and the original values ​​are numerically mapped to obtain multiple standardized indicator values. The data processing capability evaluation value of multiple nodes is generated by weighting the weight coefficients of the multiple evaluation indicators and the values ​​of the multiple standardized indicators.

8. The centralized storage system fault handling method according to any one of claims 1 to 7, characterized in that, The operation in response to a failure of the target instance, after the software controller performs an instance switching operation based on the pending requests and pending data, and processes the pending requests and pending data based on the switched-off standby instance, further includes: Performance time-series data is collected using monitoring probes; The performance time-series data is correlated and parsed with the fault records of the target instance to generate fault correlation data; Fault model nodes are created based on the fault association data, and the fault model nodes and the fault association data are updated to the fault knowledge graph of the centralized storage system, wherein the fault knowledge graph of the centralized storage system is used to query the switching strategy of software instances.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the centralized storage system fault handling method as described in any one of claims 1 to 8 when executing the computer program.

Images